Extendibility of Google Test

In this post we’ll discuss the extendibility of Google Test (GT), through the 3 most popular extension of the project from the 7 extension recognised by the project’s github page. They are the following: Google Test Adapter and C2TE, providing a GUI for GT and Cornichon, a DSL parser which can make the process of test creation much easier to less experienced end-users. The post discusses in detail the architectures, the contexts and the quality of the these softwares.

Google Test Adapter

Google Test Adapter (GTA), calls itself “a Visual Studio extension providing test discovery and execution of C++ tests written with the Google Test framework.” 1 Though being technically correct, it can also be seen as an extension on Google Test 2. It allows you to use GT with a visual interface you are probably familiar with: Microsoft Visual Studio3 . Adding a GUI has a few benefits 4, like making it easier for users to learn how GT and its functionalities are used. GTA also has some interesting features, like1:

  • Supporting parallel testing
  • Trait support by means of custom macros and/or trait assignment using regexes
  • Most important runtime options can be controlled via a toolbar without entering VS’s options
  • Team Foundation Server (TFS) support 5
  • Test output can be piped to test console
  • Installable as Visual Studio extension or NuGet development dependency

Contextual View

Here you see the contextual view of GTA 6 7. The picture speaks mainly for itself, but there are a few notes: CommonMark was used to convert markdown documents into HTML pages. This project has no logo, so the owner’s profile picture was used8 9.

GTA can be installed in three ways:

  • using the built in store of Visual Studio10
  • via an installer
  • as an NuGet package 11

The former 2 have no difference in functionality, but the latter loses some functionality, mainly the access to the “Solution test config file” and the direct debugging options. Most other options are still available, though they might require some extra set up.

Contextual view of Google Test

Code quality

The code quality is maintained in a few ways. It was originally developed with OpenCover 12 and Jetbrains 13. Now it is actively maintained using CodeCov14 and AppVeyor15. This has proven to be quite effective: With over 50 thousand downloads on the Visual Studio Store alone, there are as of writing (26 March 2020) no real issues open, only enhancements. Currently the project is not very active, with only a few small commits every month.

Catch2, Google Test and DOCtest Explorer for VSCode

Catch2, Google Test and DOCtest Explorer for VSCode (C2TE) is a C++ test explorer extension for the Visual Studio Code (VSCode) editor, it allows users to run Catch2, Google Test and DOCtest within the VSCode editor.1 Within VScode the tests and their results are visualized as shown in the figure below from the repository page.1

GUI in Visual Studio

Environment

The tool, C2TE, has an interesting environment in which it operates. From the operational viewpoint16 it is seen that the tool itself is not directly an extension of VSCode. It is an extension of the Test Explorer UI17, which is another extension of VSCode editor. Therefore we can say that C2TE is an “extension of an extension”. C2TE is therefore not directly operated, from within the VSCode editor, but is used by the Test Explorer UI as a test adapter extension17.

The Test Explorer UI itself provides an extensible user interface for running tests in VSCode for any testing framework for which a test adapter extension is available. Other tools operating on the Google Test library include gtest-runner18 and gtest-parallel19. The gtest-runner provides a “Qt5 based automated test-runner and Graphical User Interface for Google Test unit tests”.18 On the other hand, the gtest-runner provides speedup for single-threaded tests by executing them on separate processes.19 The C2TE covers both gtest-runner and gtest-parallel (partially) in a compact way by extending another tool, Test Explorer UI, for graphical visualization and using workers to perform parallel test execution.20

Code quality

The C2TE is written in TypeScript21, a strict superset of JavaScript. Where “TypeScript is a strongly typed, object oriented, compiled language”22 which compiles its code to JavaScript in order to be executed. Using TypeScript instead of JavaScript can provide developers more robust software while it is still easily deployable with a regular JavaScript application.21 A scripting language like JavaScript provides easy extensibility, while using TypeScript improves the code quality. There are almost no comments within the code and no documentation about the code could be found. There is however a small document about how to use the tool. The lack of code documentation does not have to be a problem for code quality at this point because there is only one main contributor. Test cases to test the tool, C2TE, itself are available within the repository.20 There are however no continuous integration tools used to enforce the quality of code changes.

Google Test

To test C++ code using the stand alone version of the Google Test library, tests have to be written using a certain syntax with macros. To operate and view Google Test execution in VSCode, no changes to the tests have to be made.

Comparison

The two above mentioned test adapters have a similar goal: To provide a simpler alternative for using GoogleTest within an Integrated Development Environment (IDE). Both project provides a GUI. Both support parallel testing and have a comparable amount of users (80 thousand for C2TE, 50 thousand for GTA) with no known issues on github1 20. So what are the differences?

The biggest difference is that the two extension make use of different IDEs. Google Test Adapter is a direct extension for Microsoft Visual Studio. On the other hand C2TE is an extension of Test Explorer UI which itself is an extension for Visual Studio Code. C2TE’s hierarchical architecture of extending another extension separates the concerns of visualizing the tests(Test Explorer UI) and interfacing with a test framework(C2TE). This is beneficial for the code quality and if you are planning to use more testing software. This also ensures stability and conformity among all testing programs. Finally, GTA tries to ensure continuous code quality by applying CI tools AppVeyor and Codecov.1 C2TE has currently no openly known CI systems in place on its Github repository20.

Cornichon

Cornichon is one of the extensions for Google Test. It is a small extension with only 99 commits and 1 contributor23. Cornichon is a DSL parser24. A DSL is a Domain Specific Language which is a language designed for a specific application domain25. Cornichon uses the Gherkin DSL24. The Gherkin language is designed to write test cases25 in such a way that a non-technical user can read and use those test cases. For example by using a natural language syntax. It is out of scope for this post to dive into the details of this language. After defining test cases in the Gherkin language, the test cases have to be translated to a General Programming Language (GPL) before they can be executed, for example C++. This is done in the syntax of a test framework in the given GPL, for example Google Test. In short, Cornichon translates the test cases written in the “simple” Gherkin language to test cases which can be read by a test framework in a GPL. The destination framework is up to the user. In our case, the output of Cornichon is the test cases for the Google Test framework. In this way, non-technical people can write tests, which can actually be parsed to real test cases.

Software architecture

Now that we have seen what Cornichon does and how it extends Google Test, let’s dive into the Cornichon project itself24. Cornichon support four destination languages: C#, C++, Python and Visual Basic. Each language comes with its own set of two frameworks. The project does not really have an architecture. It has four folders (one for each destination language). Within those folders there are only four files. One file for each framework, an utils file and a scenarios file. The files are also not large. The largest files contain around 100 lines of code. So the project does not really need an extensive architecture.

Software quality process

The files contain no comments at all. Normally this is a bad practice especially for open source software. However this project only has one contributor so it does not need comments at this moment to communicate to other users. But it may be useful in the future when other developers want to contribute or when the author wants to go through it years later. Tests are available to test the project using Travis CI as a Continuous Integration tool,

to enforce the software quality for each commit. However, no test coverage is communicated. Also, there are no open issues and no pull requests. This is probably due to the fact that it is a small project and it has only one contributor.

Conclusion

Google Test has seven recognized extensions on its Github page. Three of them were discussed in more detail in this post. Even though those three extensions are small and not very active, those extensions were the largest and most active projects of all the seven extensions. Therefore they are good candidates to dive into more detail. Due to the small size you barely see comments in the code, real issues and elaborate discussions in the pull requests. However, even if these extensions are small they are used relatively a lot by end users. The main goal of this post was to investigate the extensibility of the Google Test framework. As seen in the extensions, they did not require many commits and contributors to set up. This indicates that Google Test is quite extensible. Another thing we noticed are all the different programming languages that were used. We encountered TypeScript, Python and C#. This also indicates that Google Test is extensible in multiple programming languages.

  1. https://github.com/csoltenborn/GoogleTestAdapter ↩ ↩2 ↩3 ↩4 ↩5 ↩6

  2. https://github.com/google/googletest ↩

  3. https://visualstudiomagazine.com/articles/2017/08/15/ide-ranking.aspx ↩

  4. https://www.computerhope.com/issues/ch000619.htm ↩

  5. https://azure.microsoft.com/en-us/services/devops/server/ ↩

  6. https://visualstudio.microsoft.com/ ↩

  7. https://oz-code.com/ ↩

  8. https://github.com/Knagis/CommonMark.NET ↩

  9. https://github.com/Knagis ↩

  10. https://marketplace.visualstudio.com/items?itemName=ChristianSoltenborn.GoogleTestAdapter ↩

  11. https://www.nuget.org/ ↩

  12. https://github.com/OpenCover/opencover ↩

  13. https://www.jetbrains.com/ ↩

  14. https://codecov.io/ ↩

  15. https://www.appveyor.com/ ↩

  16. https://www.viewpoints-and-perspectives.info/home/viewpoints/operational/ ↩

  17. https://marketplace.visualstudio.com/items?itemName=hbenl.vscode-test-explorer ↩ ↩2

  18. https://github.com/nholthaus/gtest-runner ↩ ↩2

  19. https://github.com/google/gtest-parallel ↩ ↩2

  20. https://github.com/matepek/vscode-catch2-test-adapter ↩ ↩2 ↩3 ↩4

  21. https://www.typescriptlang.org/ ↩ ↩2

  22. https://www.tutorialspoint.com/typescript/typescript_overview.htm ↩

  23. https://github.com/zebmason/Cornichon ↩

  24. https://pypi.org/project/cornichon/ ↩ ↩2 ↩3

  25. https://en.wikipedia.org/wiki/Domain-specific_language ↩ ↩2

Google Test