Testing the vision of Google Test

Just writing code is often not enough, testing is an important aspect which should not be overlooked. Common methods to test software include assertions and unit tests. Some programming languages provide direct support for certain testing tools, many however lack all or most support1. One such language that lacks most test support is C++, the language only provides a standard library for basic assertions2.

But there is a test framework to save the day… the Google Test framework!

The Google Test repository is a merger of two former separate projects called GoogleTest and GoogleMock. GoogleTest is focused on providing testing tools, e.g. (advanced) assertions, while GoogleMock adds support for C++ mock classes3, both projects complementing each other.

Combined, both frameworks aim to be a powerful tool for testing C++ software on many different platforms, e.g. Linux, Mac OS X, Windows. With some additional changes, Google Test can also be applied for the C language.

END USER

System form

The end user can approach Google Test in different ways depending on the operating system, preferred code editor and other personal demands. If you want to start from scratch, you need to be familiar with CMake or you can just use the available material on this topic on the internet to compose a working CMakeLists.txt file.

You can also use different code editors with different support for google test and CMake. If you use a simple tool like Notepad++, you need to take care of downloading CMake, clone the google test project and the creation of the correct CMakeLists.txt files. If you use somewhat more advanced editors like VS Code or Sublime Text, you can download for example CMake as an add on, and you will probably have an easier time configuring CMake and Google Test. In these cases, you are going to use the terminal as output for the test results, when you start using the software. These solutions take more or less the same effort on different operating systems.

There is also a possibility to use more advanced code editors which makes it even easier to start with googletest. One of them is Visual Studio (VS) which automates almost every configuration. VS uses NuGet packages to make it simpler to integrate external tools (like googletest) into your project. It also provides a GUI in the form of the Test Explorer, which is compatible with multiple external testing tools including Google Test. After adding the package, you can simply create a Google Test project. Creating such a project, makes every necessary configuration and, except for some final workarounds, you can immediately start the testing.

System functionality

The end users implement unit test for their C++ code with the help of the Google Test library functions. Google Test provides switches which enables certain features of the testing tool making C++ testing even easier. For example, with certain switches you can define the behaviour of a test. If you have a fault, which only appears rarely, you can use the following switches to repeat the test a large amount of times and detect a failure immediately. –gtest_repeat=1000 –gtest_break_on_failure. Upon failure the tool goes into debug mode immediately. Among the great amount of built in assertions it also supports fatal (_ASSERT) and non-fatal (_EXPECT) assertions. You can also use custom assertions if needed. When you have completed a functional unit in your software in C++, you can start working on the test with Google Test. Usually you need to create a test folder with the test files inside it. In the test file you can define a test with the function TEST(TestCaseName, TestName). Moreover you can also define certain test conditions with the built in assertions like EXPECT_EQ, EXPECT_TRUE, ASSERT_EQ. After running the test, you will see the results in your UI.

CHARACTERIZATION

A characterization of the key capabilities and properties the system should provide:

  • An open source unit testing tool like Google Test should be to start with.

  • The project should have a clear documentation and should be easy to utilize for various needs.

  • It should be flexible in respect of various testing problems, with providing customizability besides the built in functions.

  • Portability is also an important aspect as the ability to use a software on multiple platforms is essential for its popularity.

  • As a unit test tool, it should provide various built in functions to make testing easier.

  • Continuing after a non-fatal failure is also important in order to discover such non-fatal or fatal failures in our code, which would only appear after a non-fatal failure.

STAKEHOLDERS AND CONTEXT

Google Test 4 is by code developers and testing engineers, for code developers and testing engineers. As C++ programming and bug fixing can be a painful and time consuming activity, it is no wonder tools like this one are being developed. It is not easy to make a complex system in C++ compared to some of the newer players, like python. It does however provide degrees of freedom and high precision that has a place in the world that will not soon disappear. C++, and the need of projects like Google Test, could even get a bigger market with the rise of the IoT. Many IoT projects have a limited amount of resources and therefor programmed using C++. Google Test could even get a bigger user base if they add support for the other C++ based languages.

As mentioned before, Google Test is meant for people that make programs using C and C++ 5, with maybe more of the C family to come. The project allows them to test their projects more thorough and rapidly. The users are also often domain experts, which means a lot of the feedback for the developers of Google Test that will be useful. Google creates more open source software, even though they do not make money out of it in a direct way. It does give them a good name under their potential future customers and employees. Google is also a stakeholder in increasing the amount of technology, as it allows for more data harvesting. Making a tool like this open source also means they themselves are getting some man-years of work for free in this tool, which they probably also use themselves. Though it is odd to speak of a “customer” when talking about open source code, companies do often have some policy about using it, as using it might leave legal issues surrounding intellectual property. However , as this project is a unit testing framework 6, this risk is minimized. This means a company has little reason to reject the use of GoogleTest with the exception of no guarantee for continued support.

FUTURE ROADMAP

Concerning the set of functionalities Google Test is already quite complete. All activities currently on the github page 7 are subject to fixing bugs, adding and changing documentation, cleaning the code etc. The list of issues also shows no desire of adding or changing entire features. However there are quite some related open source projects implementing an extension on the Google Test framework. These extensions may be added to the Google Test framework in the future, just as the Google mock extension. The future roadmap of Google does probably not include to add the functionalities of those extensions directly in the framework since they are created by external programmers. However they want to stay open for adding such extensions (as with Google mock). When a specific extension really becomes an important addition to the framework they want to add it as fast as possible. So scalability is really an important feature of this framework.

This way of working is completely contradictory with another big unit testing framework JUnit 8. JUnit actively adds extensions within their own framework whereas Google Test let other programmers create extensions and only include them when they can’t get around it. JUnit also has more features at the moment than Google Test. JUnit for example has a user interface with a progress bar included whereas Google Test only has this user interface as an external related project.

Some of the related projects of Google Test are:

  • Google Test UI 9 - This creates a user interface with the results of the tests instead of the text output in the command.

  • GTest TAP Listener 10 - This gives you the ability to write tests which uses event listeners with the TAP protocol.

  • gtest-parallel 11 - This can run tests in parallel which will result in a speed up.

There is one limitation of Google Test specified explicitly in their documentation. This could imply that this bug has a larger priority than all the other fixes in the issues and therefore explicitly added in their roadmap. The limitation states that Google Test is not thread safe on systems that does not have the pthreads library. The help of volunteers is asked in the documentation to fix this limitation.

To conclude, the general future roadmap of Google Test depends mostly on external programmers developing extensions. Bugs, documentation and refactoring are topics on which the project itself is still active.

  1. https://en.wikipedia.org/wiki/Unit_testing ↩

  2. https://support.microsoft.com/en-us/help/154419/frequently-asked-questions-about-the-standard-c-library ↩

  3. https://github.com/google/googletest/blob/master/googlemock/docs/for_dummies.md ↩

  4. https://github.com/google/googletest ↩

  5. https://en.wikipedia.org/wiki/Google_Test ↩

  6. https://stackoverflow.com/questions/3846198/when-to-use-debug-vs-unit-tests ↩

  7. https://github.com/google/googletest ↩

  8. https://github.com/junit-team ↩

  9. https://github.com/ospector/gtest-gbar ↩

  10. https://github.com/kinow/gtest-tap-listener ↩

  11. https://github.com/google/gtest-parallel ↩

Google Test