For those that don’t know - the favourite single-header C/C++ library that I’ve ever created is my unit test helper - utest.h. The main features are:

• Single header (duh).
• Works with C and C++.
• Allows a single executable to be created with both C and C++ tests linked in.
• Blindingly fast to compile and run.

After reading this post on another attempt at a googletest replacement jctest, and all the awesome analysis that Matthias included comparing the compile and runtime performance of jctest as compared to googletest, it suddenly hit me that I haven’t actually provided my performance analysis in any blog to show how good utest.h holds up against googletest (which was my original motivation for creating my own library). So lets start analysing utest.h against googletest, but also since jctest is the new kid of the block (or at least new kid to me) I’ve ran the tests against all three libraries.

Feature Parity⌗

The three libraries contain roughly the same set of base features - EG. the things you would need to add a bunch of tests and run them. There are some differentiation points that are worth calling out.

*: needs a single file to contain #define JC_TEST_IMPLEMENTATION

So as can be shown in the table - for the base features you need from a unit testing framework, each of the three libraries covers most of the bases required. If you want to use my utest.h library you will give up parameterized tests (although I do have indexed tests that allow you to do most of what parameterized tests do!), but you do get the ability to test C-only code to.

Lines of Code (loc)⌗

The googletest library is heavy in C++, heavy in includes and a lot of code. On the other hand jctest claims to be under 1000 loc (although the header file is 1478 loc, maybe he is meaning without comments?). My own utest.h is a mere 856 loc.

If we look at lines of code after preprocessing we can really see the difference between the libraries.

Some caveats:

• googletest has two rows in the chart - this is because it is the only library that has source files. The cost of adding googletest to your project is effectively 78209 + N * 58801 where N is the number of source test files you are using.
• jctest has two rows in the chart - this is because even though it is a single source library, a single C++ source file must have the #define JC_TEST_IMPLEMENTATION to bring in the actual implementation. The cost of adding jctest to your project is effectively 3718 + (N - 1) * 2541.
• the cost of adding utest.h to your project is effectively 5201 * N for C files and 5405 * N for C++ files.

You can see that jctest is the smallest in terms of lines of code, at nearly half the lines of code of utest.h once preprocessing has taken place. But both jctest and utest.h are over 10x than googletest.

Compile Time⌗

One of the points heavily emphasised in the jctest blog was the compile time improvements from switching away from googletest. Avoiding the STL and thus the include bloat seems to have been the main benefit for jctest in improving its compile time performance against googletest. My own utest.h uses a bunch of the same techniques as jctest to improve performance - avoid headers where you can and use nothing from the STL.

To compare compile time performance I’ve added simple test cases that include the header for the library and have a single test case defined.

#include "utest.h"

UTEST(test, x) {
  ASSERT_TRUE(1);
}

UTEST_MAIN();

I’ve used the default compile options to clang on MacOS (Apple LLVM version 9.1.0 (clang-902.0.39.1)), and ran it in hyperfine to get stable results.

clang -c -Iutest.h/ compile_tests/utest.c

So with all this in mind, lets look at the compile time performance of the three libraries!

You can see the correlation very clearly to the lines of code - we can see that both jctest and utest.h perform significantly better than googletest in cost to compile. Both libs are nearly 10x faster to compile than googletest in this simple test. Wowser, that’s a big difference.

There is very little difference between utest.h and jctest in time to compile - jctest is marginally faster to compile when not defining the implementation, but slower when defining the implementation.

Run Time⌗

To measure the time it takes for each unit testing framework to run, I added 5000 test cases to a single file, and then ran 3 configurations of these tests:

• Filter for a single test case defined in the 5000 (this pattern is often used during development when iterating on a single failure).
• Test 5000 passing tests (no failures).
• Test 5000 failing tests (they all fail).

I choose 5000 tests simply because it was the lowest number with which I could get repeatable runs through hyperfine without statistical anonmalies.

And the results look pretty good! Lets do some analysis:

• utest.h is just blindingly fast compared to the other two libraries - roughly 5x faster than jctest and 4x-9x faster than googletest depending on whether you are looking at the all pass / all fail cases.
• googletest is really slow at dealing with failing test cases.
• googletest beats jctest when all cases pass and also when filtering the test case.

So I did some digging, and it looks like utest.h is so much faster because, and you could probably have guessed this, I avoided any C++. To handle test cases in utest.h I didn’t have the ability to use classes to define the test case, and thus I had to resort to functions and taking pointers-to-functions to handle the test case registration. Both jctest and googletest resort to defining classes with virtual members (mostly as a way to handle test case fixtures), but this means there is inherently extra bloat because of C++.

If you want to compare the code of jctest vs utest.h in godbolt.org here are some links:

• utest.h - https://godbolt.org/z/Kmfl2w
• jctest - https://godbolt.org/z/XawJ0l

Conclusion⌗

All three of the libraries are pretty good in all honesty - googletest has a bazillion features but they come at a heavy compile time cost, jctest fixes the compile time cost but still has some run time overhead, and my own utest.h gives you C/C++ with fast compile and run times. I think my utest.h is the best library (but which author of a useful tool wouldn’t inherently think that?) but the real winner is the open source projects that make use of any of these libraries to improve the quality of their software.