Software Philosophy

Compile-Time Unit Testing

 #cpp   #unit testing   #debugging   #static typing   #static binding   #static assert   #constant expression   #partial specialization   #compile-time 

2016-04-04

These days, unit testing became a major part of software engineer's work. We familiarized ourselves with unit testing frameworks based on xUnit architecture (e.g. JUnit, SUnit) where tests are executed during run time of the test program. As of time of this writing, it's the most popular way of testing in an object-oriented environment. Few years ago, I started using compile-time unit testing in my C++ experiments. As it turned out quite well, I will try to share my experience in here. This article describes reasons for using compile-time unit testing, provides examples of code that can be tested this way, and shows how to debug such tests.

NOTE

Examples in this article are written in C++, but used terminology and the concept of compile-time unit testing applies to many other statically-typed languages. All examples should be compilable with gcc4.9. They can be found on github in mchalapuk/static-tests repo.

Static Binding

There are two ways of binding values to symbols in C++ language—static (AKA early) binding and dynamic (AKA late) binding. To make a long story short—a (not only) example of dynamic binding is method polymorphism, where pointer to method's implementation is resolved during program run time (wich is late). Static binding happens during compilation. Value is computed by the compiler and inserted into the output code instead of a computation procedure.

The simplest example of a value that is bound statically is a literal of any type.

auto i = 0; // zero is statically bound (value is known during the compilation)
auto s = "static binding"; // string literal is statically bound

But what makes static binding really powerful is a computation performed at compile time.

// Mathematical PI constant.
// Literal of type double is bound statically.
const auto PI = 3.141592L;

// Degrees to radians convertion ratio.
// The compiler will compute right-hand side expression and replace it with
// single value, because all values used in the expression are bound statically.
// This means that value of DEG_2_RAD is also bound statically.
const auto DEG_2_RAD = 2 * PI / 360;

Those examples are just scratching the surface of static binding in C++.

Constant Expressions

Since C++11 standard, we have a feature of constant expressions in the language. It extends the posibilities of static binding from simple expressions to whole blocks of code.

// Function converting degrees into radians.
// Declared as constant expression, which means that it can be executed
// during the compilation (the code is statically bound).
constexpr double deg2rad(double deg) {
  return deg * DEG_2_RAD;
}

There are many requirements that a function must meet to be a valid constant expression1. In essence, blocks of code declared as constant expressions can not have any side effects. As its code is statically bound, deg2rad function can be tested during the compilation.

Writing compile-time unit tests is quite easy since C++11 brougth static_assert into the game. It works the same way as normal assertion but with one difference—it is evaluated during the compilation. This implies that its argument must be a constant expression. If expression doesn't convert to true, compilation fails.

#include <cmath>

// A helper function for comparing floating point numbers.
constexpr bool float_equals(double lhs, double rhs) {
  return std::abs(lhs - rhs) < 0.00001L;
}

namespace deg2rad_tests {

static_assert(float_equals(3.141592L, deg2rad(180)), "deg2rad(180) == 3.141592L");
static_assert(float_equals(0.017453L, deg2rad(1)), "deg2rad(1) == 0.017453L");
static_assert(float_equals(17.104226L, deg2rad(980)), "deg2rad(980) == 17.104226L");

} // namespace deg2rad_tests

Above code contains three compile-time unit tests of deg2rad function. Compiling this code runs the tests. Tests in this form are not very maintainable as they contains many duplications. Function calls contain the same information as assertion messages. This problem can can be resolved by using a macro-definition instead of a helper function.

#include <cmath>

#define STATIC_ASSERT_FLOAT_EQUALS(expected, actual) \
  static_assert(std::abs(expected - actual) < 0.00001L, #expected " == " #actual);

namespace deg2rad_tests {

STATIC_ASSERT_FLOAT_EQUALS(3.141592L, deg2rad(180));
STATIC_ASSERT_FLOAT_EQUALS(0.017453L, deg2rad(1));
STATIC_ASSERT_FLOAT_EQUALS(17.104226L, deg2rad(980));

} // namespace deg3rad_tests

All further examples of tests will use macro-definitions.

Object methods, static methods and construtors can also have constexpr modifier. The next example contains a simple, statically bound class that represents a vector in 2D space. Its constructor, getters and arithmetic operators are unit-tested during the compilation.

// Two-dimentional vector.
// Class is immutable, all operations (including constructor) are constant expressions.
class vec2 {
 public:
  constexpr vec2(int x = 0, int y = 0) : _x(x), _y(y) {
  }

  constexpr int x() const {
    return _x;
  }
  constexpr int y() const {
    return _y;
  }

  constexpr vec2 operator+(vec2 const& rhs) const {
    return vec2(_x + rhs._x, _y + rhs._y);
  }
  constexpr vec2 operator-(vec2 const& rhs) const {
    return vec2(_x - rhs._x, _y - rhs._y);
  }

 private:
  int _x;
  int _y;
}; // class vec2

#define STATIC_ASSERT_EQUALS(expected, actual) \
  static_assert(expected == actual, #expected " != " #actual)

namespace vec2_tests {

STATIC_ASSERT_EQUALS(1, vec2(1, 2).x());
STATIC_ASSERT_EQUALS(2, vec2(1, 2).y());
STATIC_ASSERT_EQUALS(3, (vec2(1, 2) + vec2(2, 3)).x());
STATIC_ASSERT_EQUALS(5, (vec2(1, 2) + vec2(2, 3)).y());
STATIC_ASSERT_EQUALS(-1, (vec2(1, 2) - vec2(2, 3)).x());
STATIC_ASSERT_EQUALS(-1, (vec2(1, 2) - vec2(2, 3)).y());

} // namespace vec2_tests

Number of unit tests in above example is minimal. Production code will probably contain more than one test per operation.

Meta-Functions

There is a popular statement that meta-language of C++ is C++ itself. It's true, but more precicely, meta-programming in C++ is done using C++ templates, which are whole another, purely functional programming language, embedded into C++. Templates provide a way to abstract out a common code from a family of classes (or functions) without use of runtime polymorphism2. C++ code is generated from template specializations during the compilation. That's why templates can be compile-time unit-tested.

Meta-function is a template structure that takes a type as parameter and returns another type in a typedef or a value in a static variable (or both). One of the simplest meta-functions is the one that adds pointer to a type.

// Since C++11, there is a one-liner for doing this.
template<class arg_t> using add_pointer_t = arg_t*;

In order to test this meta-function, one of type traits from the standard library (std::is_same) must be used. It takes two types in template parameters and returns true it they are identical.

#include <type_traits>

#define STATIC_ASSERT(expr) static_assert(expr, #expr)

namespace add_pointer_t_tests {

STATIC_ASSERT((std::is_same<add_pointer_t<int>, int*>::value));
STATIC_ASSERT((std::is_same<add_pointer_t<long>, long*>::value));
STATIC_ASSERT((std::is_same<add_pointer_t<float>, float*>::value));
STATIC_ASSERT((std::is_same<add_pointer_t<int*>, int**>::value));

} // namespace add_pointer_t_tests

The next example is less trivial. It contains a meta-function that counts types passed in its template parameters. Function uses:

// Meta-function that counts types.
// It takes a parameter pack (which can be empty) in its template argument.
// There is no implementation, because everything will be handled in specializations.
template <class ...args_t>
struct count_types;

// Partial specialization that will be chosen by the compiler
// if there is at least one type (head_t) in parameter pack.
template <class head_t, class ...tail_t>
struct count_types<head_t, tail_t...> {
  // value is computed recursively by calling the same meta-function
  static const size_t value = 1 + count_types<tail_t...>::value;
};

// Partial specialization that will be chosen only if parameter pack is empty.
template <>
struct count_types<> {
  static const size_t value = 0;
};

#define STATIC_ASSERT_EQUALS(expected, actual) \
  static_assert(expected == actual, #expected " == " #actual)

namespace count_types_tests {

STATIC_ASSERT_EQUALS(0, (count_types<>::value));
STATIC_ASSERT_EQUALS(1, (count_types<int>::value));
STATIC_ASSERT_EQUALS(3, (count_types<int,float,double>::value));
STATIC_ASSERT_EQUALS(10, (count_types<int,int,int,int,int,int,int,int,int,int>::value));

} // namespace count_types_tests

Debugging

GCC compiler prints very limited information when static assertion fails. To demonstrate this, let's change one of the partial specializations of count_types, so it will contain a bug.

template <class head_t, class ...tail_t>
struct count_types<head_t, tail_t...> {
  // A programmer forgot to pass tail_t... to recursive call
  static const size_t value = 1 + count_types<>::value;
};

Now, when processing following assertion...

STATIC_ASSERT_EQUALS(3, (count_types<int, float, double>::value));

...the compiler will produce following output.

error: static assertion failed: 3 == (count_types<int, float, double>::value)
   static_assert(expected == actual, #expected " == " #actual)

The information of wich assertion has failed is present, but it is not known what value was returned from the tested meta-function. A way to print a static value is to put it in a template parameter that will be visible in the ‘required from’ stack trace (a part of compilation error output). Next code snippet contains an utility for printing static values this way.

// Compilation error must be generated in order to print anything.
// This function generates this error (static assertion always fails).
// Template parameters of calls containing failed static assertions
// are never printed by gcc compiler (val_t will not be printed).
template <class val_t>
constexpr val_t staticAssertFail(val_t value) {
  // Value must be passed to trick the compiler
  // that it's needed to evaluate the assertion.
  static_assert(value && false, "debugging value");
  return value;
}

// Another function that calls the function above.
// Template parameters of this function will be printed
// in 'required from' stack trace.
template <class val_t, val_t value>
constexpr val_t debugStaticValue() {
    return staticAssertFail(value);
}

// Helper macro-definition for usage in debugged code.
// To use it, just wrap a value you wish to debug.
#define DSV(v) \
  debugStaticValue<std::remove_cv<std::remove_reference<decltype(v)>::type>::type, v>()

DSV helper macro is reusable. You may copy and paste it into your project if you like (or download debug.hpp file from mchalapuk/static-tests project).

// Erroneus value wrapped in DSV macro.
STATIC_ASSERT_EQUALS(3, DSV((count_types<int, float, double>::value)));

Above code produces following compiler output.

required from ‘constexpr val_t debugStaticValue() [with val_t = size_t; val_t value = 1ul]’
required from here
error: static assertion failed: debugging value
error: static assertion failed: 3 == DSV((count_types<int, float, double>::value))
note: in expansion of macro ‘STATIC_ASSERT_EQUALS’

Use of helper macro resulted in three extra lines at the beginning of the output. First line contains erroneus value (1ul) and its type (size_t).

Unfortunately, above method can not be used on all values. There are strict requirements that a value must meet to be a valid non-type template parameter.

Rationale

There are many reasons for using static unit testing. First of all, having statically bound code implies statically bound tests. If all code is executed during the compilation, but tests are done in runtime, resulting program will contain only comparisons between meaningless values.

// For example following program.
int main() {
  assert std::numeric_limits<unsigned char>::max == 255;
}

// Will compile to something like this.
int main() {
  assert 255 == 255; // What are the meanings of this values?
}

Using static assertions instead of runtime ones will result in everything (code and tests) being run on the compiler. Resulting test program will be empty, which is much more elegant.

Elegance is not a dispensable luxury but a quality that decides between success and failure.

~ Edsger W. Dijkstra

Static assertions are also the only way of checking that code is statically bound, as intended. Skipping this type of test may result in instructions sliping to runtime by accident which can drastically affect program's performance.

Compile-time unit testing reduces turn around time, as full project doesn't have to be built before running any tests. Static assertions fail during compilation of a single file which can greatly reduce feedback time and increase maintainability of a code base.

Summary

Examples from this article include C++ constructs of three types: statically bound functions, statically bound classes, and meta-functions. These (more or less) cover the full spectrum of code which can be tested during the compilation. Things to take out from this article are that many features of modern C++ utilize static binding, and that statically bound code requires compile-time unit tests.

As of now, compile-time unit-testing in C++ is not ideal. Besides static_assert, there aren't any debugging facilities of compile-time code. Templates provide a simple way of debugging statically bound values, but this technique works only with types that are accepted as template parameters, which of—most notably—floating point numbers are not part. Also, the fact that there are no test reports may be interpreted as a problem. The only way of proving that test is run by the compiler is to make it not pass. On the other hand, this is consistent with Linux's ‘empty output means no error’ convention, so maybe it's not such a downside after all.

Good knowledge of rescent C++ standards and build environments is definitely a requirement for an efficient use of compile-time unit testing. The technique still needs work, but the benefits most definitely outweight the problems. If you are not up-to-date with current C++, this may be a good reason to brush up your skills.

References

  1. A White paper titled “Compile-time Unit Testing” written by Aron Barath and Zoltan Porkol (Lorand University) contains many examples written in C++ and in experimental language created for purposes of testing this concept.
  2. Peter Dimov's article titled “Simple C++11 metaprogramming” contains many examples of operations on type lists implemented using parameter packs.
  3. Eric Niebler wrote a blogpost titled “Tiny Metaprogramming Library” where he argues that we don't need big meta-programming libraries like Boost.MPL, as type lists are now implemented in C++ with just one line of code. He provides many examples.
  4. An old StackOverflow thread titled “Runtime vs Compile time” in great detail describes the difference between runtime and compile-time errors.

Annotations

{1} Please consult the documentation of constexpr keyword for full list of requirements.

{2} There is even a template technique called static polymorphism, where implementations are bound to calls during the compilation.

Maciej Chałapuk

blog comments powered by Disqus