Unit testing

Imagine that you're finishing up your project for data structures. You need your program to pass all of the professor's test cases this time or you won't get a good grade.

Your program works for the few tests you've written and you're pretty confident that it's in working order. You scp it to your homework server, submit it and wait until it's graded...

Project X: 5/15 test cases passed.

If only you had written more unit tests!

But what is a unit test?

A unit test is test that asserts the functionality of each individual, testable "unit" of code.

Consider the following class that simply reverses a std::string:

class Reverser {
  public:
    Reverser(std::string& s) : s_(s) {} 
    void reverse() {
      std::size_t len = s_.size();
      for(std::size_t i = 0; i <= len/2; ++i) {
        std::swap(s_[i], s_[len-i-1]);
      }
    }
    std::string get_s() const { return s_; }
  private:
    std::string s_;
};

When we test it, it seems fine.

#include <iostream>

int main() {
  std::string s = "hello";
  Reverser r(s);
  std::cout << "Before reverse:\n" << r.get_s() << std::endl;
  r.reverse();
  std::cout << "After reverse:\n" << r.get_s() << std::endl;

  return 0;
}

Running this, we see the output is:

Before reverse:
hello
After reverse:
olleh

Done. We've written the code that reverses a string and we're happy with the results.

However, we haven't tested it with with many inputs.

Let's try again, but this time, instead of std::string s = "hello"; we will use std::string s = "hi";

Before reverse:
hi
After reverse:
hi

And just like that, we figured out our code was wrong. Aren't you glad that we tested a little bit?

Unit testing is more formal than this manual testing that we've done in this contrived example.

Unit testing frameworks

For C++, there are myriad unit testing frameworks. For this text, I will use the Catch framework. It is quite intuitive and is header-only, which is nice for rapid testing.

To get this working, you'll want to clone his repo from GitHub. Working on a Mac, I elected to clone my copy to ~/Library, but this is really your choice.

Once cloned, you'll want to add the following line to your ~/.bashrc file:

export CATCH_DIR='/path/to/Catch/include'

This will save us a lot of typing when we compile our tests.

Next, you'll want to create a test file. I called mine test_reverse.cc whose contents are as follows:

#define CATCH_CONFIG_MAIN
#include "catch.hpp" 
#include "reverse.h" // Header that contains my Reverser class
#include <string>


TEST_CASE( "Odd length strings are reversed", "[reverse]" ) {
  std::string s("Hello");
  Reverser r(s);
  r.reverse();
  REQUIRE( r.get_s() == "olleH" );
  std::string s2("Howdy");
  Reverser r2(s2);
  r2.reverse();
  REQUIRE( r2.get_s() == "ydwoH" );
}

TEST_CASE( "Even length strings are reversed", "[reverse]" ) {
  std::string s("howdy!");
  Reverser r(s);
  r.reverse();
  REQUIRE( r.get_s() == "!ydwoh" );
}

TEST_CASE( "Zero length strings are reversed", "[reverse]" ) {
    std::string s("");
    Reverser r(s);
    r.reverse();
    REQUIRE( r.get_s().empty() );
}

I'll explain this in detail soon, but for now we will compile and run the code.

You can compile this with the following command:

g++ test_reverse.cc -I${CATCH_DIR} -o test_reverse

Now we have an executable to test our reverser; namely, test_reverse. When we execute it, we should see the following:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
test_reverse is a Catch v1.3.2 host application.
Run with -? for options

-------------------------------------------------------------------------------
Even length strings are reversed
-------------------------------------------------------------------------------
test_reverse.cc:19
...............................................................................

test_reverse.cc:23: FAILED:
  REQUIRE( r.get_s() == "!ydwoh" )
with expansion:
  "!ywdoh" == "!ydwoh"

===============================================================================
test cases: 3 | 2 passed | 1 failed
assertions: 4 | 3 passed | 1 failed

Congratulations. You've written and run a unit test.

Now let's get back to the test we've written and look at each unfamiliar piece, line-by-line:

First,

#define CATCH_CONFIG_MAIN

This line forces Catch to provide a main for the test cases that will be provided. This should only be included in one .cc file.

Next,

#include "catch.hpp" 

When we compiled, we used the flag -I${CATCH_DIR}, which tells the compiler to look for headers in ${CATCH_DIR}, which we conveniently defined in our ~/.bashrc. This line includes the Catch header file which is found at that location.

Next,

TEST_CASE( "Foo", "[bar]" ) 

This line is a macro that actually does the heavy lifting. It runs a test with description Foo and (optionally) adds the tag bar to it. The body of the macro is the setup for the assertion you want to make. As you can see, we can write multiple tests in one test case (see "Odd length strings are reversed").

Finally,

  REQUIRE( expression );

This line is why we write the tests. The REQUIRE macro determines the validity of the expression.

More information about Catch can be found at its GitHub site. I will use features of Catch throughout this text to verify correctness.

Back to the test. Catch is direct as it tells us exactly which assertion failed and the values that it failed on:

test_reverse.cc:23: FAILED:
  REQUIRE( r.get_s() == "!ydwoh" )
with expansion:
  "!ywdoh" == "!ydwoh"

This is great because we know for which subset of tests it seems to fail: even length strings. It seems to pass on empty strings and odd-length.

We can also seem that the output of r.get_s() is actually quite close to the real reversed string. We can see that only the middle two characters are incorrect. This is a huge hint.

Since we are only iterating to half the length of the input string, the last character would be the middle character(s). For odd length strings, we would stop before we got the the middle character, but for even strings we are actually reversing the middle two characters twice since our stop-check is inclusive.

We found the bug - an off-by-one error in our for-loop. If we replace

void reverse() {
  std::size_t len = s_.size();
  // bug with i <= len/2
  for(std::size_t i = 0; i <= len/2; ++i) {
    std::swap(s_[i], s_[len-i-1]);
  }
}

with

void reverse() {
  std::size_t len = s_.size();
  // bug fixed
  for(std::size_t i = 0; i < len/2; ++i) {
    std::swap(s_[i], s_[len-i-1]);
  }
}

then recompile and rerun the tests:

===============================================================================
All tests passed (4 assertions in 3 test cases)

we find all of our tests pass. This is great news and it didn't take much time at all.

I could've figured that out without unit tests!

Maybe. Not all code is as contrived as a string reverser, though. In large, complex systems much like the ones you'll see in industry, it isn't enough to debug by inspection.

Something else that unit tests provide that debugging by inspection doesn't is developer efficiency.

Consider an application with an enormous class hierarchy, complex relations, and tons of methods. In this case, recompiling and rerunning code to find the offensive lines after a few changes is wasteful. Compilation time for tests is seconds and can help you identify bugs in no time.