Quicktest: wrap *testing.T for fun and profit

Many people use Go modules such as testify and gocheck to provide some sugar above the basic primitives provided by the standard library testing package, for example an easy way to make assertions.

Quicktest is exactly such a module. Its essence is captured by this type and constructor:

type C struct {
	testing.TB
	// contains filtered or unexported fields
}
func New(t testing.TB) *C

A *C value just wraps a *testing.T or *testing.B value, adding three main features:

- assertions
- pretty printing
- deferred execution

I’ll describe these features in turn.

Assertions

Quicktest’s assertion language is taken wholesale from Gustavo Niemeyer’s excellent gocheck module.

To some extent, that was because there are many hundreds of thousands of lines of tests using this idiom at Canonical, where quicktest was developed, so it’s desirable to be able to move existing tests over to quicktest mechanically. But mostly it’s because it’s a clean, simple and expressive design.

It has built-in support for using the cmp package for comparisons, and is easily extendable with your own Checker implementations, which means that the set of checkers provided by quicktest itself can be reasonably small.

An assertion looks like this, where qt.Equals could be replaced by any available checker. If the assertion fails, the underlying Fatal method is called to describe the error and abort the test.

c.Assert(someValue, qt.Equals, expectedValue)

If you don’t want to abort on failure, use Check instead, which calls Error instead of Fatal:

c.Check(someValue, qt.Equals, expectedValue)

Pretty printing

On test failure, quicktest takes care to produce a nice looking error report, including pretty-printing both the obtained value and the expected value (it uses github.com/kr/pretty by default, but that’s configurable) or, for deep value comparisons, using cmp.Diff to show the difference between the two.

Here’s an example test failure. Note that it also shows the actual assertion statement that failed, which is often useful.

--- FAIL: TestFoo (0.00s)
    quicktest.go:219: 
        error:
          values are not deep equal
        diff (-got +want):
            &test.T{
          - 	A: "hello",
          + 	A: "goodbye",
            	B: 99,
            }
        stack:
          /home/rog/src/s/tst_test.go:22
            c.Assert(x, qt.DeepEquals, &T{
                A:  "goodbye",
                B:  99,
            })

Deferred execution

This is like the Go defer statement except test-scoped instead of function-scoped. It’s useful for creating OS-level resources such as temporary directories. Just like the function calls passed to defer, functions passed to Defer are executed last-in, first-out order.

func (c *C) Defer(f func())

To trigger the deferred behaviour, call c.Done:

func (c *C) Done()

For example, here’s the quicktest’s implementation of C.Setenv, which sets an environment variable for the rest of the test, returning it to its original value afterwards:

func (c *C) Setenv(name, val string) {
	oldVal := os.Getenv(name)
	os.Setenv(name, val)
	c.Defer(func() {
		os.Setenv(name, oldVal)
	})
}

If you create a *C instance at the top level, you’ll have to add a defer to trigger the cleanups at the end of the test:

defer c.Done()

However, if you use quicktest to create a subtest, Done will be called automatically at the end of that subtest. For example:

func TestFoo(t *testing.T) {
	c := qt.New(t)
	c.Run("subtest", func(c *qt.C) {
		c.Setenv("HOME", c.Mkdir())
		// Here $HOME is set the path to a newly created directory.
		// At the end of the test the directory will be removed
		// and HOME set back to its original value.
	})
}

Comparison

Quicktest invites comparison with other more popular Go testing frameworks. Two commonly used ones are testify and gocheck, so I’ll talk a bit about how they contrast with quicktest.

Testify

One very popular testing module is testify, which provides, amongst others, an assert and a require package.

The API surface area of these packages is impressively large. Between them, they export over 650 functions and methods. This contrasts with quicktest which exports about 20. The testify code is about 13000 lines of Go, where the quicktest code is under 1200)

This difference is largely because of orthogonality (any quicktest checker can be used both with Assert and with Check), and because quicktest restricts itself to the most commonly used check operations.

Testify also provides a suite package which suffers from the same potential issues as gocheck’s suites, detailed in the next section.

Quicktest’s API is both more general (it provides more functionality, such as Defer) and easier to learn (it has a much smaller API). It’s also interesting to note that quicktest is entirely compatible with testify – because *qt.C implements testify.TestingT, it can be used as an argument to any of the testify functions.

Gocheck

Another well known testing framework is gocheck. It was experience with gocheck that provided most of the impetus for quicktest’s design. As mentioned earlier, the assertion language is elegant and natural feeling, but some other aspects don’t work as well.

Gocheck doesn’t fit neatly with the standard testing package. The testing package provides subtests, but gocheck doesn’t use that functionality or provide it. Tests in gocheck are defined as methods on "suite" types; you need to remember to call the top level TestingT function, and it introduces a bunch of new command line flags to control test execution instead of just using the existing infrastructure.

Gocheck doesn’t support parallel tests, and it wouldn’t be easy to make it do so. As all gocheck tests are written as methods, and it’s common for those methods to mutate the receiver (for example to set up per-test state), it would be racy to run tests concurrently. Quicktest instead ties state to local variables, which means that it’s trivial to support parallel tests.

Gocheck does provide functionality something similar to quicktest’s Defer in the form of setup/teardown methods that can be defined on a test suite value. It’s common to embed helper types inside a suite type, but it turns out that it’s awkward and somewhat error-prone to compose such helper types together: because you need to define a single type containg all the setup and teardown logic for your tests, you tend to end up with large types that embed several helpers. For example, see this example:

// FakeHomeSuite sets up a fake home directory before running tests.
type FakeHomeSuite struct {
    CleanupSuite
    LoggingSuite
    Home *FakeHome
}

Why does FakeHomeSuite embed LoggingSuite? That’s because it’s awkward to include two independent helpers in your test suite, because you need to remember to call all the fixture methods (SetUpTest/TearDownTest etc) on all the helpers. But because of this, these testing helper types become non-orthogonal. If you embed two types that both embed the same helper type, you’ve duplicated the helper type’s state and bad things can happen.

By contrast, quicktest’s Defer method composes easily and feels natural and "Go-like" to use.

Summary

Quicktest provides a small but useful layer on top of the standard library’s *testing.T type. The primitives that it provides map straightfowardly to the underlying testing package’s functionality and compose nicely, making it easy to build expressive testing helper functionality without straying far from standard Go idioms.