Does it seem like writing unit tests is endlessly boring and time consuming? Does your test suite require constant attention and tweaking, and yet the stream of bugs never seems to end? I feel the same way. Moreover, if I would knew how much time I would spend writing unit tests, I most likely would have picked a different profession. There has to be a better way.
In this post, I’d like to share one better way for testing binaries. A way that allows you to add new test cases in seconds, avoid writing manual assertions, and makes it easy to write self documenting tests by non technical users. The catch? You must upgrade to dune 2.7 and enable the new
$ cat dune-project (lang dune 2.7) (cram enable)
Now dune will treat every file and directory that ends with
.t as a cram test.
Let’s create a trivial test
wc.t to test the word counting utility:
Test the behavior of wc.t. Any line that doesn't start with 2 spaces is a comment (like this one).
Next, we’ll create a sample file to feed to wc:
Note the two spaces before the command: $ cat >sample.txt <<EOF > a > b > c > EOF
The command above creates a file with 3 lines. Note the leading 2 spaces and the
$ denoting a command. We also use the heredoc syntax to pipe multiple lines to
We’ll finally write a test that makes sure that
Count the lines: $ wc -l sample.txt
Note how we didn’t mention the expected output anywhere. This is where secret sauce comes in. We just run the test with dune:
$ dune runtest | $ wc -l sample.txt +| 3 sample.txt
And dune is helpful enough to fill in the output for us after promoting:
$ dune promote $ dune runtest # now the tests pass
If we modify the
wc utility to give a different result, the test will now fail because the command produced a different output. This style of testing is called expectation (or snapshot) testing. Here this style is dressed up in a shell like syntax to give us the cram test.
Dune 2.7 offers full support for this style, and we recomend it to all users. Do cram tests scale? In the dune project, this is our main testing mechanism and we have over 200 cram tests in our test suite. We use this approach to test and document both new features and regressions. So far we’ve been very satisfied with this approach, and we’re happy to share it with our users.
As usual, there’s far too much to describe in a single blog post. The rest is thoroughly documented in our manual
I look forward to answer any questions you might have about cram.