Cram - Tests on Short Notice

Does it seem like writing unit tests is endlessly boring and time consuming? Does your test suite require constant attention and tweaking, and yet the stream of bugs never seems to end? I feel the same way. Moreover, if I would knew how much time I would spend writing unit tests, I most likely would have picked a different profession. There has to be a better way.

In this post, I’d like to share one better way for testing binaries. A way that allows you to add new test cases in seconds, avoid writing manual assertions, and makes it easy to write self documenting tests by non technical users. The catch? You must upgrade to dune 2.7 and enable the new cram extension:

$ cat dune-project
(lang dune 2.7)
(cram enable)

Now dune will treat every file and directory that ends with .t as a cram test.

Let’s create a trivial test wc.t to test the word counting utility:

Test the behavior of wc.t. Any line that doesn't start with 2 spaces
is a comment (like this one).

Next, we’ll create a sample file to feed to wc:

Note the two spaces before the command:
  $ cat >sample.txt <<EOF
  > a
  > b
  > c
  > EOF

The command above creates a file with 3 lines. Note the leading 2 spaces and the $ denoting a command. We also use the heredoc syntax to pipe multiple lines to cat.

We’ll finally write a test that makes sure that wc works.

Count the lines:
  $ wc -l sample.txt

Note how we didn’t mention the expected output anywhere. This is where secret sauce comes in. We just run the test with dune:

$ dune runtest
 |  $ wc -l sample.txt
+|         3 sample.txt

And dune is helpful enough to fill in the output for us after promoting:

$ dune promote
$ dune runtest # now the tests pass

If we modify the wc utility to give a different result, the test will now fail because the command produced a different output. This style of testing is called expectation (or snapshot) testing. Here this style is dressed up in a shell like syntax to give us the cram test.

Dune 2.7 offers full support for this style, and we recomend it to all users. Do cram tests scale? In the dune project, this is our main testing mechanism and we have over 200 cram tests in our test suite. We use this approach to test and document both new features and regressions. So far we’ve been very satisfied with this approach, and we’re happy to share it with our users.

As usual, there’s far too much to describe in a single blog post. The rest is thoroughly documented in our manual

I look forward to answer any questions you might have about cram.

17 Likes

What would be the canonical way to express as a cram test a run of a binary that is expected to fail (i.e. non zero exit code), while also comparing the output (e.g. for testing error reporting of a compiler-like program).

Cram has a [ .. ] syntax for expecting a particular exit code:

  $ dune build cycle.exe
  Error: Dependency cycle detected between the following libraries:
     "a" in _build/default
  -> "b" in _build/default
  -> "c" in _build/default
  -> "a" in _build/default
  -> required by library "c" in _build/default
  -> required by executable cycle in dune:17
  [1]

(Example taken from Dune here.)

If the error code isn’t as expected, it’s shown in the diff:

 |Cycle detection
 |---------------
 |
 |  $ dune build cycle.exe
 |  Error: Dependency cycle detected between the following libraries:
 |     "a" in _build/default
 |  -> "b" in _build/default
 |  -> "c" in _build/default
 |  -> "a" in _build/default
 |  -> required by library "c" in _build/default
 |  -> required by executable cycle in dune:17
-|  [2]
+|  [1]
5 Likes

Great ! Thanks a lot.
It might be useful to link to the cram test format specification at one point in the dune manual, to help discover such tricks.