An Emacs mode for Cram tests

Hi all! Recently I’ve been playing with tests, and have fallen in love with dune’s cram tests :heart:

As I wasn’t able to find any support for it elsewhere, I’ve written a small Emacs mode to highlight cram tests to make it a little easier to distinguish between comments, commands and outputs:

Here’s what it looks like:

There’s probably some things that I’m not doing properly w.r.t to Emacs conventions, but I’ve found it works well enough for my usecases.

Aside: Cram tests really are awesome! I think they’re the best method of writing tests that I’ve found in OCaml, and I’m only saddened that I didn’t try it earlier! The ease of creating new tests (simply a new file, no boilerplate needed), updating expected outputs (dune runtest --autopromote), enabling a result to be tested (simply make sure it has a conversion to string – no need to hassle around with deriving equality etc.) are leagues above the other approaches I’ve used so far.

My current approach to testing my projects - (so far, mainly SQL-backed applications), is to first write a runner binary that calls the appropriate library functions based on its arguments and prints the result (sometimes I generate this programmatically from the types of my APIs), and then write cram tests to document the different interactions between calls that I expect. Especially when interacting with Lwt, I’ve found my cram tests to be far more readable than the corresponding unit tests that I might write with Alcotest.

5 Likes

Do you know about janestreet/ppx_expect: Cram like framework for OCaml?

2 Likes

Thanks! I also like cram tests – I like that it incentivizes us to write small binaries for our libraries, with a decent user interface (because we are the first users and we care about the readability of the tests).

One related need I have that is not fulfilled right now, so I mention just in case, is a standalone tool to do the “easy” part of cram tests, namely run the tests in it (in the current directory and environment, unless specifid otherwise) and print the updated file (the whole file with current test outputs). This would be a very convenient way, for example, to include benchmarks in a README.md file:

## Benchmarks

  $ hyperfine ...
  (current benchmark output)

I thought of using the cram tool, that I think inspired dune in the first place, to do this, but in fact it was designed to be run in CI scripts and is basically unusable as a standalone general-purpose tool. (For example it can only run things in a separate temporary directory, and you have to do a complex dance to copy there the files that you need in your cram commands.)

Unless I am missing another option, there is a need here for a reasonably-designed command-line tool, and we could write it in OCaml just as well as any other language – it could be an occasion to get non-OCaml users to use an OCaml program. I think there is an opportunity, but I haven’t had the time to do any work in this direction, so I’m just floating the idea out here.

3 Likes

I think that GitHub - realworldocaml/mdx: Execute code blocks inside your documentation allows running cram tests in cram syntax, even though it is not very well advertised… (I discovered it by reading the code!)

$ ocaml-mdx test cram.t
$ cat cram.t.corrected 
Text

  $ echo bli > blu

text2


  $ cat blu
  bli

  $ cat bli
  cat: bli: No such file or directory
  [1]

  $ echo bluuu
  bluuu
2 Likes

Interesting! I didn’t know. In view of outreach outside the OCaml community, I would prefer a tool focused on the cram part without the ocaml-toplevel part (or a language-generic toplevel part? interesting but hard): do one thing and do it well. But it should certainly be possible for such a tool to share library code with mdx and/or dune.

One interesting aspect of mdx is that the input syntax is slightly different from cram, it looks like it only execute commands inside a sh code block, instead of any two-indented dollar sign. This gives a better markdown rendering that the default cram tests (but with standard cram syntax we can have them inside code blocks if we want), it opens the door to extensibility with other syntaxes (as in “ocaml toplevel”) and it looks just a bit more robust and readable than the standard indentation-based cram approach.

(It also supports inline configuration with <!-- MDX ... --> pragmas, but it is unclear which of the currently-supported pragmas would be necessary for a general-purpose shell-focused tool.)

1 Like

Actually, mdx supports three syntaxes: markdown, mli, and cram. It will try to detect the syntax used, or use the --syntax argument if given.
If the syntax is cram, I think it is the “regular” cram syntax.

I agree with your comment about having a tool focused on cram tests, independently of OCaml (or other languages), would be nice!

1 Like

Returning to the original topic of this thread, thanks for creating the Emacs cram-mode! Here is a slightly improved cram-mode that deals with indentation more correctly. (I wasted about half an hour trying to figure out why my cram tests weren’t working, until I realized that the Emacs indentation was inserting tab characters rather than spaces.)

4 Likes

Oh wow!! that’s super cool! I’m glad to hear that cram-mode was useful for you and it’s super cool to see it still being extended upon!

btw: if you’re okay with it, you should definitely add yourself to the list of authors in your gist!! That’s a super useful addition.

Haha, I added myself to the copyright line the first time but forgot the Authors line. Done now.

I also made another change: the regexp for auto-mode-alist should be "\\.t$" rather than "\\.t", otherwise it’ll match any file extension that starts with t (like .tex).

2 Likes