[ANN] Testo 0.1.0 - a new testing framework for OCaml

Ecosystem
,
1

On this 86th anniversary of the first synthesis of LSD by Albert Hofmann, it is my pleasure to announce Testo, a new testing library for OCaml.

It borrows a lot of ideas from Alcotest and is similar in spirit but adds a few key features that seemed too difficult to incorporate into Alcotest. For a gentle introduction, check out our tutorial. Important features include:

  • support for many options when creating a test of type Testo.t;
  • capturing stdout or stderr output for comparison against the expected output aka snapshots;
  • reviewing and approving tests without re-running them;
  • support for nested categories while keeping the test suite as a flat list;
  • parallel execution using multiprocessing.

This is the first official release of Testo and its interface is likely to change in minor ways until we release version 1.0.0. We’ve been using it internally at Semgrep for about a year and it’s been working well for us.

Happy testing!

11 Likes
2

Hi @mjambon, thanks and congrats on the release!

I have a question for DJ Testo: What are her thoughts on inline snapshots tests? (It wasn’t clear to me from the tutorial whether Testo supports it).

Some heavy users of this kind of style in OCaml do report a joyful experience. The moonbit team has embraced it. Looks like the Deno guys are on it as well. There must be something in the water. Hopefully not LSD?!

2 Likes
3

I don’t have a firm opinion on inline snapshots because I haven’t used them and I haven’t felt a need for them yet. However, there’s a wide variety of OCaml projects out there and I think Testo should strive to accommodate them all. Here are a few considerations:

  • Snapshots can get quite large in which case I think it’s best to keep them separate from source code anyway.
  • Very small snapshots that take a line or two seem like a good use case. At the moment, Testo won’t update inline snapshots for you. Meanwhile, the closest solution is to use Testo.with_capture as illustrated in the tutorial.
  • Updating source code automatically requires access to the source code. This isn’t very different than requiring access to standalone snapshots so there may a be way to add this feature to Testo without complications. The key to making this work without depending on the build system is to run the test program outside of the temporary build workspace e.g. at the project root rather than inside of _build/ - and we’re already doing this.
  • I’m a bit worried about the reliance on a preprocessor of OCaml source code to keep track of the location of the snapshot strings. More generally, I’d rather minimize the amount of magic.

In conclusion, I don’t see a major concern here other than a desire or need to get it done.

4

The sort of equilibrium that ppx_expect drove me to is to have most expect tests written outside of the libraries, so they don’t feel like they are disrupting the actual source code. I’m now accustomed to working on projects with pretty large expect traces and still find it very pleasant and convenient. I value not having to select among too many options and keeping changes incremental during development.

The dune-promote workflow and its integration with editors is one of my favorite parts of Dune! In my opinion, a certain level of automation is a required ingredient — once you’ve gotten accustomed to it, I don’t think you can go back to adjusting snapshots manually. I hope these kinds of tools and workflows will continue to inspire folks to get on board!

It’s really nice to hear that you’re considering extending Testo’s scope to a wide range of OCaml projects. Thanks for doing this!

4 Likes