Hi everyone!
I’m happy to announce the release of Windtrap, a testing framework for OCaml that brings the different testing methods of the ecosystem into one library:
- Unit tests, with an Alcotest-like API.
- Expect and snapshot tests, with a companion PPX compatible with
ppx_expectandppx_inline_test, but usable without it. - Property tests as seen in QCheck, with a library of textbook laws (round trips, orders, associativity, and more).
- Stateful and differential tests, as seen in qcheck-stm and Monolith, on one domain or several.
- Test coverage (as with Bisect_ppx) and mutation testing (as with mutaml).
Every kind of test composes with the others into one suite and one output, and the library has no dependencies beyond OCaml (the optional PPX uses ppxlib).
Here’s a small suite that tests the standard library with a unit test, a property stated as a textbook law, and a stateful test of Stack against a list:
open Windtrap
(* A reference for Stack: a list, its top first. *)
module Model = struct
type t = int list ref
let create () = ref []
let push x m = m := x :: !m
let length m = List.length !m
let pop m =
match !m with
| [] -> raise Stack.Empty
| x :: rest ->
m := rest;
x
end
let stack = abstract "s" ~pp:(fun ppf m -> Testable.pp (list int) ppf !m)
let lists =
group "list"
[
test "rev reverses a list" (fun () ->
equal (list int) [ 3; 2; 1 ] (List.rev [ 1; 2; 3 ]));
prop "rev is an involution" (Gen.list Gen.int)
(Law.involutive (list int) List.rev);
]
let stacks =
group "stack"
[
stateful "behaves like a list"
[
command "create" (Gen.unit @-> makes stack) Model.create Stack.create;
command "push"
(Gen.int @-> stack ^-> returns unit)
Model.push Stack.push;
command "pop" (stack ^-> returns int) Model.pop Stack.pop;
command "length" (stack ^-> returns int) Model.length Stack.length;
];
]
let () = exit (run "stdlib" [ lists; stacks ])
We built Windtrap at Invariant as part of our work on making code written by agents more trustworthy. Agents replicate what they find in a repository, so a project full of strong tests gets more of them. Windtrap comes with an agent skill that guides agents towards the right kind of test, but we care just as much about humans, and put a lot of effort into making the API and output ergonomic and delightful.
You can read more about it in the announcement post. It walks through an example suite, what a failure looks like, coverage and mutation testing, and an experiment where we used Windtrap to test the OCaml standard library.
To install it (when the PR is merged on opam-repository):
opam install windtrap ppx_windtrap
The manual (written by agents with my guidance, I hope you’ll find the writing voice bearable, but if enough people complain I’ll look into rewriting it by hand) has one page per kind of test. Feedback is very welcome, here or on the issue tracker.
Happy hacking!
Acknowledgments
Windtrap builds on ideas and code from several OCaml projects:
- Alcotest by Thomas Gazagnaire: test structure and runner design.
- Craig Ferguson’s Alcotest PRs (#294, #247): API design, subcomponent diffing.
- QCheck2 by Simon Cruanes et al.: generator design and integrated shrinking.
- Monolith by François Pottier, and qcheck-stm by Jan Midtgaard et al.: stateful testing against a reference, on one domain or several.
- ppx_expect and ppx_inline_test by Jane Street: expect test paradigm and dune integration, and ppx_expect’s test corpus.
- Bisect_ppx by Anton Bachin et al.: coverage instrumentation and runtime.
- mutaml by Jan Midtgaard: mutation testing for OCaml as a PPX.
- mtime by Daniel Bünzli: the monotonic clock.
