Any interest in inline test alcotest backend?

I’ve written an inline test backend that uses alcotest as the test framework which I’m using locally for my own personal projects - I was wondering if there would be any interest in the community for publishing this on opam for others to use.

The main difference from ppx_inline_test is that the testing assertions are written in terms of Alcotest assertions, and the test results are shown using alcotests’ interface

Example

Add inline tests to your dune file, and preprocess using the test runner:

(library
 (name example)
 (inline_tests)
 (preprocess (pps ppx_inline_alcotest)))

Then, in OCaml, define tests using the same syntax as ppx_inline_test :

let double a = a + a

let%test "doubles argument" = Alcotest.(check int) "same int" (double 3) 6 
let%test "doubles argument" = Alcotest.(check int) "same int" (double 4) 8 

Then, to run the inline tests, simply run dune build runtest , and this will call out to Alcotest, grouping tests by the files.

Testing `example'.
This run has ID `BA802DB5-953C-47A7-ACF2-5783B9B33163'.

  [OK]          example/example.ml          0   doubles argument.
  [OK]          example/example.ml          1   doubles argument.
5 Likes

Can you briefly describe which are the benefits (for you) of using alcotest vs. ppx_inline_test stuff? It’s hard to decide if your thing us useful or not for me because, I never tried alcotest…

2 Likes

Hi @Gopiandcode,

I’ve been wanting a PPX to generate my Alcotest boilerplate for a while, and am very happy to see others thinking along the same lines! :tada: There’s an open issue about it, and I have an old prototype implementation of a similar PPX which I had hoped to finish and release “soon”. I think it’s definitely worthwhile releasing ppx_inline_alcotest as-is; I’m sure folks will find it useful!

Here are some general thoughts I had about the problem, which you may or may not find interesting :stuck_out_tongue:

When first working on the PPX, I became frustrated with the fixed 2-level structure that Alcotest imposes (i.e. each test suite must be a list of lists of tests, with no scope for more or less nesting). I’ve always found this restrictive, but trying to map the OCaml AST onto the Alcotest API really put it into sharp focus: OCaml syntax is much better at grouping than the Alcotest API will allow. Ideally, I’d like to be able to structure my tests into modules/files however I want and have it “just work”:

module%test Lorem = struct
  let%test "ipsum" = ()
  let%test "dolor" = ()
end

module%test Sit = struct
  module%test Amet = Some_other_compilation_unit

  let%test "adipiscing" = ()
end

We have some existing test suites written in this style (e.g. here), but the rigid Alcotest.test type makes things harder than they need to be.

Another limitation is that Alcotest doesn’t let the user attach source locations to test cases, so if a test fails then Alcotest isn’t able to point at it: I always have to search the code rather than having my editor jump straight to it. With a PPX, we can obviously attach the source location for free; the Alcotest runner just needs to support it.

This line of thinking led to this RFC for an improved Alcotest API, which is what my PPX prototype is now based on. The above example outputs this:

In addition to better grouping & location information, I decided on a few other requirements for such a PPX to be able to replace my own manual use of Alcotest:

  • playing well with multi-file test suites. In the Irmin project, we tend to split large test suites out over multiple files and pull them together in a single test binary per folder (e.g. here).

  • playing well with functors. If I define some tests inside a functor body & apply that functor to different arguments, the tests should run once for each set of arguments.

  • playing well with Lwt/Async test cases. I should be able to mix and match e.g. Alcotest_lwt and Alcotest test cases and have the generator do the right thing. Ideally, adding an Lwt test case to a previously non-Lwt suite wouldn’t require changing any of the existing tests.

As an aside, as mentioned in the ppx_alcotest issue, I think there are some other good candidates for other PPX features for the Alcotest API. At the very least, we should provide a deriver for “testables” and a set of test assertion functions that embed source locations automatically, but I think there’s also scope for things like writing test assertions as partial matches:

let%test "basic queue operations" =
  let q = Queue.(empty |> add 1 |> add 2) in
  let%check Some (1, q) = Queue.pop q in
  let%check Some (2, q) = Queue.pop q in
  let%check None = Queue.pop q in
  ()

(This would fix the limitation of not being able to bind new variables as part of a regular Alcotest.check.)

To conclude:

  1. ppx_inline_alcotest is definitely worth releasing.
  2. The Alcotest test registration API is likely to become more conducive to PPX in the future.
  3. I’m very interested in adding a ppx_alcotest package to the mirage/alcotest repo once (2) has happened, which I hope will handle test registration as well as some other things. You’re of course very welcome to contribute to this effort!
  4. If you have any thoughts / feedback on either the proposed Alcotest API, my prototype PPX implementation, or any other PPX ideas for Alcotest, then I’d love to hear them!

Thanks for the excellent library!

Cheers,
Craig

2 Likes

I’m probably not the best person to give a pitch for alcotest - I’m actually pretty new to it myself, and the main reason I made the ppx was so that I could try using it as a testing backend for my own projects with a slightly more ergonomic interface.

I think the Alcotest test registration API was the primary reason I tried to write a inline test backend in the first place as well - when I first tried out alcotest for some of my projects, I found the process of writing tests to be needlessly cumbersome with alcotest and this quickly lead me to abandon my testing.

Ideally, writing a test should be as easy as possible, but alcotest imposes a lot of extra boilerplate on top of just writing the logical test.

This is great to hear, and I look forward to it’s release - some more automation of the boilerplate in alcotest will make it a lot more approachable and ergonomic to use!

1 Like