When writing ppx rewriters, how do y'all go about _testing_ compile-errors?

Ecosystem
, , ,
1

Hi there; I’m twiddling around with my first(-ish) ppx_rewriter, and I’m trying to be a Good Boy and include thorough testing.

However, a good chunk of “writing a good ppx” seems to be about having good compile-errors (see, e.g. the ppxlib docs on Handling Errors.) Errors during compilation don’t produce successful tests, of course; so how do y’all go about writing tests for these?

I’m considering something like: 1. one .ml file per error, 2. excluding these from Dune, and 3. writing bats/cram tests that simply call the complier directly, I guess, and inlining the expected error-output of the compiler?

Is there a better way? I haven’t been able to find much in the way of common practices for this by browsing public repositories on GitHub, heh … :sweat_smile:

2

You can link the rewriter into a standalone driver (a program that applies the rewrite) and use that in expect tests (effectively cram tests). You can find an example repo here.

3

Kind of like CRAM tests, I use mdx and write tests that run in the toplevel. I like this approach b/c it does two other things:

  1. provides proof that it all works in the toplevel, not just with batch compilation
  2. produces documentation (I use mdx on .asciidoc files) for users.
4

can you explain a bit on how you do this?

5

Sure, here’s an example: https://github.com/camlp5/pa_ppx_regexp/blob/master/README.asciidoc

Look at the bottom of the file.

This is both the README and a unit-test for pa_ppx_regexp, which provides compact Perl-like regexp syntax for OCaml. So, uh … it might be pretty cryptic for non-Perl users. I’ve used mdx to write other tests, but (read below) not any “failing to compile” tests, until now. [again, read on for why; I was actually surprised I hadn’t done so, b/c I certainly wrote code to support it.]

Some comments and caveats:

  1. This is written using and for Camlp5-based PPX infrastructure. Since mdx supported this sort of thing before, I’m sure it’s possible to make it work with the standard PPX infrastructure. Indeed, I had to submit a (tiny) patch to MDX to make it work with Camlp5.

  2. I use Makefiles, so Dune users might have trouble figuring out how to duplicate; I can’t help there, b/c I don’t use dune at all. At toplevel, I run make test which in turn runs make README.asciidoc.TEST. That does the following:

    $ make README.asciidoc.TEST
env TOP=. ocamlfind camlp5-buildscripts/LAUNCH -- ocaml-mdx test -o README.asciidoc.corrected 
README.asciidoc
perl -p -i -e 's,.*: added to search path,: added to search path,' README.asciidoc.corrected
perl -p -i -e 's,.*: loaded,: loaded,' README.asciidoc.corrected
diff -Bwiu README.asciidoc.corrected README.asciidoc

    What’s going on here?

    • First, run mdx on README.asciidoc
    • If you look at the file, you’ll see that I load packages (typically the case, since all PPX rewriters are in packages), and those packages are installed at directories that differ from installation to installation; so we need to erase the pathnames that are printed-out by the loading process
    • then diff the output file (.corrected) against the original: if they’re identical, then the test passes

  3. Funny thing, I didn’t have any examples of compiler errors in my MDX-based tests. I could have sworn I did, b/c I certainly added the support for them. So hey presto, I added a couple of tests at the end of the above-cited file.

  4. Because Camlp5 produces errors a little differently than the OCaml parser, I had to insert a tiny bit of error-conversion code – you can see it here: https://github.com/camlp5/pa_ppx_regexp/blob/479fbaa67ba69850a8c95c12d062e4a71f72d9ee/README.asciidoc?plain=1#L91

Again, this should not be necessary with the standard PPX infrastructure.

OK, with those caveats, here are the two tests I just added. I’m quoting the tests from README.asciidoc, but even still, it’s going to get displayed wrong b/c both .asciidoc and Markdown use “three-backticks” for preformatted text, AND mdx uses it with either ocaml or sh after the three backticks, to select how to process the block (you can read more about that in the mdx documentation – it’s very straightforward).

I REPEAT: the code below is NOT an example to use: you need to go to the bottom of README.asciidoc to see the actual code, b/c the rendering process in this forum software is mangling the code.

== Examples of Errors

=== A match regexp with an unrecognized modifier

# [%match {|(a)?|} / foo];;
File "_none_", line 1, characters 19-22:
Failure: extract_options: malformed option: <<(MLast.ExLid (<loc>,
                                        (Pp_MLast.Ploc.VaVal "foo")))>>
Line 1, characters 0-0:
Error: Stdlib.Exit

=== A match regexp specifying more than one regexp syntax

# [%match {|(a)?|} / re_perl pcre];;
File "_none_", line 1, characters 0-32:
Failure: match extension: can specify at most one of <<re>>, <<pcre>>: re_perl pcre
Line 1, characters 0-0:
Error: Stdlib.Exit
6

thanks for the explanation.

7

We need to write a section about end-to-end testing of the PPXs in the ppxlib manual, but currently we don’t have time for that. If anyone wants to open a PR, that’s very appreciated!

In the meanwhile: We always link to the blog post that @emillon has linked to above because that’s the last complete and well-written resource we have. It’s a bit old though and in the meanwhile dune cram tests have made our lives way easier. You can have a look at the ppxlib tests themselves to see how we use dune cram tests to test the output of the PPX driver. E.g., talking about error reporting: https://github.com/ocaml-ppx/ppxlib/tree/main/test/driver/parse_error_locations. The only thing you need to do differently: You’ll need to link the PPX you’re testing into the driver, i.e. your dune file needs to be

(executable
 (name standalone)
 (libraries ppxlib <your_ppx>))

(cram
 (deps standalone.exe))

You can use that same test workflow then to test both the successful behavior of your PPX and its error reporting:

$ ./standalone.exe <file containing code whose rewriting you want to be tested>
1 Like