Updating files in a test

Learning
1

Hello hello,

I have tests that will open some JSON in an examples/ folder and compare them with what they produce.

I use a dune test-rule that looks like this:

(test
 (name my_tests)
 (deps
  (source_tree examples))

I sometimes make changes that means I have to update all of the JSON files in examples/ folder. I thought that it’d be easy to just add an environment variable when I want the test to just overwrite the original file instead of reading it.

The problem is that it seems to 1) not have the rights to write files and 2) only has access to the files that were copied within the _build/.... path.

I guess I’m looking for a simple and clean alternative to make this work here, any ideas?

2

When dune copies the dependencies to the _build directory, they are made read-only. This is in particular so that it plays nice with the caching system. See Dune should document that files cannot be marked as writeable · Issue #5570 · ocaml/dune · GitHub for a pointer (there might be a more specific ticket).

I guess I’m looking for a simple and clean alternative to make this work here, any ideas?

The JSON files contain input and expected output parts, is that correct? An idiomatic way to deal with updating files with dune is to use promotion. The idea is that your test executable, instead of doing the comparison, should emit a “corrected” version (with inputs being the same and outputs being the actual outputs from the system under test).
Then you’d attach a rule that does diffing and promotion between the JSON file in tree and the JSON file produced by your test executable.

Single-file example:

(executable
 (name test_json_file))

(rule
 (with-stdout-to test.json.corrected
  (run ./test_json_file.exe %{dep:test.json})))

(rule
 (alias runtest)
 (action
  (diff test.json test.json.corrected)))

Once this is in place, dune runtest will display any diffs (if any - and exit with corresponding exit code), and dune promote will apply the diff back to the JSON file in your source tree.

(you need to format JSON consistently for this to work)

4 Likes
3

On that note, it seems the Yojson pretty-printer changed formatting between versions 1 and 2, so it may be worth finding a formatter with a stable output like maybe jq

4

If that’s an issue you can also depend on yojson >= 2.

5

I would also argue that depending on a particular pretty printing of JSON is a fragile test to begin with since for most cases the exact formatting of JSON shouldn’t matter (e.g. swapped keys in objects, whitespace changes). I’d rather suggest a structural comparison in such case.

6

Yes, my answer is in the context of:

I thought that it’d be easy to just add an environment variable when I want the test to just overwrite the original file instead of reading it.

But it might be preferrable to stick to either something that consumes an existing format (say, third party test vectors) or something that makes promotion easier (e.g. ppx_expect).

7

Mmm I see, I’m wondering how to integrate this with the alcotest framework cleanly now

8

I think that someone made an integration between alcotest and the promotion mechanism, it was on this forum.

2 Likes