Forcing dune to run expect tests

Ecosystem
1

It appears to me that dune runtest -f doesn’t re-run custom expect-tests. I have a tests/foo.ml with tests/foo.expected, and a tests/dune file with (tests (names foo)), and dune runtest -f doesn’t seem to run the test – I can tell either by the fact that running the test is supposed to take a long time and doesn’t, or if I modify the test to make some other side effect like writing to a file on disk. If I remove foo.expected then dune runtest -f does always run the test, and of course it gets run whenever its dependencies change. Is there some different configuration I have to do to make this work, or some different option I can give? I’m using dune 2.9.0.

2

This looks possibly like a bug, given the documentation of -f. Maybe you should open an issue at https://github.com/ocaml/dune/issues? Browsing issues with “runtest force expect”, I find the open issue #4390: give a better meaning to force to be rather relevant.

3

This is not a bug, but may be an usability issue. The stanza (tests (names foo)) gets translated roughly to

(executables
 (names foo))

(rule
 (with-stdout-to foo.output (run ./foo.exe)))

(rule
 (alias runtest)
 (action (diff foo.expected foo.output)))

When you do dune runtest -f you are forcing the rebuilding of the runtest alias. This means re-running the diff action which is promptly done. The foo.output file, however, is not rebuilt because it has already been built.

If you want to have better control on when foo.output is built, you need to use plain rules instead of the simplified tests stanza.

Cheers,
Nicolas

4

How would I do that?

And however I would do it, wouldn’t that be a better translation of (tests (names foo)) than the current code that you quoted?

5

Not most of the time. The first task of a build system is to avoid rebuilding a file if it is already up-to-date. What is the point of re-running your test if you are going to obtain the same results that you already have?

Now, in some specific situations you may be inclined to do so (eg for side-effects or if your action is non-deterministic or …). In those cases you can write a rule like

(rule
 (deps (universe))
 (action (with-stdout-to foo.output (...)))

Then the file foo.output is going to be regenerated every time thanks to the (universe) dependency. You can also attach an action to an alias directly, eg

(rule
 (alias foo)
 (action (...)))

Then dune build -f @foo will re-execute the action every time.

Cheers,
Nicolas

6

I’m not talking about “most of the time”; I’m specifically talking about runtest -f. This is not the general behavior of a build system, but of a testing system. I think it’s reasonable to expect in that context that a command called runtest will, well, run the tests, especially when passed a flag -f that is documented to “force actions to be re-executed even if their dependencies haven’t changed”.

Even if we can’t agree on whether runtest should generally re-run tests whose dependencies haven’t changed, shouldn’t it re-run them when the -f flag is given, because the user explicitly specified that that’s what they want? And even less than that, shouldn’t it at least behave consistently, either re-running all tests or none of them, rather than re-running ordinary tests but not expect-tests? I can see how the current translation of (tests (names foo)) leads to this confusing behavior, but I think that behavior is wrong and therefore the translation should be changed.

1 Like
7

This is essentially what is suggested in Give a better meaning to `--force` · Issue #4390 · ocaml/dune · GitHub, but I guess no-one has stepped up to implement it yet.

Cheers,
Nicolas