It appears to me that
dune runtest -f doesn’t re-run custom expect-tests. I have a
tests/foo.expected, and a
tests/dune file with
(tests (names foo)), and
dune runtest -f doesn’t seem to run the test – I can tell either by the fact that running the test is supposed to take a long time and doesn’t, or if I modify the test to make some other side effect like writing to a file on disk. If I remove
dune runtest -f does always run the test, and of course it gets run whenever its dependencies change. Is there some different configuration I have to do to make this work, or some different option I can give? I’m using dune 2.9.0.
It appears to me that
This looks possibly like a bug, given the documentation of
-f. Maybe you should open an issue at https://github.com/ocaml/dune/issues? Browsing issues with “runtest force expect”, I find the open issue #4390: give a better meaning to force to be rather relevant.
This is not a bug, but may be an usability issue. The stanza
(tests (names foo)) gets translated roughly to
(executables (names foo)) (rule (with-stdout-to foo.output (run ./foo.exe))) (rule (alias runtest) (action (diff foo.expected foo.output)))
When you do
dune runtest -f you are forcing the rebuilding of the
runtest alias. This means re-running the
diff action which is promptly done. The
foo.output file, however, is not rebuilt because it has already been built.
If you want to have better control on when
foo.output is built, you need to use plain rules instead of the simplified
How would I do that?
And however I would do it, wouldn’t that be a better translation of
(tests (names foo)) than the current code that you quoted?
Not most of the time. The first task of a build system is to avoid rebuilding a file if it is already up-to-date. What is the point of re-running your test if you are going to obtain the same results that you already have?
Now, in some specific situations you may be inclined to do so (eg for side-effects or if your action is non-deterministic or …). In those cases you can write a rule like
(rule (deps (universe)) (action (with-stdout-to foo.output (...)))
Then the file
foo.output is going to be regenerated every time thanks to the
(universe) dependency. You can also attach an action to an alias directly, eg
(rule (alias foo) (action (...)))
dune build -f @foo will re-execute the action every time.
I’m not talking about “most of the time”; I’m specifically talking about
runtest -f. This is not the general behavior of a build system, but of a testing system. I think it’s reasonable to expect in that context that a command called
runtest will, well, run the tests, especially when passed a flag
-f that is documented to “force actions to be re-executed even if their dependencies haven’t changed”.
Even if we can’t agree on whether
runtest should generally re-run tests whose dependencies haven’t changed, shouldn’t it re-run them when the
-f flag is given, because the user explicitly specified that that’s what they want? And even less than that, shouldn’t it at least behave consistently, either re-running all tests or none of them, rather than re-running ordinary tests but not expect-tests? I can see how the current translation of
(tests (names foo)) leads to this confusing behavior, but I think that behavior is wrong and therefore the translation should be changed.
This is essentially what is suggested in Give a better meaning to `--force` · Issue #4390 · ocaml/dune · GitHub, but I guess no-one has stepped up to implement it yet.