I would like to get the count of failures of a cram test with dune.
For instance, the following test script has 2 failing tests:
$ echo First
$ echo Second
$ echo last
$ echo one
Any clues on how to do that?
There’s currently no good way to do it.
Why do you need this information? If you can describe a convincing use case, the team might consider adding something to do this.
I am teaching an undergraduate course where students should write some OCaml code as part of their activities. I would like to use the testing mechanism provided by
dune runtest in order to see how well their code passes the expected tests I will be providing. This would be used to assign a grade to them.
It seems to be common among unit testing frameworks (like alcotest) to be able to give a summary of the results from the tests. So it would be a good idea to have this possibility with dune as well.
That makes sense. Unfortunately, I don’t see a way to integrate test counts into cram’s output since it’s already a well defined format that we cannot change. I see two ways forward:
The easy way is to add some sort of a
$DUNE_CRAM_SUMMARY variable that contains the summary. Then you would need to add
$ echo $DUNE_CRAM_SUMMARY to all your cram tests.
The hard way is to implement a structured output format for tests. Then, make dune understand the output and print a summary as well as the diff if it’s requested.