Counting failing cram tests with dune

Ecosystem
1

I would like to get the count of failures of a cram test with dune.

For instance, the following test script has 2 failing tests:

First test
  $ echo First
  first

Second test
  $ echo Second
  second

Third test
  $ echo last
  last
  $ echo one
  one

Any clues on how to do that?

1 Like
2

There’s currently no good way to do it.

Why do you need this information? If you can describe a convincing use case, the team might consider adding something to do this.

3

I am teaching an undergraduate course where students should write some OCaml code as part of their activities. I would like to use the testing mechanism provided by dune runtest in order to see how well their code passes the expected tests I will be providing. This would be used to assign a grade to them.

It seems to be common among unit testing frameworks (like alcotest) to be able to give a summary of the results from the tests. So it would be a good idea to have this possibility with dune as well.

1 Like
4

That makes sense. Unfortunately, I don’t see a way to integrate test counts into cram’s output since it’s already a well defined format that we cannot change. I see two ways forward:

  • The easy way is to add some sort of a $DUNE_CRAM_SUMMARY variable that contains the summary. Then you would need to add $ echo $DUNE_CRAM_SUMMARY to all your cram tests.

  • The hard way is to implement a structured output format for tests. Then, make dune understand the output and print a summary as well as the diff if it’s requested.