[ANN] New release of craml, a tool to test command line applications

I am happy to announce the first release of craml, a toool for testing command-line applications.

To install it:

   opam install craml

The tool is already used by real-world OCaml v2 to test shell script examples and by merlin. Feedback is alway welcome, so feel free to open new issues on the tracker to discuss about new features.

Here is a copy of the README, with a more complete description of the project:

craml: a tool for testing command-line applications

CRAM is a is functional testing framework for command line applications. craml is freely inspired by the Python tool, which was itself based on Mercurial’s unified test format.

craml is released as a single binary (called craml) and supports the following syntax:

  • Lines beginning with two spaces, a dollar sign, and a space are commands and will be run in the shell.
  • Multi-lines commands end by \ and continue with two spaces and a > sign on the next line:
    $ <line1> \
    > <line2> \
    > <line3>
  • Lines beginning with two spaces are considered command output.
  • Command outputs can contains ellipsis: .... These will match any possible outputs (on zero, one or multiple lines).
  • Lines starting by <-- are command pre-conditions; they will determine the conditions where the command is run. Currently, only non-deterministic modes are supported as pre-conditions (see below).
  • Lines starting by --> are command post-conditions. Currently, only exit codes are supported as post-conditions (see below).
  • Anything else is a comment. It is not possible to put comments in the middle of an output as it is not clear what should be done to them when the output changes.

To run the tests described in a <file>, use craml <file>. This will run all the commands in sequence and will generated <file>.corrected if one of the output do not match the expected command outputs.

Non-deterministic Outputs

craml supports non-deterministic outputs:

<-- non-deterministic
  $ <command>

In that case, craml <file> will run the command but will not generate <file>.corrected if the new output differs from the one described in the file. Use craml --non-deterministic <file> to come back to the default behaviour.

Non-deterministic Commands

craml supports non-deterministic outputs:

<-- non-deterministic [skip]
  $ <command>

In that case, craml <file> will not run the command. Use craml --non-deterministic <file> to come back to the default behaviour.

Exit Codes

craml tests exit codes:

  $ <command>
--> exit 10

If <command> does not exit with code 10, then craml <file> will generate <file>.corrected with the right exit code. Note that @@ exit 0 will not be displayed.


Just a small request: I note that the README, which many users will never see on installing from opam, contains quite detailed explanations of the package, but the documentation, which one can browse online, contains none of this information. Might I make a plea for you to put all that information into the documentation for the package itself?

Where did you browse the documentation online?

Also note that an odig readme craml will do.

1 Like

A naive google brought me to:


1 Like

Only if I know to do that. Not everything has a readme, and often the readme isn’t relevant, so it isn’t one’s instinctive first place to go.

Sure that’s why I try to educate you and, as a side effect, readers of this forum.

In that case a readme is certainly not a good place for these things.

I do however think that opam users should now grow the habit of odig (changes|readme) the packages they install and complain upstream if that doesn’t work.

1 Like

odig is a quite reasonable thing to be using, but unless there was something that mentioned a changes or readme was prominently something to look at, I don’t think I would necessarily think to. I tend to think the things you expect users to read to learn how a package works be in the manual for the package. If people really want to put stuff that’s important to someone using rather than building the package into other files, perhaps they could get automatically combined in to the manual?

Hum indeed. That library is not supposed to be used at all so I’ll see if I can publish a simple manual page instead. Thanks for the feedback.

1 Like