Questions around the test and executable stanzas in dune

Learning
,
1

I am getting the sense from the documentation of dune that a custom test and an executable are roughly the same things, however I am very puzzled by various differences which make it hard for me to do what I want. I have several questions but let me first describe the setting (this is organized in a somewhat bizarre way, I will tidy it up later):

myproject/
| lib/
  | dune
  | mylib/
    ...
  | test/
    | mytest.ml
    | example.json

In mytest.ml, I have a with_open_bin "example.json".

In myproject/lib/dune:

(subdir mylib
  (library
    (name mylib)
    ...))

(subdir test
  (test
    (name mytest)
    (libraries ... mylib)
    (deps "example.json")))

So far so good. dune runtest lib/test compiles and runs my test, correctly picking up example.json. However, I would like to pass command line arguments to my test. This requires me to identify the specific test and not just the directory, but I was only able to do that using dune exec lib/test/mytest.exe -- ARGS. This compiles and runs but when I do this, the file example.json is evidently not visible to the program anymore. So my questions are:

  1. How to ensure examples.json is in the right place when using exec?
  2. How to run an individual test with runtest? None of dune runtest lib/test/mytest.exe, dune runtest lib/test/mytest, dune runtest lib/test/mytest.ml and dune runtest mytest work.
  3. How to pass (non-constant) arguments to a test when using runtest?
1 Like
OCaml real pain points
2

I actually had this reply written out back when I saw this question. I apologise for not replying sooner.

It’s true that (tests) and (executables) are doing very similar things but the main difference to be aware of is with the intention.

  1. You are running the executable in your current working directory when doing dune exec -- lib/test/mytest.exe. Therefore it doesn’t see a file called examples.json but it could see one called lib/test/examples.json. You probably want to run this a bit differently by cding to the lib/test directory and trying dune exec -- mytest.exe there.

  2. Currently dune runtest doesn’t support running an individual test declared by a (tests) stanza, although it is able to run individual cram tests. This is a feature I am currently working on implementing however so expect to see it soon.

  3. dune runtest doesn’t know anything about running executables nor about passing arguments to them. It only knows about doing things like dune build @runtest, which means building any targets attached to the runtest alias, and running individual cram tests.

If you want to test a specific executable that you have with its arguments then I would recommend that you just use the executable(s). stanza instead. Then you can test specific argument invocations yourself with a rule:

(rule
 (alias runtest)
 (action
  (run mytest.exe --your --args foo)))

If you want to run this specific test, one thing you can do is attach it to a second alias by using the (aliases) field. Say (aliases runtest this-specific-test). That way you can do dune build @this-specific-test.

The user defined rule also takes a (deps) field where you would like want to put your dependencies for your test. Read more about it here: rule - Dune documentation

3 Likes
3

Thanks a lot for the detailed answer!

I eventually understood from experimenting that dune exec runs in the current directory. I think there are two issues I hit here:

  1. I wanted to run an individual test in a directory that has several. I realized exec was the only way, but then it breaks any dependency the test has on some files. This would be fixed by runtest support for individual modules or anything equivalent, great to know it is being worked on. Ultimately, this is not what exec is for.
  2. I wanted to have a test that can run as a test, but also be called manually with command line arguments to export its results in a certain way (it’s actually a benchmark). My impression from experimenting and what you’re saying is that this is not possible in the most straightforward way, there is no way of specifying the path that will work for both ways of running the test (from the project root). The solution I came up with was to split the shared behaviour in its own module and have two small wrappers that set the path differently (although I have yet to try if it works). The solution you’re suggesting of just making it an executable that reads its command line, and defining a wrapper rule as a test seems simpler.
    It would still be nice, though, if you could give command line arguments to arbitrary tests on the command line, while still running them as tests (that is, in the same directory as when doing runtest). For instance, Alcotest accepts some command line arguments.

Another question that came up when I was trying to make all this work is if there is a direct way to in fact run tests from specific directories. I wanted to do this because I eventually moved certain data files to a data/ directory, since I use them in several places. I then have to use relative paths to access them in the code, e.g. in lib/test/mytest.ml: let filename = "../../data/example.json". I don’t like this very much because it is easy to get wrong and because it is not robust to being moved around. You can avoid the relative paths in the dune file if you write deps %{project_root}/data/example.json but this doesn’t work in the code. It seems to me that the solution is again to define the test as an executable and have a wrapper rule that does chdir, it’d be great if there was something less wordy though. It also wasn’t that straightforward to find the correct form, what I ended up with is this (the tricky bit is the %{dep:, without it you have to write the full path to mytest.exe):

(rule
  (alias runtest)
  (deps %{project_root}/data/example.json)
  (action (chdir %{project_root}/data
    (run %{dep:mytest.exe}))))

(executable
  (name mytest))
4

That’s right. dune exec is meant to be like the usual unix exec command however with dune having built the specific binary you intend to run in the appropriate enviornment.

dune runtest initially started off as a command alias for dune build @runtest which built all targets attached to the runtest alias (and in any subdirectories). However later support was added for specifying paths to run tests in and then eventually specifying particular cram tests. This trend is continuing and the ability to run tests from (tests) stanzas and libraries with (inline_tests) will follow. The goal is to make dune runtest an easy way to point-and-click when it comes to tests.

The (tests) stanza is really just (executables) with the automatic adding of a rule that runs the executable to the runtest alias. So defining a “wrapper” rule is not too far off from what’s going on under the hood. There you of course have access to a lot more flexibility and you can run the test in whichever manner you please.

This is an interesting proposal and might be worth opening an issue about in the dune repo. The current dune runtest implementation essentially just finds the correct alias in order to run a test and doesn’t have that much control over how the test is run.

For context, every test produces an alias: cram tests foo.t produce and alias foo, (tests) stanzas produce runtest-name_of_test_exe and inline tests produce runtest-name_of_lib. There are a few other caveats when it comes to js_of_ocaml, but that’s not important here.

When you write a dune file and run dune, it loads the rules associated with each of the stanzas you defined and then adds the targets / actions to various aliases. It’s helpful to think of aliases as (mathematical) sets of targets. There are aliases to pick for the various test stanzas I outlined above.

So it’s not really dune runtest’s job at the moment to dictate how a test is run, you can think of it as a helpful utility to guide users to the correct alias to build. In order for dune runtest to modify how test rules are setup it would have to do a lot more than it is currently doing. (Though it’s not impossible).

It’s also not entirely clear to me how the design of such a future will end up looking like, so if you do open an issue, please try to include examples from other places that have a similar feature.

One way you might be able to make your job easier is instead of moving your test to where your data is, you move your data to where your test is.

We have a (copy-files) stanza in dune that can allow you to “copy” the data that you want for your test from your directory. That way you only have a single directory to make relative and as far as the other (test) stanzas are concerned, you have the data locally (relative to your tests).

So with this suggestion you should have something like:

(copy_files %{project_root}/data/example.json}

(rule
  (alias runtest)
  (deps example.json)
  (action
   (run mytest.exe)))

(executable
  (name mytest))
1 Like
5

Ah yes, the copy_files solution is indeed nicer, thank you. You don’t even need the rule stanza then, a regular test works fine.

I actually tried this (my actual test has several files) and was pleasantly surprised that the glob over files that need to be copied is not problematic and that the subdirectory does not need to exist:

(subdir data
  (copy_files %{project_root}/data/*.json))

(test
  (name mytest)
  (deps (glob_files data/*.json)))

I will think of opening an issue over test arguments. And also over the ability to specify a working directory for tests (which could be both as a command line option and a test stanza option). Although for my use case, the copy_files solution is fine really.