Running rules against multiple input files for promotion testing

I currently have a directory full of a bunch of *.mltt files that I want to run promotion tests against. Currently I have to hard-code these:

(rule (with-stdout-to let.mltt.stdout.exp     (with-stdin-from let.mltt     (run mltt))))
(rule (with-stdout-to monoid.mltt.stdout.exp  (with-stdin-from monoid.mltt  (run mltt))))
(rule (with-stdout-to type.mltt.stdout.exp    (with-stdin-from type.mltt    (run mltt))))

(rule (alias runtest) (action (diff let.mltt.stdout     let.mltt.stdout.exp)))
(rule (alias runtest) (action (diff monoid.mltt.stdout  monoid.mltt.stdout.exp)))
(rule (alias runtest) (action (diff type.mltt.stdout    type.mltt.stdout.exp)))

This is pretty error prone, and frustrating to keep up to date. Is there a more systematic way of describing these rules based on globbing the test input files, or would I have to write a custom test harness for this?

I tried this before I really understood how dependencies and globbing worked:

; note: this does not work!

  (deps (:mltt (glob_files tests/*.mltt)))
    (with-stdout-to %{mltt}.stdout.exp
      (with-stdin-from %{mltt} (run ./mltt.exe)))))

  (alias runtest)
  (deps (:mltt (glob_files tests/*.mltt.stdout)))
    (diff %{mltt} %{mltt}.exp)))

By searching github I’ve seen that some people generate the dune file with OCaml, but this seems a bit like a bodge, and liable to break in the future?

Dynamically generated rules are not supported by Dune at the moment. There are two ways to achieve what you need though:

  1. Using a generated dune file: you write a rule to generate a dune file with the list of stanzas and then include it in your main dune file. The main downside is that the generated dune file needs to be committed for this to work. You typically use promotion to automate the handling of changes to the generated dune file:
(rule (with-stdout-to (run ...)))
(include ;; is committed
 (alias all)
 (action (diff
  1. Using OCaml syntax to dynamically generate a dune file (which you mentioned). This is similar to the previous case, except that it is a bit less modular (as Dune needs to re-generate the Dune file each time it runs). It is unlikely this will break in the near to medium term though.


1 Like

Thanks! It still feels a bit clunky, but I went with number 1.