Limiting tests executed during package build when generating opam files with dune

Learning
1

Is there a way to limit which tests get executed by default by dune? Especially in case when generating opam files.

Basically my use case is that, in letters, I have some tests that I would like to execute always, while some tests I would like to skip and expect to be executed manually. I don’t want CI to run some of the tests, because they need non-released version of a dependency and depend on an availability of an external service (I’m using ethereal.email to test that I can successfully send some emails).

I would prefer dune to generate opam files, but I don’t see a way to limit which tests gets executed when doing the release. Instead of having:

build: [
  ["dune" "subst"] {pinned}
  [
    "dune"
    "build"
    "-p"
    name
    "-j"
    jobs
    "@install"
    "@runtest" {with-test}
    "@doc" {with-doc}
  ]
]

I would like to have something like:

build: [
  ["dune" "subst"] {pinned}
  [
    "dune"
    "build"
    "-p"
    name
    "-j"
    jobs
    "@install"
    "@test/runtest" {with-test}
    "@doc" {with-doc}
  ]
]

The difference is with the to have @test/runtest instead of @runtest. I haven’t been able to find anything from the documentation that would allow me to change this part of the generated opam file.

Thanks.

2 Likes
2

You can use Dune’s .opam.template feature to achieve this:

  • create a letters.opam.template file that contains just the build field that you want to specify;
  • run dune build @install.

Any fields in the template file will be inserted verbatim into the corresponding opam file, overriding Dune’s auto-generation for those fields.

3

I’d recommend that your default runtest stanza remains as the set of core tests that you always want to run. You can define a custom alias (e.g. longtests) for the ones that require a specific environment.

The reason for this is that if you compositionally embed letters inside a larger monorepo, a toplevel dune runtest will still do a sensible thing. This is a convention and not a rule, but a useful one, as running tests from within a monorepo is an extremely quick way to iterate on large changes that span multiple libraries.

5 Likes
4

@avsm, what you described sounds like what I would actually want to achieve, but it is not clear to me how to get there.

If I have folders test and service-test with dune files declaring them as tests, how could I prevent execution of service-test when running dune runtest? Should I use a custom stanza in the dune file of service-test. Does that mean that instead of having (test ...) I should use my custom stanza? Do you have any pointers to good examples?

Thanks.

2 Likes
5

For tests that aren’t expect tests, the (test) stanza is sugar for:

  • setting up an executable
  • adding a rule to run the executable with alias runtest.

To create a test that runs with dune build @runtest-all (and not dune build @runtest), you can set up these stanzas:

(executable
 (name main)
 (libraries alcotest))

(rule
 (alias runtest-all)
 (action (run ./main.exe)))

Your tests in test/ would then want two rule stanzas, one for each alias that you create for them.

6

Anil, I’ve noticed that some packages have tests as part of their build that take … inordinately long. E.g. elpi. Is there some sort of rule for how long tests can run? Maybe it might be good to provide a worked-example of how to separate tests into “for the install” and “for developers when building/modifying” ? Just a thought.

7

It took me a while to test and make sure everything works as expected also through the release process.

Now I can keep my service tests in the codebase and simply expect them to be executed manually.

Thanks for the advise.

1 Like
8

Kakadu@ pointed out that for both deps and build rules, you can specify “with-test” and it’ll only be execued when you do “opam install -t package-name” There are many examples in opam-repository of this idiom.

9

There seems to be something missing in dune here. If you use a custom alias like longtests you will have plenty of issues with dune’s other test definition stanzas that require runtest. For example, there’s no way to run inline tests with longtests.

Another workaround is to add (enabled_if (= %{profile} dev)) for some of these tests. This will make sure that they don’t run in release mode.

1 Like