How to disable tests in dune

I have a number of expect tests in tests/ in my dune project. Now I would like to quickly disable a few of them to save some time, and possibly enable them again later. Basically, dune build should skip them. I couldn’t get this info out of the docs for some reason.

More generally, what do I have to do to tell dune to skip some rule when building the default target?

I tried to include a stanza in tests/dune which redefined default:

(alias 
	(name default)
	(deps test1))

where test1 is the name of the test that I want to keep, but that didn’t do it…

1 Like

What does appear to work is to do dune build @tests/default which indeed now skips the unwanted tests. However, the standard dune build still includes them, which I would like to stop.

Even simpler, how can I exclude all tests when calling dune build? I would assume that’s easy? I was not able to find out from the docs.

From my understanding, dune build means “build everything” and everything includes every test. If you only want to build some targets, you need to tell so to dune, with something like dune build src/main.exe.

I guess I would like more granularity: A simple way to exclude a subset of tests without having to fully specify the rest of the artefacts that should still be built. That seems reasonable, no?

It would be fine to have to move the disabled test to another directory, or to use a rule stanza instead of a test stanza for them. I also played with enabled_if fields but this did not enable me to find a solution…

Yes it does sound reasonable. I think one way to do it is to use packages : dune build -p my-package. Another would be a custom rule that depended on everything you want to build, and then you would build that rule.

When you use the test stanza, the test is automatically added as a dep of the runtest alias. What I do to have a longtests alias for tests that only need to be run more rarely is to write the executable and rule stanzas by hand. Example here: Allow `(alias ...)` in test(s) stanza to attach the test to a different alias, e.g. `longtests` · Issue #6152 · ocaml/dune · GitHub

Of course, that does not answer your question of excluding things from the default alias. I don’t know how to do that, and to me that does not seem like a great idea…

Thanks @bnguyenvanyen , I did try to run the long test as a plain executable under a different alias as you proposed. That works although it’s a bit wordy, but as you mentioned it is also included in the default build.

I think running long tests can be reasonable to profile the program during development of a particular feature, but for ‘normal’ builds the wait gets too long. so maybe they should not be called tests but benchmarks. they can’t function as a safeguard against regression but that’s not their purpose.

I’m thinking there must be a way to exclude some rules from the default alias by redefining the default alias in a subdirectory. i tried to do that in my tests/ subdirectory but wasn’t able to make it work.

I don’t use the default alias very much anymore, exactly for this kind of reason. I now tend to use directly the @check, @install or @runtest (or @longtests) aliases.

I think your problem is that up from the tests directory, the default alias is defined by default as

(alias
 (name default)
 (deps (alias_rec all)))

(see Command-Line Interface — Dune documentation)
So you would need to also redefine it there, but I don’t know how to do that so that it catches every target, except for the tests directory…

It seems to me the simpler solution is to use dune build @runtest or dune build tests/. with your redefinition of the tests default alias (I think that works).

Yes, that alias_rec in the default alias may be the reason. But I kind of like the practice of always calling dune build unless something special is happening – that’s similar to expecting make to build the project in the least surprising way when using Makefiles. If there really is no easy way to achieve that, probably it’s time for a feature request.

I think I have a workable solution now. I introduce a new dune profile named benchmark and run the test only under this profile, using the following stanzas in the tests/dune file

(executable
	(name long_test)
	(modules long_test)
	...)

(rule
	(action (with-stdout-to 
						./long_test.output 
						(run ./long_test.exe)))
	(enabled_if (= %{profile} benchmark)))
(rule
	(alias runtest)
	(action 
		(diff long_test.expected long_test.output))
	(enabled_if (= %{profile} benchmark)))

I can then run dune build which builds everything but does not run the test, or dune build --profile=benchmark to run the long test. However, no diff will be shown and dune promote is not supported afterwards. What seems to work best is to call dune test --profile=benchmark which will run the long test, display the diff and allow promotion. In addition it will also build all the short tests as far as I can tell.

[edit] I tried to merge the last two rules into one but this did not generate a persistent long_test.output on disk and also seemed to interfere with dune promote.

That’s a limitation of the current dune model, yes. It’s possible to redefine the default alias, but there’s currently no way to express “aliases defined in all subdirectories except this set”. This is a reasonable feature request.