Sanity check: omake vs dune progress counters report different number of things to compile

Ecosystem
,
1

At the top-level of my project I run omake, which provides status of compilation:

[====     ]  00803 / 04960

(then a git clean -fx happens to eliminate all build artifacts)

When I do the same with dune build --profile=release, I get a higher number:

Done: 3580/6118 (jobs: 4)

Is dune doing more work, or is it simply accounting for number of things left to-do differently?

I’m using dune 2.0 in my project file, so it should only be building native. Additionally, the same set of .exe files are generated by both omake and dune (and I confirm no .bc files appear). The counts of generated .cmx files are the same as well, +/- 2.

If it matters, dune takes about 5% longer to finish building the whole tree than omake.

EDIT: I’m aware that the number of things to-do may increase as the build progresses, but even when they’re both at 99% done they’re still about a thousand total to-do items apart.

2

Some things to keep in mind:

  • Is your omake build using -bin-annot or -g? Those may slow down builds by using too much IO
  • I know omake builds often manually write -I paths. Make sure that the include paths are roughly equal between the two projects.
  • Are you sure that dune build is building what you think it should? See if `$ dune build results in the same number of jobs
  • Are you using ppx? Dune & omake handle ppx differently. In particular, dune does not use the -ppx flag and preprocesses as a separate step. This can result in a higher number of jobs but it should be in fact faster overall.

Dune should certainly not be slower than omake, so if you’re still unable to find the reason. Please make an issue.

3

Are you sure that dune build is building what you think it should?

Well, no. Kinda why I posted this :slight_smile:

So, after dune build --profile=release finishes I observe:

$ cat _build/log | grep ocamlc.opt | wc -l
923

That’s roughly the number of extra total to-dos that dune has!

It seems like I’m still building bytecode when the omake build does not. I confirm by noticing .cma and .cmo files in my dune build that the omake build does not produce.

What’s the way to completely, unquestionably, stop building bytecode in dune for the entire project? I have dune 2.0 in my dune-project file. I understand this stops .bc files from being built, but perhaps not the intermediate files. Any other things to tick off?

When I try this from the dune docs I get a syntax error:
(modes (exe object shared_object))

When I try this I also get a syntax error about atoms being expected:

(modes ((best exe)
        (best object)
        (best shared_object)))

When I change it to this:

(modes (best exe)
       (best object)
       (best shared_object)))

The total to-dos increase by a few hundred!

So confused.

4

Dune is a bit different from omake and my advice would be not to mess around with modes - at least in the beginning. Mainly because byte code builds are quire important for functional libraries: limited platforms, REPL support, jsoo. Moreoever, they are rarely the bottleneck in OCaml builds, and I’ve found that generating bytecode is generally quite fast.

I would recommend that you try to use a different target than @all (the default for dune build). If you just want a fast binary, then just build it directly: $ dune build src/foo.exe. This will not build any unnecessary bytecode. If you just want to check that your project “builds” in development, then there’s a $ dune build @check target. This will be much faster than both $ omake and $ dune builld.

5

Also note that dune uses ocamlc to compile mlis to cmis as a separate step, while omake may use ocamlopt.

6

Whoa. Okay. Hol’ up. That’s the command that prints money.

I’ve mostly been caring about whole tree rebuilds because my CI system takes forever to test build each new feature.

$ time omake -j 4
[...]
real	19m43.295s
user	65m52.760s
sys	7m38.632s

$ dune build @all
[...]
real 22m2.198s
user 70m24.540s
sys 10m2.404s

$ dune build @check
[...]
real	2m29.260s
user	7m3.660s
sys	1m22.184s

10xish faster! :star_struck:

Any limitations to be aware of?

1 Like
7

I assume that for CI building check will not be enough, since you probably want to run the tests. Check just makes sure everything type checks. In ci dune runtest should be slower, but it will still not build any unnecessary bytecode.

8

Right. It’s best to give people quick feedback that their feature won’t build or fails tests. If we can’t have that, second best is quick feedback that it won’t build and slower feedback that it fails tests.

Thanks for the help!

Would be nice to figure out why dune does more work than omake in our case. The project is using ppx; listing all of the .exe files in the dune build invocation didn’t change the story much.

But this is good enough for now.