Diagnosing and preventing slow builds (especially in Dune)

Aside from running dune's internal profiler, are there any other ways to get insight into which bits of an OCaml build are slowing down a particularly long build, and why they do? Anything from a list of things to avoid to reading material to automated tooling (or even pointers as to how to interpret the traces coming out of the dune profiler!) would be great :slight_smile: .

I currently have a medium-sized project compile that’s taking about 2.5min to build (using 4.07.1 ocamlopt), and it feels like it certainly shouldn’t be taking that long. Pointing chromium at the dune profile reveals that almost a minute of that is PPX rewrites (I’m currently pulling in ppx_derivers.std and ppx_jane), and the rest is almost entirely just a lot of small, half-parallelised runs on files.

There are a few files that seem to be taking longer (which looks like it’s directly proportionate to the amount of code involved, though the most egregious ones are ones that contain functors producing large modules with other modules included within), and a few instances where the build is sequentialised due to everything depending on one library whose build depends on one module (and, on the flip side, sometimes I get full parallelism of 4 jobs on 4 cores), but I can’t draw many obvious conclusions as to what I’m doing wrong/suboptimally.

EDIT: odoc builds on the same tree slow to a literal crawl, too, so I figure there’s something that’s genuinely nasty in there, but I’m not well-equipped to find out what.

EDIT 2: I have a sneaky suspicion the ‘genuinely nasty’ behaviour is a mountain of transitive module type include, which seems to be making odoc’s output for the affected types snowball to ludicrous (~300MB a file) proportions. Ouch.

1 Like

@jonludlam has just hooked up odoc to some new continuous benchmarking infrastructure we’ve setup at http://bench.ocamllabs.io. We’d really appreciate a small-ish testcase of that pathological odoc build performance reported on https://github.com/ocaml/odoc/issues if you can extract out the pattern causing it.

The dune traces can be run through chrome profiler (see “dune profiling” section) and you can follow the slow command invocation through there.

1 Like

Yeah, your build is indeed quite slow, sorry about that. I don’t think the slowness is related to dune, but as you already know, dune can help you find the bottleneck.

Some of your files are compiling absurdly slow. Compiling language_definitions.ml is taking 6 seconds on my laptop! In my experience, this is unheard of.

I think you’re on the right track that reducing all the includes will speed up your builds. A clue here is that your cmt files are extremely bloated (many are 5-10 MB). Another thing you can try is to use (implicit_transitive_deps false) option in your dune-project file. This will trim the size of your -I arguments. It should speed things a little bit, but I wouldn’t expect any miracles.

1 Like

Thanks @rgrinberg, especially for giving the compile a go (and apologies for how ridiculous the build was…)!

I agree that this isn’t actually a dune problem—just figured that if I specifically mentioned that I was using dune that might help with finding where to go next. (TBH, I guess the best thing to do is to learn how to work out useful information from a dune trace; certainly it’s pointed out some of the more egregious compiles).

@avsm: I don’t really have a MWE of odoc's build blowing up yet, but I think the problem is quite literally just a huge accidental pyramid of module includes, things like

module type Bar = sig
  include Barbaz
  include Foobar
end

module type Baz = sig
  include Barbaz
  include Bazbaz
end

module type Foo = sig
  include Bar
  include Baz
end

which accumulated quite without me realising in my codebase. I’d say this is probably more a programmer mistake (or, at least, an issue of a programmer accidentally making the compiler’s life a misery) than a problem with odoc.

Oddly, it seems that the amount of odoc churn isn’t directly proportional to how big the odoc output is (I don’t know much about how odoc works, but I guess it’s probably pulling in a lot more information about module dependencies than it outputs on HTML), and it seems to still be choking a lot on modules that don’t include many modules, but do reference the tips of some rather oversized modules, functors, and interfaces.

I get the feeling that this is what happens when I skimp on reading about how OCaml actually compiles modular code, and then proceed to produce something that looks elegant and does what I want but explodes magnificently when sent to the compiler/odoc. (I’m guilty of passing around extremely large modules full of included extension operators and functors and suchlike; I’m not sure whether moving the extensions onto extension functors that I apply only when I need them would make things better or worse. Argh!)

1 Like

Just coming back to say thanks!, as discovering --trace-file=FILE is awesome. Helped me discover that there are ~3 15s sub processes running on build!

I realize that this is an old thread but since its already been resurrected I figured I would drop a note around using the @install build alias. @rgrinberg introduced me to these about a month ago: Command-line interface — dune documentation and using them can have a very noticeable impact on build times. The project referenced earlier doesn’t seem to build cleanly on my machine but using @install still produces a ~20% improvement for what part of the project actually builds.

$ rm -rf _build ; time dune build
Command exited with non-zero status 1
real    0m 35.45s
user    2m 20.92s
sys     0m 34.20s

$ rm -rf _build ; time dune build @install
Command exited with non-zero status 1
real    0m 29.64s
user    1m 52.00s
sys     0m 23.31s

Just a note that this “optimization” works simply because @install builds less targets than @all (the default target). If you know exactly which artifact you need inside the project, you may also just build it directly dune build ./foo/bar.exe. This will likely be even faster than building everything that is installable.