Diagnosing and preventing slow builds (especially in Dune)

dune
#1

Aside from running dune's internal profiler, are there any other ways to get insight into which bits of an OCaml build are slowing down a particularly long build, and why they do? Anything from a list of things to avoid to reading material to automated tooling (or even pointers as to how to interpret the traces coming out of the dune profiler!) would be great :slight_smile: .

I currently have a medium-sized project compile that’s taking about 2.5min to build (using 4.07.1 ocamlopt), and it feels like it certainly shouldn’t be taking that long. Pointing chromium at the dune profile reveals that almost a minute of that is PPX rewrites (I’m currently pulling in ppx_derivers.std and ppx_jane), and the rest is almost entirely just a lot of small, half-parallelised runs on files.

There are a few files that seem to be taking longer (which looks like it’s directly proportionate to the amount of code involved, though the most egregious ones are ones that contain functors producing large modules with other modules included within), and a few instances where the build is sequentialised due to everything depending on one library whose build depends on one module (and, on the flip side, sometimes I get full parallelism of 4 jobs on 4 cores), but I can’t draw many obvious conclusions as to what I’m doing wrong/suboptimally.

EDIT: odoc builds on the same tree slow to a literal crawl, too, so I figure there’s something that’s genuinely nasty in there, but I’m not well-equipped to find out what.

EDIT 2: I have a sneaky suspicion the ‘genuinely nasty’ behaviour is a mountain of transitive module type include, which seems to be making odoc’s output for the affected types snowball to ludicrous (~300MB a file) proportions. Ouch.

1 Like
#2

@jonludlam has just hooked up odoc to some new continuous benchmarking infrastructure we’ve setup at http://bench.ocamllabs.io. We’d really appreciate a small-ish testcase of that pathological odoc build performance reported on https://github.com/ocaml/odoc/issues if you can extract out the pattern causing it.

The dune traces can be run through chrome profiler (see “dune profiling” section) and you can follow the slow command invocation through there.

1 Like
#3

Yeah, your build is indeed quite slow, sorry about that. I don’t think the slowness is related to dune, but as you already know, dune can help you find the bottleneck.

Some of your files are compiling absurdly slow. Compiling language_definitions.ml is taking 6 seconds on my laptop! In my experience, this is unheard of.

I think you’re on the right track that reducing all the includes will speed up your builds. A clue here is that your cmt files are extremely bloated (many are 5-10 MB). Another thing you can try is to use (implicit_transitive_deps false) option in your dune-project file. This will trim the size of your -I arguments. It should speed things a little bit, but I wouldn’t expect any miracles.

1 Like
#4

Thanks @rgrinberg, especially for giving the compile a go (and apologies for how ridiculous the build was…)!

I agree that this isn’t actually a dune problem—just figured that if I specifically mentioned that I was using dune that might help with finding where to go next. (TBH, I guess the best thing to do is to learn how to work out useful information from a dune trace; certainly it’s pointed out some of the more egregious compiles).

@avsm: I don’t really have a MWE of odoc's build blowing up yet, but I think the problem is quite literally just a huge accidental pyramid of module includes, things like

module type Bar = sig
  include Barbaz
  include Foobar
end

module type Baz = sig
  include Barbaz
  include Bazbaz
end

module type Foo = sig
  include Bar
  include Baz
end

which accumulated quite without me realising in my codebase. I’d say this is probably more a programmer mistake (or, at least, an issue of a programmer accidentally making the compiler’s life a misery) than a problem with odoc.

#5

Oddly, it seems that the amount of odoc churn isn’t directly proportional to how big the odoc output is (I don’t know much about how odoc works, but I guess it’s probably pulling in a lot more information about module dependencies than it outputs on HTML), and it seems to still be choking a lot on modules that don’t include many modules, but do reference the tips of some rather oversized modules, functors, and interfaces.

I get the feeling that this is what happens when I skimp on reading about how OCaml actually compiles modular code, and then proceed to produce something that looks elegant and does what I want but explodes magnificently when sent to the compiler/odoc. (I’m guilty of passing around extremely large modules full of included extension operators and functors and suchlike; I’m not sure whether moving the extensions onto extension functors that I apply only when I need them would make things better or worse. Argh!)

1 Like