Aside from running dune's internal profiler, are there any other ways to get insight into which bits of an OCaml build are slowing down a particularly long build, and why they do? Anything from a list of things to avoid to reading material to automated tooling (or even pointers as to how to interpret the traces coming out of the dune profiler!) would be great .
I currently have a medium-sized project compile thatās taking about 2.5min to build (using 4.07.1 ocamlopt), and it feels like it certainly shouldnāt be taking that long. Pointing chromium at the dune profile reveals that almost a minute of that is PPX rewrites (Iām currently pulling in ppx_derivers.std and ppx_jane), and the rest is almost entirely just a lot of small, half-parallelised runs on files.
There are a few files that seem to be taking longer (which looks like itās directly proportionate to the amount of code involved, though the most egregious ones are ones that contain functors producing large modules with other modules included within), and a few instances where the build is sequentialised due to everything depending on one library whose build depends on one module (and, on the flip side, sometimes I get full parallelism of 4 jobs on 4 cores), but I canāt draw many obvious conclusions as to what Iām doing wrong/suboptimally.
EDIT: odoc builds on the same tree slow to a literal crawl, too, so I figure thereās something thatās genuinely nasty in there, but Iām not well-equipped to find out what.
EDIT 2: I have a sneaky suspicion the āgenuinely nastyā behaviour is a mountain of transitive module type include, which seems to be making odocās output for the affected types snowball to ludicrous (~300MB a file) proportions. Ouch.
@jonludlam has just hooked up odoc to some new continuous benchmarking infrastructure weāve setup at http://bench.ocamllabs.io. Weād really appreciate a small-ish testcase of that pathological odoc build performance reported on https://github.com/ocaml/odoc/issues if you can extract out the pattern causing it.
The dune traces can be run through chrome profiler (see ādune profilingā section) and you can follow the slow command invocation through there.
Yeah, your build is indeed quite slow, sorry about that. I donāt think the slowness is related to dune, but as you already know, dune can help you find the bottleneck.
Some of your files are compiling absurdly slow. Compiling language_definitions.ml is taking 6 seconds on my laptop! In my experience, this is unheard of.
I think youāre on the right track that reducing all the includes will speed up your builds. A clue here is that your cmt files are extremely bloated (many are 5-10 MB). Another thing you can try is to use (implicit_transitive_deps false) option in your dune-project file. This will trim the size of your -I arguments. It should speed things a little bit, but I wouldnāt expect any miracles.
Thanks @rgrinberg, especially for giving the compile a go (and apologies for how ridiculous the build wasā¦)!
I agree that this isnāt actually a dune problemājust figured that if I specifically mentioned that I was using dune that might help with finding where to go next. (TBH, I guess the best thing to do is to learn how to work out useful information from a dune trace; certainly itās pointed out some of the more egregious compiles).
@avsm: I donāt really have a MWE of odoc's build blowing up yet, but I think the problem is quite literally just a huge accidental pyramid of module includes, things like
module type Bar = sig
include Barbaz
include Foobar
end
module type Baz = sig
include Barbaz
include Bazbaz
end
module type Foo = sig
include Bar
include Baz
end
which accumulated quite without me realising in my codebase. Iād say this is probably more a programmer mistake (or, at least, an issue of a programmer accidentally making the compilerās life a misery) than a problem with odoc.
Oddly, it seems that the amount of odoc churn isnāt directly proportional to how big the odoc output is (I donāt know much about how odoc works, but I guess itās probably pulling in a lot more information about module dependencies than it outputs on HTML), and it seems to still be choking a lot on modules that donāt include many modules, but do reference the tips of some rather oversized modules, functors, and interfaces.
I get the feeling that this is what happens when I skimp on reading about how OCaml actually compiles modular code, and then proceed to produce something that looks elegant and does what I want but explodes magnificently when sent to the compiler/odoc. (Iām guilty of passing around extremely large modules full of included extension operators and functors and suchlike; Iām not sure whether moving the extensions onto extension functors that I apply only when I need them would make things better or worse. Argh!)
I realize that this is an old thread but since its already been resurrected I figured I would drop a note around using the @install build alias. @rgrinberg introduced me to these about a month ago: Command-line interface ā dune documentation and using them can have a very noticeable impact on build times. The project referenced earlier doesnāt seem to build cleanly on my machine but using @install still produces a ~20% improvement for what part of the project actually builds.
$ rm -rf _build ; time dune build
Command exited with non-zero status 1
real 0m 35.45s
user 2m 20.92s
sys 0m 34.20s
$ rm -rf _build ; time dune build @install
Command exited with non-zero status 1
real 0m 29.64s
user 1m 52.00s
sys 0m 23.31s
Just a note that this āoptimizationā works simply because @install builds less targets than @all (the default target). If you know exactly which artifact you need inside the project, you may also just build it directly dune build ./foo/bar.exe. This will likely be even faster than building everything that is installable.
.