Opam and Dune -j parameter

The dune documentation recommends using -j in Opam files:

You should set the build: field of your .opam file as follows:

build: [
  ["dune" "subst"] {pinned}
  ["dune" "build" "-p" name "-j" jobs]

Is this still relevant? Dune by itself tries to find the optimal number of processes to use. Is there something that Opam knows that Dune doesn’t?

It allows it to be overridden in the individual builds by setting OPAMJOBS. However, since we don’t implement a jobserver protocol, setting it to anything other than 1 and doing parallel package builds results in the machine being overcommitted. (i.e. if you install 5 packages in parallel, then they will all spawn 5*NCPU jobs).

It’s worth opening an issue in ocaml/dune on this, since it could implement a jobserver protocol fairly easily, and then opam could pass on this information to both dune and make to reduce the overcommit.

1 Like

The whole story is still a bit unclear on what is best to me.

You have different kinds of parallelism:

  1. opam building independent packages in //
  2. The package itself using a // build system.

For example the performance impact of having four packages building in parallel each using 4 auto-discovered jobs on a 4 core machine is unclear to me.

So in some sense it might be better to letter opam determine that jobs number rather than let each individual packages auto-discover it.

Also I can’t find discussions about this again but note that it was never really clear in opam whether OPAMJOBS and that jobs variable was 1. or 2. but I think that nowadays it is defacto 2.

I see the point that opam knows more about the global workload and might pass on this information to dune. In particular, opam itself accepts a -j parameter but I don’t think -j 1 implies that installation on the dune level cannot use concurrency.

As I said there’s confusion about 1 and 2 in opam at the moment. See this discussion.

1 Like