Opam and Dune -j parameter

Ecosystem
1

The dune documentation recommends using -j in Opam files:

You should set the build: field of your .opam file as follows:

build: [
  ["dune" "subst"] {pinned}
  ["dune" "build" "-p" name "-j" jobs]
]

Is this still relevant? Dune by itself tries to find the optimal number of processes to use. Is there something that Opam knows that Dune doesn’t?

2

It allows it to be overridden in the individual builds by setting OPAMJOBS. However, since we don’t implement a jobserver protocol, setting it to anything other than 1 and doing parallel package builds results in the machine being overcommitted. (i.e. if you install 5 packages in parallel, then they will all spawn 5*NCPU jobs).

It’s worth opening an issue in ocaml/dune on this, since it could implement a jobserver protocol fairly easily, and then opam could pass on this information to both dune and make to reduce the overcommit.

1 Like
3

The whole story is still a bit unclear on what is best to me.

You have different kinds of parallelism:

  1. opam building independent packages in //
  2. The package itself using a // build system.

For example the performance impact of having four packages building in parallel each using 4 auto-discovered jobs on a 4 core machine is unclear to me.

So in some sense it might be better to letter opam determine that jobs number rather than let each individual packages auto-discover it.

4

Also I can’t find discussions about this again but note that it was never really clear in opam whether OPAMJOBS and that jobs variable was 1. or 2. but I think that nowadays it is defacto 2.

5

I see the point that opam knows more about the global workload and might pass on this information to dune. In particular, opam itself accepts a -j parameter but I don’t think -j 1 implies that installation on the dune level cannot use concurrency.

6

As I said there’s confusion about 1 and 2 in opam at the moment. See this discussion.

1 Like