Explorations on Package Management in Dune

Community
25

My view on this. When people talk about Opam, this covers different aspects.

  • the opam files and files repositories. The main repository is (for us OCamlers) ocaml/opam-repository, but the Coq community also has one, and many companies use their private repository. We do not want to break this workflow. The Dune package management proposals aim to stay fully compatible with this workflow and work with any opam packages (using dune or not). There’s also exciting ongoing work for package signing that we do want to see land at one point. We have also built and are operating an extensive CI infrastructure around these repositories – for instance, ocaml-repo-ci is building 100,000 jobs daily on all the Tier1 supported platforms for OCaml. We do not want to rebuild this once more time!

  • the opam client(s). The main one is the opam CLI, but many more tools use the opam files metadata. There are a few tentatives of generating nix derivations for those files. There’s also esy and the package managers that try to close the gap between the OCaml and JavaScript ecosystems. The client is built around a library (opam-lib), but this has never been designed properly. When I wrote opam initially, it was only focused on the CLI. Later with @AltGr, we tried to split it a bit more cleanly, but the API is still painful to use (for instance, every function that needs to load the filesystem needs a value that holds that state that would take dozens of arguments - then, as these functions perform file-system or network effects, you somehow need to keep these values synchronised with the new filesystem state – that’s painful and error prone). This API can somehow be split into various parts:

    1. Reading the opam repository state: parsing opam files, building a dependency graph (that’s the part you mention @gasche)
    2. Resolving constraints: opam has a pluggable interface for constraint solving, and by default, it will use what solver is installed on your system (or some internal heuristics which used to be very naive but seems much better nowadays). Opam needs to serialise and parse solver requests, including solver errors that need to be somehow pretty printed to the user.
    3. If the solver can devise a building plan, parse it and prepare it by downloading (and caching) the build/package sources.
    4. run the build commands for all the packages and install them locally.

Nowadays, most package managers also have the option to snapshot the state of the build plan between 2. and 3. That’s for instance, what tools like opam-monorepo lock or opam lock are doing.
But when you do opam install --locked, opam still calls the solver (to check that your lock file is consistent and complete), so doing 3-2-3-4. And opam-monorepo pull does 3. and delegate 4. to dune build (so all your dependencies need to use dune and be co-installable in a dune workspace).

So to come back to your question @gasche: The Dune package management experiment will be using opam-lib to do 1, 2 and 3 (whether it’s the current opam-lib or an improved version that relies a bit less on filesystem state is still in discussions - whatever the result is it will be upstreamed). And it will be using the dune scheduler to do 4 (but still using the opam build instructions).

2 Likes