Dune lock directories record the names of any system packages needed to build projects or their dependencies. We’ve recently changed how this works in the
interest of portability.
Background on depexts
in Opam
A system package, or external dependency, or depext
as I’ll refer to them from now on, is a non-Opam package which must be installed in order for some Opam package to be built or for code in a package to be executed at runtime. These packages must be installed by the system package manager, or by some other non-Opam means such as manually building and installing the package from source. Common types of depext
are build tools such as the pkg-config
command, often run to determine linker flags while building a package, or shared libraries such as libgtk
, which an OCaml project might link against to create GUIs.
Opam usually installs depexts
automatically. Opam knows how to invoke many different system package managers (such as apt
or pacman
), so when installing a package with depexts
Opam can run the commands appropriate to the current system to install the required packages using the system’s package manager. For this to work, Opam needs to know the name of the package within the package repository appropriate to the current system, and these names can vary from system to system. For example the pkg-config
command is in a package named simply pkg-config
in the apt
package manager on Ubuntu/Debian systems, whereas in the third-party homebrew
package manager on MacOS it’s in a package named pkgconf
. In order to determine the right package name for the current system, the package metadata for Opam packages with depexts
contains a list of all the different known package names along with the conditions under which that name is correct. Here is that list for the conf-pkg-config
Opam package:
depexts: [
["pkg-config"] {os-family = "debian" | os-family = "ubuntu"}
["pkgconf"] {os-distribution = "arch"}
["pkgconf-pkg-config"] {os-family = "fedora"}
["pkgconfig"] {os-distribution = "centos" & os-version <= "7"}
["pkgconf-pkg-config"] {os-distribution = "mageia"}
["pkgconfig"] {os-distribution = "rhel" & os-version <= "7"}
["pkgconfig"] {os-distribution = "ol" & os-version <= "7"}
["pkgconf"] {os-distribution = "alpine"}
["pkg-config"] {os-distribution = "nixos"}
["pkgconf"] {os = "macos" & os-distribution = "homebrew"}
["pkgconfig"] {os = "macos" & os-distribution = "macports"}
["pkgconf"] {os = "freebsd"}
["pkgconf-pkg-config"] {os-distribution = "rhel" & os-version >= "8"}
["pkgconf-pkg-config"] {os-distribution = "centos" & os-version >= "8"}
["pkgconf-pkg-config"] {os-distribution = "ol" & os-version >= "8"}
["system:pkgconf"] {os = "win32" & os-distribution = "cygwinports"}
["pkgconf"] {os-distribution = "cygwin"}
]
depexts
in Dune
Dune doesn’t install depexts
automatically as the Dune developers are a little nervous about running commands that would modify the global system state. This may change at some point, but for now Dune only provides support for listing the names of depexts
, leaving it up to the user to install them as they see fit.
The dune show depexts
command can be used to list the depexts
of a project. For that command to work the project must have a lock directory. Here’s an example of listing the depexts
of a project:
$ dune pkg lock
...
$ dune show depexts
libao
libffi
pkgconf
sdl2
I ran these commands on a Mac with homebrew installed, so the package names are from the homebrew package repo. Each package listed there is one of the depexts
of a package whose lockfile appears in the project’s lock directory. Let’s look at how this information is stored. Using pkg-config
as an example:
$ cat dune.lock/conf-pkg-config.pkg
(version 4)
(build
(run pkgconf --version))
(depexts pkgconf)
The relevant part for us is the depexts
field. The current released version of Dune only stores the package’s depexts
for the system where dune pkg lock
was run. The command dune show depexts
simply concatenates the depexts
fields from each lockfile in the lock directory.
When thinking about portable lock directories I always like to imagine what the experience would be using Dune for a project where the lock directory is checked into version control. I frequently switch between using two different machines for development - one running Linux and the other running MacOS. If I was to check in the lock directory I just generated on my Mac, and then check it out on Linux and continue development, dune show depexts
would show me a list of packages for the wrong system!
Portable depexts
in Dune
To make depexts
portable, one’s first instinct might be to use the same approach as taken with the depends
field outlined here, listing the depexts
for each platform for which the solver was run. Indeed such a change was added to the Dune Developer Preview when we first introduced portable lock directories, however we quickly realized a problem.
The depends
, build
, and install
fields of a package rarely vary between OS distribution. It’s reasonably common for those fields to be different on different OSes, but very rare for them to also be different on different OS distributions. As such, it’s expected that users will elect to solve their projects for each common OS, but there would be little value in solving projects for each OS distro. In fact solving for multiple distros would slow down solving and bloat the lock directory, and users would somehow need to come up with a definitive list of distros to solve for.
But the depexts
field is highly-dependent on the OS distro since package names are specific to the package repository for a particular distro. Recall that the depexts
field in Opam package metadata lists package names along with the conditions under which that package name should be used, e.g.:
["pkg-config"] {os-family = "debian" | os-family = "ubuntu"}
["pkgconf"] {os-distribution = "arch"}
["pkgconf-pkg-config"] {os-family = "fedora"}
["pkgconfig"] {os-distribution = "centos" & os-version <= "7"}
These conditions almost always involve the name of the OS distro, and to make matters worse they also sometimes involve the OS version, as packages can change their names between different versions of the same OS. Evaluating these conditions at solve time for platforms with no distro or version specified tends to result in lockfiles with no depexts
at all, since all the conditions evaluate to false
.
The use case we have in mind for depexts
in Dune is that a user will solve their project coarsely, usually just for each common OS with no consideration for distribution or version. Then when they run dune show depexts
, the depexts
will be listed using names appropriate to the current machine. This means Dune needs to store enough metadata about depexts
to compute system-specific depext
names at a later time. This means storing the same names and conditions as are currently stored in Opam files, and deferring evaluation of the conditions until as late as possible, such as right when dune show depexts
is run.
The latest version of the Dune Developer Preview does just this; translating the depexts
field from each package’s Opam file into a Dune-friendly S-expression. After this change, the depexts
field of conf-pkg-config
’s lockfile is:
$ cat dune.lock/conf-pkg-config.4.pkg
...
(depexts
((pkg-config)
(or_absorb_undefined_var
(= %{os_family} debian)
(= %{os_family} ubuntu)))
((pkgconf)
(= %{os_distribution} arch))
((pkgconf-pkg-config)
(= %{os_family} fedora))
((pkgconfig)
(and_absorb_undefined_var
(= %{os_distribution} centos)
(<= %{os_version} 7)))
((pkgconf-pkg-config)
(= %{os_distribution} mageia))
((pkgconfig)
(and_absorb_undefined_var
(= %{os_distribution} rhel)
(<= %{os_version} 7)))
((pkgconfig)
(and_absorb_undefined_var
(= %{os_distribution} ol)
(<= %{os_version} 7)))
((pkgconf)
(= %{os_distribution} alpine))
((pkg-config)
(= %{os_distribution} nixos))
((pkgconf)
(and_absorb_undefined_var
(= %{os} macos)
(= %{os_distribution} homebrew)))
((pkgconfig)
(and_absorb_undefined_var
(= %{os} macos)
(= %{os_distribution} macports)))
((pkgconf)
(= %{os} freebsd))
((pkgconf-pkg-config)
(and_absorb_undefined_var
(= %{os_distribution} rhel)
(>= %{os_version} 8)))
((pkgconf-pkg-config)
(and_absorb_undefined_var
(= %{os_distribution} centos)
(>= %{os_version} 8)))
((pkgconf-pkg-config)
(and_absorb_undefined_var
(= %{os_distribution} ol)
(>= %{os_version} 8)))
((system:pkgconf)
(and_absorb_undefined_var
(= %{os} win32)
(= %{os_distribution} cygwinports)))
((pkgconf)
(= %{os_distribution} cygwin)))
That’s a 1:1 translation of the depexts
field from conf-pkg-config
’s Opam file. There’s enough information there so that the appropriate package name can be computed on demand rather than just at solve time.
This bring us a step closer to a world where Dune users can check their lock directories into version control with confidence that their builds are reproducible across different platforms. To try out the latest version of the Dune Developer Preview, go to preview.dune.build.