[ANN] opam-0install 0.1 - fast opam solver for CI, etc

Community
,
1

I’ve just released opam-0install 0.1, a faster solver for opam dependencies.

Opam’s default solver is designed to maintain a set of packages over time, minimising disruption when installing new programs and finding a compromise solution across all packages (e.g. avoiding upgrading some library to prevent uninstalling another program).

In many situations (e.g. a CI system building in a clean environment, a project-local opam root, or a duniverse build) this is not necessary, and we can get a solution much faster by using a different algorithm.

opam-0install does that by using 0install’s (pure OCaml) solver with opam packages. It is currently being used in several places:

Running make test runs some solves using the default opam solver and the new 0install one, and compares the results. On my machine, I get:

I’ve only shown the first few tests, which are somewhat cherry-picked. After that, it starts picking packages at random, and the results aren’t quite so biased :wink:

When 0install and opam find different solutions, it should always be the case that 0install’s solution is better for at least one component (but it could be worse for several). Surprisingly, it seems that the reverse is not true. In the screenshot above, opam’s solution for irmin.0.10.0 appears to be strictly worse than 0install’s choice.

The most common reason for finding different solutions is to do with jbuilder. The “best” version of jbuilder is jbuilder.transition, which depends on dune 1 (and conflicts with dune 2). 0install will often pick this “best” version of jbuilder, which then forces dune < 2 and older versions of other packages. You can avoid this in your own packages by listing your dune dependency before whatever depends on jbuilder, since 0install optimises components in order. Of course, if you require a package that needs dune 2 then that will also force the issue.

It’s also possible that I’m not using the opam API quite correctly in the tests. When I use a version of opam-repository since OCaml 4.10.0 release, opam likes to choose ocaml-system.4.08.1, a package that isn’t even available. I guess I’m missing a call to some init function somewhere.

7 Likes
2

Awesome, thanks! Eager to test this :slight_smile:

In the screenshot above, opam’s solution for irmin.0.10.0 appears to be strictly worse than 0install’s choice.

This would be either a bug, or more probably a misconception in the criteria given to the solver: the SMT returns a provable optimum w.r.t. the criteria (when it returns :wink: ).

One criterion that is disputable is the one that provides the “freshness” of the solution: there is no canonical way to define that, so what we do is define the “age” of a release as the number of newer releases, and minimise the sum of ages (opam calls it “version-lag”) ; that may bias towards packages with frequent releases though.
Here 0install’s solution has many more packages to install. I am guessing that these are not at their latest version, so adding them resulted in a worse version-lag overall. Indeed that is not really expected (there is already a specific criterion to minimise new installations, which has a lower priority); we might want to adjust the criteria to avoid this, but that would require more thinking.

Another note, it would be nice if this could be plugged in opam’s solver API, rather than going through the CLI… but maybe using the Cudf standard would be too constraining here ?

I had a failure compiling (not using --with-test) ; is a 0install installation required ?

% opam install opam-0install
[...]
# File "src/tests/dune", line 3, characters 10-23:
# 3 |   (deps %{bin:0install})
#               ^^^^^^^^^^^^^
# Error: Program 0install not found in the tree or in PATH
#  (context: default)

EDIT: solved by upgrading dune from 2.4.0 to 2.5.1

1 Like
3

Thanks :slight_smile:

Another note, it would be nice if this could be plugged in opam’s solver API, rather than going through the CLI… but maybe using the Cudf standard would be too constraining here ?

@kit-ty-kate made a CUDF version for opam-health-check. I hope that branch can be merged in at some point as another option.

I had a failure compiling (not using --with-test ) ; is a 0install installation required ?

Shouldn’t be. That error looks like it’s coming from the main 0install package, which shouldn’t be involved here. 0install-solver is the only thing that should be needed, and shouldn’t be running those tests (even with -t).

It builds for me in a clean Docker container:

$ docker run --rm -it ocurrent/opam 
opam@5e58ce0bf59d:~$ cd opam-repository && git pull && opam update && opam depext -i opam-0install
opam@5e58ce0bf59d:~/opam-repository$ opam-0install utop
[NOTE] Opam library initialised in 2.72 s
base-bigarray.base base-bytes.base base-threads.base base-unix.base camomile.1.0.2 charInfo_width.1.1.0 conf-m4.1 cppo.1.6.6 dune.2.5.1 dune-configurator.2.5.1 dune-private-libs.2.5.1 lambda-term.3.0.1 lwt.5.3.0 lwt_log.1.1.1 lwt_react.1.1.3 mew.0.1.0 mew_vi.0.3.0 mmap.1.1.0 ocaml.4.10.0 ocaml-base-compiler.4.10.0 ocaml-config.1 ocamlbuild.0.14.0 ocamlfind.1.8.1 ocplib-endian.1.1 react.1.2.1 result.1.5 seq.base topkg.1.0.1 trie.1.0.0 utop.2.5.0 zed.3.0.1
[NOTE] Solve took 0.31 s
1 Like
4

opam-0install-cudf is now released, with @kit-ty-kate’s CUDF adaptor.

2 Likes