opam2nix generates nix expressions from the opam OCaml package repository. It works similarly to bundix, node2nix, etc:
You run an (impure) command to resolve all transitive dependency versions using the current opam repository, generating a .nix file that locks down the exact package sources and versions. Then this file can be imported to provide buildInputs for building your ocaml project in nix.
What is nix and why would I care? Well, that’s a long story but the headline benefits of nix are:
reproducible builds (if it builds for me, it builds for you)
stateless (you don’t set up switches and then install packages, each expression specifies everything it needs, and anything you don’t have is fetched/built on demand)
language agnostic (takes care of non-ocaml dependencies)
It’s sadly not a shallow learning curve, but those benefits are hard to find elsewhere, so I obviously think it’s worthwhile. So if you use nix (or would like to), please give it a try and provide feedback. I’ll (slowly) start working on upstreaming it into nixpkgs.
This is exciting! I’m keen to try this for reproducible builds in particular.
Did you encounter any hardships with wrapping opam for this, or interpreting depexts? Please do file upstream issues if future versions can be simplified for this usecase.
Thanks, will do. Overall I was super happy with how reusable opam’s internals were, it took a while to get familiar with some of the more complex parts (formulas are way more powerful than I knew from using opam), but it could have been much harder.
The main difficulty I found was that anything tied to OpamState had to be reimplemented, because that assumes certain things which aren’t true in opam2nix (e.g. there’s no one lib directory, there’s one per dependency). I’ll detail the exact issues in github though
Guix does have a more pure bootstrap story, true. But both should be equivalent in terms of reproducibility. If you use the default binary cache, you’re sidestepping the bootstrap process anyway.
Very nice! However, the solving step can be quite slow. For example (deliberately choosing a hard-to-solve-for package ):
$ ./result/bin/opam2nix resolve --ocaml-version 4.07.0 datakit-ci
+ ./result/bin/opam2nix resolve --ocaml-version 4.07.0 datakit-ci
Fetching...
Importing opam-repository 8d9cac4f0a35a5da2366efb9aacc8a176df03592 into nix store...
Loading repository...
Loaded 14257 packages
Solving...
[ERROR] Sorry, resolution of the request timed out.
Try to specify a simpler request, use a different solver, or increase the allowed time by setting
OPAMSOLVERTIMEOUT to a bigger value (currently, it is set to 60.0 seconds).
Fatal error: exception OpamStd.OpamSys.Exit(60)
Since you’re just picking versions to run a single program (not trying to maintain an opam switch), you’ll probably find it faster to use the opam-0install solver library instead, e.g.
In general it seems pretty solid. It reuses as much of opam as possible, and in particular with most packages tending towards jbuilder everything works well with the findlib-style of installation rather than hardcoding paths. I use OCaml just for personal projects though, so I haven’t stress tested it on larger software.
Supporting git is certainly possible (nothing too tricky, just a little tedious to prefetch it then map that into nix’ fetchGit expressions), honestly I’ve just never encountered a (non-pinned) opam package distributed via git before.
Thanks, that looks very nice! I’ve definitely had a bunch of frustration with solve times, particularly when I didn’t get the external solver stuff working right.
You have some parity tests in there, are divergences from opam considered bugs or does it have known different results? Any edge cases where it makes a bad choice?
You have some parity tests in there, are divergences from opam considered bugs or does it have known different results?
It’s a bug if it finds a solution that opam rejects, or if it finds a solution that is strictly worse than another valid solution (i.e. some package’s version is worse and none is better).
But it can find a different solution to opam. Here’s a screenshot from the tests showing it making some different choices:
Opam will try to minimise a cost function across all packages, whereas 0install gives preference to the package you asked to install. In most cases there’s no trade-off needed and they both give the same solution.
Any edge cases where it makes a bad choice?
The newest version of jbuilder (jbuilder.transition) conflicts with dune 2, so you can either have the “best” jbuilder or the best dune. 0install will prioritise jbuilder if that comes first in the optimisation order, whereas opam tends to favour dune 2 in that case. But the package you’re trying to install is always the first priority, so it will never choose an older version of that just to get a better jbuilder.
):