An opam package has a version and it reflects the version of its source code. This version does not change when its dependencies are updated and the source is compiled again, now using updated dependencies, leading to a different binary and different behaviour. The same would happen when using a different compiler or different flags.
I would argue that for end users this difference should matter but it seems that we don’t think about versions in this way. Is anybody else struggling with this problem? Are there any best practices to capture the version of an OCaml project?
If you’re thinking about the RPM model that a given source package should emit the same binary version, then the best way to model the version number of a package is the concatenation of all its dependent package names and versions. You could hash the result in order to have a non-monotonic version string that is suitable for creating RPMs from (upstream Fedora/OCaml adopts a similar approach)).
I would like to improve this though. The forthcoming Dune 1.10 will have the facility to store opam metadata inside the dune-project file and generate the opam files. Part of the reason motivating this feature was to reduce the amount of boilerplate in a dune/opam project, but the other reason I started it was to improve the RPM/Deb packaging situation.
When all OCaml project dependencies are in a duniverse. it should be possible to generate RPM/Deb files that have accurately upgraded monotonic version numbers for their individual binary packages. We would have to do this by fine-grained analysis of all the constituent dependencies of a given opam package, but of course dune has this information as part of the build process. My intention is to make this upstreamable – that is, we should be able to generate an Ubuntu PPA from the bulk of OCaml source code in opam, with accurately incrementing source epoch numbers such that OS package manager upgrades work.
There are a few other features awaiting implementation into dune/opam before this will really work end-to-end. @rjbou is working on integrating the depext plugin into opam 2.1, and I’ve got a spec I’m writing for adding that field into dune-project as well.
I’m very interested to improve RPM/Deb packaging because this seems to be a fundamental impedance mismatch so far: you either have as few RPM/Debs as possible and rely on Opam for package management (hence: subverting RPM/Debs), or you have to re-create what Opam already does at the RPM/Debs level. This is not a problem unique to OCaml but affects any language that has its own package manager - so I would expect Python and Ruby to suffer from this as well.
Would be delighted to get help on that The best way to start to contribute would be to to design the RPM templates for a duniverse repository (which could be the xapi project). From the RPM templates, we can work backwards to figure out what metadata we need from the dune build that is unique to this problem, such as the source epoch numbers for the constituent opam packages.
Ideally, I would like this approach to scale to every major open source package manager, and Deb/RPM are amongst the hardest here due to their exacting versioning requirements.
Building a Deb/RPM package is happening in a buildroot which is different from the final destination in the file system. I haven’ tried recently but does this work for opam/dune? Right now I’m hacking around the problem but it is not satisfying.
Thanks for the pointer. I am currently not using lock files but it would indeed fix the version. The version of the lockfile would describe the version of the product or user-facing RPM/DEB package.
You might be interested in the Nix package manager, where a transitive closure of a package dependencies is folded into a cryptographic hash value, so that you have a guarantee that if you depend on a package with hash X it will be always the same binary on any deployment. Nix packages support OCaml and we’ve been releasing to it for the last couple of years without a problem.
The Nix packages look like an ideal first candidate to autogenerate from a dune/opam combo. Are the packages there manually updated at the moment? I’m not familiar with nix at all, but it looks like we can generate the fine-grained dependency information to make every opam package accurately represented in the Nix pkg tree as soon as it’s released to opam.
Before we do this, we really need Nix support in ocaml-dockerfile in order to hook it up to our CI processes though. Contributions welcome there too…
I’m not an expert in the Nix infrastructure, but I can say that BAP packages are written very manually
That sounds like a great idea. One of the problems that we’re currently having, is that we release 64 opam packages as 1 nix package, therefore we loose modularity and granularity. Just because we don’t have time (or desire) to repeat our opam specifications (3,000 lines of specs per release) in Nix. So we came up with a poor man solution, that just builds all packages in one package. Having it all automated will be a great win for us.
I will look into it. But before I’m going to do some research, is it possible/feasible to do this translation. I feel that Nix is a stricter/stronger specification system. Therefore, inherently, not all configurations from opam are representable in Nix.
A small status update after some research. I’ve talked to people who are involved in NixOS and got some points. Preliminary, it looks like that it is doable, since this is what is done for Haskell’s Cabal/Hackage. However, what scares me is that the repo is 45 kLoC of Haskell
Another good find is that the work has been already started, and there is the opam2nix package as well as the opam2nix-package repository (which is not yet a part of nix distribution, as far as I understood). Both projects are in active development, so we should seek the contact with the authors. Maybe François @bobot could give us some insights on the status.
Concerning the docker support, we have a NixOS builder, which we can use as a source of inspiration for the ocaml-dockerfile support. Besides, @avsm what do you think we need to add to ocaml-dockerfile to enable support for Nix packages?
opan2nix is an awesome development done by @timbertson. We are using it for our continuous integration because it allows to easily test different versions of the compiler or opam libraries. People in nixpkgs started to discuss with @timbertson on using it but it didn’t go far.
PS: We currently use the opam v1 version because the opam v2 opam2nix does not work well for us with ocaml < 4.07.
Thanks very much for looking into this @ivg and @bobot! Just one clarification: rather than wrapping existing packages in nix (which is complex and adds multiple layers of indirection), I was thinking about this scheme:
build a duniverse of the set of opam packages you want to publish in nix.
Check that it all builds using dune build from the toplevel directory.
Now we add a build target to dune that will generate<name>.nix packages based on the fine-grained build information available to dune. These should not require opam to install at all, since nix replaces opam here as the coordination mechanism for ensuring that they are compiled and installed into the system.
Dune 1.10 has support for package metadata in dune-project, so the only missing metadata that nix would need here is the depext information, which I would also like to add into dune to aid configurator probing.
We primarily just need the runes to know how to drive Nix in order to generate the Dockerfiles:
I know nothing about Nix, so even an issue on there with instructions would be a good start towards getting Nix/OCaml containers being built regularly. It definitely seems like the next best target we should aim for in our CI.
It might not be obvious but I believe using duniverse (or an approach that compiles all packages without installing them) is essential: an opam install relies for the compilation of a package on the earlier installation of other packages: as it goes through the installation of packages, it interleaves compilation and installation. This does not work in a $BUILDROOT environment as used by Debian/RPM packages which strictly separates compilation and installation.
The best way to start to contribute would be to to design the RPM templates for a duniverse repository (which could be the xapi project). From the RPM templates, we can work backwards to figure out what metadata we need from the dune build that is unique to this problem, such as the source epoch numbers for the constituent opam packages.