I quite like the opam-dockerfiles because they allow me to use them as builder image for a number of systems with OPAM and a switch preinstalled so I don’t have to rebuild the world. But they haven’t been updated in a while, so OCaml 4.06.1 does not exist there at all and the Debian 9 image is apparently old enough that the packages from security.debian.org 404 and break builds.
I don’t know who to reach out to, therefore I am posting here kindly asking for these to be updated. I’d be happy to help with things if people would point me to whatever I can do to make the update happen.
Note that until opam 2.0 is released, this infrastructure is considered to be in a beta stage and subject to change.
So… we have opam 1.2 which is apparently unsupported now and opam 2.0 which is currently pre-release. Just one of those things I love about the OCaml ecosystem.
Thanks @samoht. Maybe it would make sense to archive the opam-dockerfiles repo, so people like me aren’t confused. Or add a note in the README because I see some pull requests there and accidentally submitting pull requests to a repo that is not meant to be updated is probably a wasted effort.
I think the issue @BrendanLong complains about is that there are no maintained OPAM 1.2 Docker images, while the maintained Docker images are OPAM 2 which is in prerelease status.
Personally I have long switched to OPAM 2 anyway and was already hoping for OPAM 2 Docker images so this works pretty well for me.
Thanks for the offer of help @Leonidas, it’s much appreciated! (I’m choosing to ignore @BrendanLong’s unkind and unconstructive message).
The situation with those containers is:
ocaml/opam : these were the first cut at building a container matrix, and were done from the first version of avsm/ocaml-dockerfile. They are built on the Docker Hub builder, but suffered from not having a good way to expire images. That meant that every time a new OCaml release comes out (including patch releases), we have to build an every-growing matrix that takes up 100s of VMs and close to a terabyte of images! We also started tracking opam2 in there, which made it a terrible burden to maintain as that keeps changing.
To solve this, enter…
ocaml/opam2-staging: this repository is generated from the latest version of ocaml-dockerfile (5.0 that I released yesterday). It solves some of the pressing maintenance problems of the original repository, and also focusses on supporting opam2 as best as possible. This includes some really nice new features: image sandboxing support, the images are multiarchitecture (ppc64, arm64) and automatically built using Buildkite and a large Docker machine cluster hosted at the Cambridge Computer Lab and Packet.net rather than ongoing cloud services (which I personally stuck on my credit card for far too long). In order to test this, @kit-ty-kate has been deploying it on the opam-repository CI for the past few months, and we are finally happy.
A few days, I recreated:
ocaml/opam2 - this is the place where the latest images for opam2 are pushed from the automated container builds, and will be used as the basis for ongoing CI. An important change to the naming layout here is that we drop patch releases from the container names (e…g ocaml/opam2:4.06) so that we don’t need to rebuild a huge container base. As distros expire, we can now easily push “do not use this image anymore” containers so that we can keep our build matrix reasonably sized.
Reconstructing the opam1 containers is tricky and difficult to contribute to – I’m planning to spend some time rebuilding and parking them as soon as we can shift the majority of opam-ci testing over to opam2. The important thing here is that opam1 will be tested beyond the opam2 release – however, the amount of testing will be reduced. For example, reverse dependency testing for newly submitted packages has always happened on opam2.
For new contributions, I’m going to archive the old repositories as you mentioned and am documenting the infrastructure. Stay tuned for a post about a cool new CI tool in another post today or tomorrow for more details
The new infrastructure is also satisfyingly maintainable – @dra27 has gotten Windows Buildkite workers, and I have Windows Docker containers working, and *BSD agents also activated. Hooking these in will happen over the next few months, and a fantastic area for new contributors to get involved in.
My apologies that your time was wasted looking into the myriad of repositories for various iterations of these containers – it has been a process of co-evolution as we fixed quite a few issues in the underlying Docker technologies over the past couple of years as we refined the OCaml container support The latest version of Docker with containerd is running incredibly smoothly, and I’m looking forward to integrating with the forthcoming BuildKit as well for even faster builds.
Sorry, my original message was jumping to conclusions. The first response made me assume that the opam 1.2 Docker infrastructure was wound down because it was replaced by opam 2.0 (even though opam 2.0 isn’t out yet), but it seems like what actually happened was that the original opam Docker infrastructure (which was both 1.2 and 2.0) was impossible to keep using, and the replacement for 2.0 was just made before the replacement for 1.2.
And for the record, I am excited about opam 2.0 (especially not needing an external solver), I just desperately don’t want any more pre-release software in my builds.
Glad to hear that! You don’t need the full matrix of distro containers for ordinary use of opam1 - just the :latest target should work, albeit slightly more slowly since you need to run an git -C /home/opam/opam-repository pull origin && opam update -uy && opam switch 4.06.1 to get a new compiler.
As promised, here is an update on the opam-ci tool. We’ll also be working on migrating the mirage-ci (which weirdly powers the opam-repository CI) tool and @kit-ty-kate’s opam-check-all to use portions of this new infrastructure soon, so the state of infrastructure documentation is steadily improving
Hi @avsm, I’m curious, what’s the reasoning behind having a local copy of opam-repository inside the images? Is it to reduce the load on the central repo?
Also, someone on Discord asked about what makes the opam2 images so big (even when pulling the ocaml/opam2:alpine image, the last layer is of 538.7MB size), and I kind of want to know too. I wonder if that sheer size is caused by the local opam-repo copy. Are we able to inspect the dockerfiles that generate the images? I can’t seem to find a reference to it from the wiki.
There are a few compilers preinstalled. These images are optimised for CI convenience and not image size. (e.g. you can use the preinstalled compilers to quickly run Dune tests across multiple versions of OCaml using the workspaces feature)
