A hassle-free setup to release binaries for different platforms: the opam release process experiment

Ecosystem
,
1

On top of the opam 2.1.2 announcement, I’d like share an experiment with the opam release script used for this release.

As you might know, for each releases of opam we provide pre-compiled binaries for ease of use.
We’ve had a release script which up to this point required a specific setup to get it running correctly. For instance we had to setup a local OpenBSD machine (possibliy virtualised), a macOS/x86_64 machine and a macOS/arm64. This setup is rather tedious to reproduce.

To improve this situation I’ve experimented over the past week with QEMU and Rosetta 2 to make it a “one click script”:

This change makes so that the script now only requires a macOS/arm64. From there you can:

  • compile locally for macOS/arm64 binaries
  • compile locally for macOS/x86_64 binaries (using Rosetta 2)
  • compile for BSDs (using QEMU)
  • compile for Linux (using Docker)

With this, the binaries for this release have been compiled with this more reproducible setup, and now include FreeBSD/x86_64 binaries as well :sparkles:

If someone wants to have a similar setup to distribute binaries here is the git repository (using Git LFS to store the large files). Feel free to use and experiment with it:

For now it only has OpenBSD/x86_64 and FreeBSD/x86_64 images but it could theoretically have more. Although I’m not accepting PRs for now (for obvious security reasons), I’m open to suggestions to add more platforms. See the README for high level details about the setup.

18 Likes
2

Some joyful updates: since last time, the script now takes 1h37 compared to more than 7h before, on the slowest hardware supported (the fanless M1 macbook air), thanks to some performance improvements in the latest versions of Docker and QEMU.

The latter in particular now supports emulating multicore x86_64 systems on arm64 (strong memory model on weak memory model), which allows to take full advantage of every cpu cores available on the host machine for this 5x performance improvement.
To take advantage of this, one setback i had was figuring out that OpenBSD doesn’t install the multicore capable kernel by default if you’ve installed it on single core machine (e.g. QEMU without -smp 2 or above), so i had to recreate the OpenBSD image once again.

2 Likes