Experimental new layout for the ocaml-variants packages in opam-repository

The 4.12.0 alpha1 release is out and with it is an experiment for a new layout for the ocaml-variants packages to try to reduce the number of packages needed for each compiler release and also to allow packages to depend on particular OCaml configurations.

Combinatorial explosions in ocaml-variants

This may seem like pre-history, but back in OPAM 1.x there was a distinction between the compiler and the packages installed in a switch. Every switch essentially had just one special package at its root and OPAM 1.x had the version of OCaml hard-wired into the ocaml-version variable. This set-up meant that non-vanilla installations of the compiler (for example, with flambda) had to be encoded on the version number giving us the slightly strange 4.07.0+flambda version meaning “OCaml 4.07.0 with flambda enabled”.

Fast forward to opam 2.0, and this gets generalised with the “compilers-as-packages” feature. The old compiler special package got split into ocaml-system (taking on the even more magical and hard-wired internal system switch from OPAM 1.x), ocaml-base-compiler (for the unaltered or “vanilla” compiler) and then ocaml-variants which contained, um, everything else!

However, “compilers-as-packages” allows opam 2.x switches not to have the compiler as strictly the root package. The current scheme has a couple of benefits:

  • It’s immediately clear from the version of the ocaml-variants package what you have (for example, ocaml-variants.4.08.1+fp+flambda)
  • It’s immediately clear which variants are supported by a given version of OCaml (for example, there is no ocaml-variants.4.02.3+flambda package)

However, it some serious limitations:

  • The variants should be combinable, but doing this would lead to thousands of variants. If you need a missing one, the only solution is to have a custom repository
  • Every single release of the compiler needs to have corresponding variant packages
  • The very large number of compiler packages considerably increases the pressure on opam’s solvers
  • It’s not possible to depend on OCaml options (for example, to depend on, or conflict with, no-naked-pointers mode).

@kit-ty-kate, @AltGr, and I have at various points done experiments to improve this. Thanks to @octachron, we’re trying one of them out in the safety of the 4.12 alpha/beta/rc packages and, in addition of course to testing your code with OCaml 4.12, we could really do with feedback on this new layout.

How it works

The ocaml-variants.4.12.0+trunk package and the new ocaml-variants.4.12.0~alpha1+options packages both allow compiler options to be customised by also installing ocaml-option- packages:

  • ocaml-option-32bit - 32bit build
  • ocaml-option-afl - enable AFL support
  • ocaml-option-bytecode-only - build bytecode compiler only
  • ocaml-option-default-unsafe-string - enable unsafe-string by default
  • ocaml-option-flambda - enable flambda
  • ocaml-option-fp - enable frame pointers
  • ocaml-option-musl - use musl instead of libc
  • ocaml-option-nnp - enable no-naked-pointers mode
  • ocaml-option-no-flat-float-array - disable flat float arrays in the runtime
  • ocaml-option-spacetime (for 4.12 this option is not available, since spacetime has been removed)
  • ocaml-option-static - no dynamic linking

This allows options to be combined: for example installing ocaml-option-32bit and ocaml-option-flambda enables a 32-bit flambda compiler. It’s a bit of a mouthful (more on that below):

opam switch create 4.12-32-bit-flambda --packages=ocaml-variants.4.12.0~alpha1+options,ocaml-option-32bit,ocaml-option-flambda --repos=default,beta

(obviously if the new layout is adopted at release, then the --repos part won’t be necessary)

In addition to these packages, there are also a small number of ocaml-options-only- packages (for example, ocaml-options-only-flambda). These packages conflict with other options packages, so for example with this:

opam switch create 4.12-release-builds --packages=ocaml-variants.4.12.0~alpha1+options,ocaml-options-only-flambda-fp --repos=default,beta

you get a switch with 4.12.0 alpha1 configured for flambda and frame-pointers and which cannot be changed. So if you attempt to install a package which depends on ocaml-option-nnp, for example, then opam will complain of a conflict rather than suggesting that you recompile your switch with ocaml-config-nnp.

The special package ocaml-options-vanilla can be used to make ocaml-variants.4.12.0~alpha1+options or ocaml-variants.4.12.0+trunk behave like ocaml-base-compiler.

Currently

  • :white_check_mark: variants can be combined, and they don’t need to be specified with every compiler release
  • :grey_question: the CLI invocation is slightly more characters to type, though still about as clunky as the existing package names (opam 2.1 slightly improves this)
  • :grey_question: there’s still a combinatorial explosion with the ocaml-options-only-* packages, but we’re hoping that that’s less of a problem since these only packages are mainly useful for CI/CD systems
  • :x: +trunk packages still exist
  • :x: ocaml-base-compiler.v and ocaml-variants.v+options are an unnecessary and slightly confusing distinction

Future work

  • There’s no reason to be maintaining ocaml-variants and ocaml-base-compiler separately and we could look to merging these into, say, ocaml-compiler
  • The +trunk variants have always been slightly odd - both because the maintenance branches for each release (e.g. 4.11, 4.12) are not called trunk and because the version number refers to an as-yet unreleased version of OCaml (e.g. 4.11.2+trunk). We’re looking to switch to using dev-repo field and pinning, as other packages do, but also trying to keep a lid on the number of simultaneous changes!
  • opam 2.2 has plans in the pipeline to address “package parameters” properly, which should hopefully improve both the CLI and the number of “configuration-related” packages
  • The layout could be back-ported to previous versions of the compiler. There are two questions with this: deleting the old variants packages will cause any existing switches to fail to upgrade, so we may wish to keep virtual packages for the old releases (e.g. have ocaml-variants.4.11.1+flambda depend on ocaml-compiler.4.11.1 and ocaml-options-only-flambda) and there’s also the fact that changing the opam files for these packages will cause all switches to rebuild at next opam upgrade!

Thanks for making it to the end, and all comments and feedback either to me directly, here on Discuss, or on the opam-repository issue tracker are very welcome!

7 Likes