Experimental new layout for the ocaml-variants packages in opam-repository

Ecosystem
1

The 4.12.0 alpha1 release is out and with it is an experiment for a new layout for the ocaml-variants packages to try to reduce the number of packages needed for each compiler release and also to allow packages to depend on particular OCaml configurations.

Combinatorial explosions in ocaml-variants

This may seem like pre-history, but back in OPAM 1.x there was a distinction between the compiler and the packages installed in a switch. Every switch essentially had just one special package at its root and OPAM 1.x had the version of OCaml hard-wired into the ocaml-version variable. This set-up meant that non-vanilla installations of the compiler (for example, with flambda) had to be encoded on the version number giving us the slightly strange 4.07.0+flambda version meaning “OCaml 4.07.0 with flambda enabled”.

Fast forward to opam 2.0, and this gets generalised with the “compilers-as-packages” feature. The old compiler special package got split into ocaml-system (taking on the even more magical and hard-wired internal system switch from OPAM 1.x), ocaml-base-compiler (for the unaltered or “vanilla” compiler) and then ocaml-variants which contained, um, everything else!

However, “compilers-as-packages” allows opam 2.x switches not to have the compiler as strictly the root package. The current scheme has a couple of benefits:

  • It’s immediately clear from the version of the ocaml-variants package what you have (for example, ocaml-variants.4.08.1+fp+flambda)
  • It’s immediately clear which variants are supported by a given version of OCaml (for example, there is no ocaml-variants.4.02.3+flambda package)

However, it some serious limitations:

  • The variants should be combinable, but doing this would lead to thousands of variants. If you need a missing one, the only solution is to have a custom repository
  • Every single release of the compiler needs to have corresponding variant packages
  • The very large number of compiler packages considerably increases the pressure on opam’s solvers
  • It’s not possible to depend on OCaml options (for example, to depend on, or conflict with, no-naked-pointers mode).

@kit-ty-kate, @AltGr, and I have at various points done experiments to improve this. Thanks to @octachron, we’re trying one of them out in the safety of the 4.12 alpha/beta/rc packages and, in addition of course to testing your code with OCaml 4.12, we could really do with feedback on this new layout.

How it works

The ocaml-variants.4.12.0+trunk package and the new ocaml-variants.4.12.0~alpha1+options packages both allow compiler options to be customised by also installing ocaml-option- packages:

  • ocaml-option-32bit - 32bit build
  • ocaml-option-afl - enable AFL support
  • ocaml-option-bytecode-only - build bytecode compiler only
  • ocaml-option-default-unsafe-string - enable unsafe-string by default
  • ocaml-option-flambda - enable flambda
  • ocaml-option-fp - enable frame pointers
  • ocaml-option-musl - use musl instead of libc
  • ocaml-option-nnp - enable no-naked-pointers mode
  • ocaml-option-no-flat-float-array - disable flat float arrays in the runtime
  • ocaml-option-spacetime (for 4.12 this option is not available, since spacetime has been removed)
  • ocaml-option-static - no dynamic linking

This allows options to be combined: for example installing ocaml-option-32bit and ocaml-option-flambda enables a 32-bit flambda compiler. It’s a bit of a mouthful (more on that below):

opam switch create 4.12-32-bit-flambda --packages=ocaml-variants.4.12.0~alpha1+options,ocaml-option-32bit,ocaml-option-flambda --repos=default,beta

(obviously if the new layout is adopted at release, then the --repos part won’t be necessary)

In addition to these packages, there are also a small number of ocaml-options-only- packages (for example, ocaml-options-only-flambda). These packages conflict with other options packages, so for example with this:

opam switch create 4.12-release-builds --packages=ocaml-variants.4.12.0~alpha1+options,ocaml-options-only-flambda-fp --repos=default,beta

you get a switch with 4.12.0 alpha1 configured for flambda and frame-pointers and which cannot be changed. So if you attempt to install a package which depends on ocaml-option-nnp, for example, then opam will complain of a conflict rather than suggesting that you recompile your switch with ocaml-config-nnp.

The special package ocaml-options-vanilla can be used to make ocaml-variants.4.12.0~alpha1+options or ocaml-variants.4.12.0+trunk behave like ocaml-base-compiler.

Currently

  • :white_check_mark: variants can be combined, and they don’t need to be specified with every compiler release
  • :grey_question: the CLI invocation is slightly more characters to type, though still about as clunky as the existing package names (opam 2.1 slightly improves this)
  • :grey_question: there’s still a combinatorial explosion with the ocaml-options-only-* packages, but we’re hoping that that’s less of a problem since these only packages are mainly useful for CI/CD systems
  • :x: +trunk packages still exist
  • :x: ocaml-base-compiler.v and ocaml-variants.v+options are an unnecessary and slightly confusing distinction

Future work

  • There’s no reason to be maintaining ocaml-variants and ocaml-base-compiler separately and we could look to merging these into, say, ocaml-compiler
  • The +trunk variants have always been slightly odd - both because the maintenance branches for each release (e.g. 4.11, 4.12) are not called trunk and because the version number refers to an as-yet unreleased version of OCaml (e.g. 4.11.2+trunk). We’re looking to switch to using dev-repo field and pinning, as other packages do, but also trying to keep a lid on the number of simultaneous changes!
  • opam 2.2 has plans in the pipeline to address “package parameters” properly, which should hopefully improve both the CLI and the number of “configuration-related” packages
  • The layout could be back-ported to previous versions of the compiler. There are two questions with this: deleting the old variants packages will cause any existing switches to fail to upgrade, so we may wish to keep virtual packages for the old releases (e.g. have ocaml-variants.4.11.1+flambda depend on ocaml-compiler.4.11.1 and ocaml-options-only-flambda) and there’s also the fact that changing the opam files for these packages will cause all switches to rebuild at next opam upgrade!

Thanks for making it to the end, and all comments and feedback either to me directly, here on Discuss, or on the opam-repository issue tracker are very welcome!

10 Likes
2

Problem

Here is a small point of feedback: I wanted to install a flambda switch for 4.12.0 just today, and I realized that I had forgotten how to do it. Then I looked around a bit (details below), and it took me a good five minutes to find some content on the web explaining how to do it. Could the documentation of opam switch --help be improved to reduce the search time?

User trace

What I did:

  1. I noticed that there were no 4.12.0+flambda switches anymore. I vaguely remembered this change: ah, I had to setup the +options package with some extra packages.

  2. opam search flambda told me about ocaml-option-flambda, but how to use it as a base package of a switch?

  3. Then I did opam switch create --help, hoping to find an explanation. I didn’t find clear documentation of how to proceed there. I did find, in the middle of the list of options, the --packages option, but the documentation says:

       --packages=PACKAGES
  When installing a switch, explicitly define
  the set of packages to set as the compiler.

    I don’t know what “to set as the compiler” was and it sounds a bit scary to me. It does not say “pass extra packages to be part of the base set of the switch” or something like that; I probably don’t want ocaml-option-flambda to be set as the compiler!

  4. Finally, I thought that I would search the web for the initial announce of the change, which would contain the examples I need. I just searched for ocaml-conf-flambda, which first bring to useless package page (no documentation on how to use the package), then to the 4.12.0 release notes that do contain the bit I needed, namely this example that I could adapt to my needs:

    opam switch create 4.12.0+flambda+nnpchecker --package=ocaml-variants.4.12.0+options,ocaml-option-flambda,ocaml-option-nnpchecker

Proposed solution

I would propose the following fixes to the documentation:

  1. The description of the ocaml-option-flambda package (and all other option packages) contains information on how to configure a new switch with the package enabled.
  2. opam switch --help explains the new compiler-package layout and gives an example of setting up a compiler with specific options.
7 Likes
3

By the strangest of coincidences, I committed to updating the extremely draft docs for this in time for today’s “release readiness” weekly meeting, so hopefully better-late-than-never for published documentation for this!

This is a great idea, thanks - I was expecting to put the documentation for the layout in ocaml/opam-repository’s wiki; linking it and including an example in the packages themselves is a good way of surfacing that page.

This doesn’t work, though, or at the very least it’s not as simple as just updating the help text. On the philosophical side, opam is technically language-agnostic, but more importantly it means that a change like this requires opam to be recompiled (and we’d forever be receiving reports from users on slightly-behind distributions).

1 Like
4

At the start of summer I had a similar experience to @gasche’s; the workflow for configuring a new switch needs to be clearer. After a long battle against the documentation, I finally succeeded in creating local switches with:

opam switch create . 4.12.0+options --packages=ocaml-option-flambda,...

But once I upgraded to opam 2.1 this started to fail with an error message that seemed to suggest I was already using the command correctly, so I don’t even know what the right way is anymore. I eventually figured a convoluted workaround (three commands!), but then decided that staying in opam 2.0 is simpler until the docs are clearer.

1 Like
5

Excellent point, but can we work through the difficulty? Could we ask opam repositories (which presumably have conventions (possibly language-specific) about how to setup their switches) provide “switch documentation”, and quote this documentation inside the opam switch --help output? (So the output would depend dynamically on the repositories configured.)

(Are there other parts of opam’s command-line-reachable documentation we may want to customize in this style?)

6

@debugnik - sorry about that. Ironically, the runes for opam 2.1 should be much simpler:

opam switch create . ocaml.4.12.0 ocaml-option-flambda

and all the other details are “hidden”.

@gasche - yes, sorry, “doesn’t work” read more finally than I intended! It would indeed be possible to do something at the repository level (it’s tricky because of backwards compatibility, but certainly not impossible!)

2 Likes
7

Could you help to repair opam init too?

The

opam init --compiler=ocaml-variants.4.12.0+options,ocaml-option-flambda

doesn’t work because of the [ERROR] Invalid package specification or version “ocaml-variants.4.12.0+options,ocaml-option-flambda”

The

opam init --compiler=ocaml-variants.4.12.0+options

seems to work file, but doesn’t have flambda enabled

1 Like
8

I’ve opened a meta issue for the issues raised here in: Meta issue on the problems encountered wrt the new compiler variants layout · Issue #4838 · ocaml/opam · GitHub

9

I think we need better documentation for this. It took me a good 10 minutes on how to create a bytecode only switch.

For the interested reader that made it for me.

opam switch create ocaml.4.14.1+bco --packages=ocaml.4.14.1,ocaml-option-bytecode-only
1 Like
10

We really do. I’m adding guerilla documentation to individual options packages (like ocaml-option-nnpchecker) as I get a chance, but there’s no one place where all of these are documented at the moment. PRs from anyone towards this in the opam repository are most welcome.

11

That’s certainly nice but that was not really my problem.

My problem was:

  1. I know it is possible to install a specially configured ocaml package.
  2. Where do I find the instructions to do so ?

I headed to the opam manual secretly knowing that I wouldn’t find the answer – because it’s rather an encoding trick specific to the OCaml opam repository.

Then I headed to the opam repository where I thought maybe the encoding was described, it wasn’t.

Then I thought maybe I would find documentation on the OCaml websites where after convincing my brain that documentation could perhaps live under ‘Learn’ (rather than ‘Docs’) I found this which didn’t help.

So I eventually turned to this forum where after consulting a few messages I managed to come up with this invocation by piecing up together different, seemingly contradicting (--compiler or --package ?), cli invocations.

HTH

3 Likes
12

Maybe we could offer this documentation in the description field of the ocaml package, which currently contains:

This package requires a matching implementation of OCaml,
and polls it to initialise specific variables like
ocaml:native-dynlink.

(this comment is in fact fairly obscure to me, I don’t understand what users are supposed to do from “This package requires a matching implementation of OCaml”.)

If you think of doing opam show ocaml-variants instead, this is better because the depopts listing shows all available options (… for some OCaml version).

Maybe the description of ocaml.5.0.0 could say something like:

See opam show ocaml-variants.5.0.0+options for information on how to configure the OCaml compiler.

and then ocaml-variants.<version>+options would (list available options in its deopts field and) have explanation in the description for how to setup configuration options, with working examples for common configurations.

13

As for the where this should go this could be an idea. But the surrounding material still needs to point you there somehow (that is I personally wouldn’t have thought to opam info ocaml).

Taking a step back regarding the encoding of configuration options into packages names leaves me rather sceptical. What we really want here is a way to configure a package before installing it. I made this proposal 8 years ago, maybe I would do it differently nowadays (i.e. I didn’t re-read it), but it seems to me that there’s a real need for it.

14

Maybe too late for this, but:

“Compilers-as-packages” is a good idea, but its an implementation detail. Every switch requires a compiler, so we should have --compiler=<version> - the user does not care if that involves a “package” or not. The <version> string should be just the version ID, without prefix (ocaml-variants. or ocaml-base-compiler. or whatever).

The current naming scheme is verbose and overly opaque. I think its a barrier to entry. For example, the ocaml- prefix does not tell us anything we do not already know. Such a prefix is only needed for non-standard compilers like dkml-.

The +options suffix - is it really necessary?

Ditto for ocaml-options-vanilla - if no “options” are provided, then the inference is that the compiler should be built with defaults.

The options themselves do not need the ocaml-option- prefix. GNU configure is widely-used and understood. Its convention is to use --with-<pkg> for add-ons, and --enable-<feature> for togglable features already available in the build. Following that convention would result in much more readable commands, IMO:

enable-32bit - 32bit build (default: disable)
disable-64bit - do not build 64bit stuff (default: enable)
enable-afl - enable AFL support
disable-bytecode - default:enable
disable-native - default: enable
enable-unsafe-string - enable unsafe-string as default (default:disable)
enable-flambda - enable flambda
enable-fp - enable frame pointers
with-musl - use musl instead of libc
enable-nnp - enable no-naked-pointers mode
disable-flat-float-array - disable flat float arrays in the runtime
enable-spacetime (for 4.12 this option is not available, since spacetime has been removed)
disable-dynamic-linking - instead of ocaml-option-static, no dynamic linking

E.g. instead of ocaml-option-bytecode-only we would have disable-native.

For example, instead of:

opam switch create foo --package=ocaml-variants.4.12.0+options,ocaml-option-flambda,ocaml-option-nnpchecker

we would have:

opam switch create foo --compiler=4.12.0,enable-flambda,with-nnpchecker
15

This is not true. It’s reasonable to use OPAM for packages that do not require an OCaml compiler.

16

I stand corrected. But I think the point still holds: for switches that do need a compiler, --compiler makes sense.