# Experimental new layout for the ocaml-variants packages in opam-repository
**URL:** https://discuss.ocaml.org/t/experimental-new-layout-for-the-ocaml-variants-packages-in-opam-repository/6779
**Category:** Ecosystem
**Tags:** opam
**Created:** [November 16, 2020, 12:20pm UTC](https://discuss.ocaml.org/t/experimental-new-layout-for-the-ocaml-variants-packages-in-opam-repository/6779 "2020-11-16T12:20:29Z")
**Posts on this page:** 16
**Page:** 1
### Author:  [@dra27](https://discuss.ocaml.org/u/dra27)
#### Post date: [November 16, 2020, 12:20pm UTC](https://discuss.ocaml.org/t/experimental-new-layout-for-the-ocaml-variants-packages-in-opam-repository/6779/1 "2020-11-16T12:20:29Z")
The 4.12.0 alpha1 release is out and with it is an experiment for a new layout for the ocaml-variants packages to try to reduce the number of packages needed for each compiler release and also to allow packages to depend on particular OCaml configurations.
## Combinatorial explosions in `ocaml-variants`
This may seem like pre-history, but back in OPAM 1.x there was a distinction between the compiler and the packages installed in a switch. Every switch essentially had just one special package at its root and OPAM 1.x had the version of OCaml hard-wired into the `ocaml-version` variable. This set-up meant that non-vanilla installations of the compiler (for example, with flambda) had to be encoded on the version number giving us the slightly strange `4.07.0+flambda` version meaning “OCaml 4.07.0 with flambda enabled”.
Fast forward to opam 2.0, and this gets generalised with the “compilers-as-packages” feature. The old compiler special package got split into `ocaml-system` (taking on the even more magical and hard-wired internal `system` switch from OPAM 1.x), `ocaml-base-compiler` (for the unaltered or “vanilla” compiler) and then `ocaml-variants` which contained, um, everything else!
However, “compilers-as-packages” allows opam 2.x switches not to have the compiler as strictly the root package. The current scheme has a couple of benefits:
- It’s immediately clear from the version of the `ocaml-variants` package what you have (for example, `ocaml-variants.4.08.1+fp+flambda`)
- It’s immediately clear which variants are supported by a given version of OCaml (for example, there is no `ocaml-variants.4.02.3+flambda` package)
However, it some serious limitations:
- The variants should be combinable, but doing this would lead to thousands of variants. If you need a missing one, the only solution is to have a custom repository
- Every single release of the compiler needs to have corresponding variant packages
- The very large number of compiler packages considerably increases the pressure on opam’s solvers
- It’s not possible to depend on OCaml options (for example, to depend on, or conflict with, no-naked-pointers mode).
@kit-ty-kate, @AltGr, and I have at various points done experiments to improve this. Thanks to @octachron, we’re trying one of them out in the safety of the 4.12 alpha/beta/rc packages and, in addition of course to testing your code with OCaml 4.12, we could really do with feedback on this new layout.
## How it works
The `ocaml-variants.4.12.0+trunk` package and the new `ocaml-variants.4.12.0~alpha1+options` packages both allow compiler options to be customised by also installing `ocaml-option-` packages:
- `ocaml-option-32bit` - 32bit build
- `ocaml-option-afl` - enable AFL support
- `ocaml-option-bytecode-only` - build bytecode compiler only
- `ocaml-option-default-unsafe-string` - enable unsafe-string by default
- `ocaml-option-flambda` - enable flambda
- `ocaml-option-fp` - enable frame pointers
- `ocaml-option-musl` - use musl instead of libc
- `ocaml-option-nnp` - enable no-naked-pointers mode
- `ocaml-option-no-flat-float-array` - disable flat float arrays in the runtime
- `ocaml-option-spacetime` (for 4.12 this option is not available, since spacetime has been removed)
- `ocaml-option-static` - no dynamic linking
This allows options to be combined: for example installing `ocaml-option-32bit` and `ocaml-option-flambda` enables a 32-bit flambda compiler. It’s a bit of a mouthful (more on that below):
```auto
opam switch create 4.12-32-bit-flambda --packages=ocaml-variants.4.12.0~alpha1+options,ocaml-option-32bit,ocaml-option-flambda --repos=default,beta
```
(obviously if the new layout is adopted at release, then the `--repos` part won’t be necessary)
In addition to these packages, there are also a small number of `ocaml-options-only-` packages (for example, `ocaml-options-only-flambda`). These packages conflict with other options packages, so for example with this:
```auto
opam switch create 4.12-release-builds --packages=ocaml-variants.4.12.0~alpha1+options,ocaml-options-only-flambda-fp --repos=default,beta
```
you get a switch with 4.12.0 alpha1 configured for flambda and frame-pointers _and which cannot be changed_. So if you attempt to install a package which depends on `ocaml-option-nnp`, for example, then opam will complain of a conflict rather than suggesting that you recompile your switch with `ocaml-config-nnp`.
The special package `ocaml-options-vanilla` can be used to make `ocaml-variants.4.12.0~alpha1+options` or `ocaml-variants.4.12.0+trunk` behave like `ocaml-base-compiler`.
## Currently
- ✅ variants can be combined, and they don’t need to be specified with every compiler release
- ❔ the CLI invocation is slightly more characters to type, though still about as clunky as the existing package names (opam 2.1 slightly improves this)
- ❔ there’s still a combinatorial explosion with the `ocaml-options-only-*` packages, but we’re hoping that that’s less of a problem since these `only` packages are mainly useful for CI/CD systems
- ❌ `+trunk` packages still exist
- ❌ `ocaml-base-compiler.v` and `ocaml-variants.v+options` are an unnecessary and slightly confusing distinction
## Future work
- There’s no reason to be maintaining `ocaml-variants` and `ocaml-base-compiler` separately and we could look to merging these into, say, `ocaml-compiler`
- The `+trunk` variants have always been slightly odd - both because the maintenance branches for each release (e.g. [`4.11`](https://github.com/ocaml/ocaml/tree/4.11), [`4.12`](https://github.com/ocaml/ocaml/tree/4.12)) are not called `trunk` and because the version number refers to an as-yet unreleased version of OCaml (e.g. `4.11.2+trunk`). We’re looking to switch to using `dev-repo` field and pinning, as other packages do, but also trying to keep a lid on the number of simultaneous changes!
- opam 2.2 has plans in the pipeline to address “package parameters” properly, which should hopefully improve both the CLI and the number of “configuration-related” packages
- The layout _could_ be back-ported to previous versions of the compiler. There are two questions with this: deleting the old variants packages will cause any existing switches to fail to upgrade, so we may wish to keep virtual packages for the old releases (e.g. have `ocaml-variants.4.11.1+flambda` depend on `ocaml-compiler.4.11.1` and `ocaml-options-only-flambda`) and there’s also the fact that changing the opam files for these packages will cause _all_ switches to rebuild at next `opam upgrade`!
Thanks for making it to the end, and all comments and feedback either to me directly, here on Discuss, or on the [opam-repository issue tracker](https://github.com/ocaml/opam-repository/issues) are very welcome!
---
### Author:  [@gasche](https://discuss.ocaml.org/u/gasche)
#### Post date: [September 4, 2021, 9:04pm UTC](https://discuss.ocaml.org/t/experimental-new-layout-for-the-ocaml-variants-packages-in-opam-repository/6779/2 "2021-09-04T21:04:22Z")
### Problem
Here is a small point of feedback: I wanted to install a `flambda` switch for 4.12.0 just today, and I realized that I had forgotten how to do it. Then I looked around a bit (details below), and it took me a good five minutes to find some content on the web explaining how to do it. Could the documentation of `opam switch --help` be improved to reduce the search time?
### User trace
What I did:
1. I noticed that there were no 4.12.0+flambda switches anymore. I vaguely remembered this change: ah, I had to setup the `+options` package with some extra packages.
2. `opam search flambda` told me about `ocaml-option-flambda`, but how to use it as a base package of a switch?
3. Then I did `opam switch create --help`, hoping to find an explanation. I didn’t find clear documentation of how to proceed there. I did find, in the middle of the list of options, the `--packages` option, but the documentation says:
4. Finally, I thought that I would search the web for the initial announce of the change, which would contain the examples I need. I just searched for `ocaml-conf-flambda`, which first bring to useless [package page](https://opam.ocaml.org/packages/ocaml-option-flambda/) (no documentation on how to use the package), then to [the 4.12.0 release notes](https://ocaml.org/releases/4.12.0.html) that do contain the bit I needed, namely this example that I could adapt to my needs:
### Proposed solution
I would propose the following fixes to the documentation:
1. The description of the `ocaml-option-flambda` package (and all other option packages) contains information on how to configure a new switch with the package enabled.
2. `opam switch --help` explains the new compiler-package layout and gives an example of setting up a compiler with specific options.
---
### Author:  [@dra27](https://discuss.ocaml.org/u/dra27)
#### Post date: [September 6, 2021, 7:23am UTC](https://discuss.ocaml.org/t/experimental-new-layout-for-the-ocaml-variants-packages-in-opam-repository/6779/3 "2021-09-06T07:23:48Z")
By the strangest of coincidences, I committed to updating the extremely draft docs for this in time for today’s “release readiness” weekly meeting, so hopefully better-late-than-never for published documentation for this!
> [@gasche](#):
>
> The description of the `ocaml-option-flambda` package (and all other option packages) contains information on how to configure a new switch with the package enabled.
This is a great idea, thanks - I was expecting to put the documentation for the layout in ocaml/opam-repository’s wiki; linking it and including an example in the packages themselves is a good way of surfacing that page.
> [@gasche](#):
>
> `opam switch --help` explains the new compiler-package layout and gives an example of setting up a compiler with specific options.
This doesn’t work, though, or at the very least it’s not as simple as just updating the help text. On the philosophical side, opam is technically language-agnostic, but more importantly it means that a change like this requires opam to be recompiled (and we’d forever be receiving reports from users on slightly-behind distributions).
---
### Author:  [@amongonz](https://discuss.ocaml.org/u/amongonz)
#### Post date: [September 6, 2021, 2:18pm UTC](https://discuss.ocaml.org/t/experimental-new-layout-for-the-ocaml-variants-packages-in-opam-repository/6779/4 "2021-09-06T14:18:12Z")
At the start of summer I had a similar experience to @gasche’s; the workflow for configuring a new switch needs to be clearer. After a long battle against the documentation, I finally succeeded in creating local switches with:
```bash
opam switch create . 4.12.0+options --packages=ocaml-option-flambda,...
```
But once I upgraded to opam 2.1 this started to fail with an error message that seemed to suggest I was already using the command correctly, so I don’t even know what the right way is anymore. I eventually figured a convoluted workaround (three commands!), but then decided that staying in opam 2.0 is simpler until the docs are clearer.
---
### Author:  [@gasche](https://discuss.ocaml.org/u/gasche)
#### Post date: [September 6, 2021, 3:11pm UTC](https://discuss.ocaml.org/t/experimental-new-layout-for-the-ocaml-variants-packages-in-opam-repository/6779/5 "2021-09-06T15:11:11Z")
> [@dra27](#):
>
> On the philosophical side, opam is technically language-agnostic, but more importantly it means that a change like this requires opam to be recompiled (and we’d forever be receiving reports from users on slightly-behind distributions).
Excellent point, but can we work through the difficulty? Could we ask opam repositories (which presumably have conventions (possibly language-specific) about how to setup their switches) provide “switch documentation”, and quote this documentation inside the `opam switch --help` output? (So the output would depend dynamically on the repositories configured.)
(Are there other parts of opam’s command-line-reachable documentation we may want to customize in this style?)
---
### Author:  [@dra27](https://discuss.ocaml.org/u/dra27)
#### Post date: [September 6, 2021, 4:29pm UTC](https://discuss.ocaml.org/t/experimental-new-layout-for-the-ocaml-variants-packages-in-opam-repository/6779/6 "2021-09-06T16:29:27Z")
@amongonz - sorry about that. Ironically, the runes for opam 2.1 should be much simpler:
```auto
opam switch create . ocaml.4.12.0 ocaml-option-flambda
```
and all the other details are “hidden”.
@gasche - yes, sorry, “doesn’t work” read more finally than I intended! It would indeed be possible to do something at the repository level (it’s tricky because of backwards compatibility, but certainly not impossible!)
---
### Author:  [@Kakadu](https://discuss.ocaml.org/u/Kakadu)
#### Post date: [September 14, 2021, 4:56pm UTC](https://discuss.ocaml.org/t/experimental-new-layout-for-the-ocaml-variants-packages-in-opam-repository/6779/7 "2021-09-14T16:56:32Z")
Could you help to repair `opam init` too?
The
```
opam init --compiler=ocaml-variants.4.12.0+options,ocaml-option-flambda
```
doesn’t work because of the _[ERROR] Invalid package specification or version “ocaml-variants.4.12.0+options,ocaml-option-flambda”_
The
```
opam init --compiler=ocaml-variants.4.12.0+options
```
seems to work file, but doesn’t have flambda enabled
---
### Author:  [@kit-ty-kate](https://discuss.ocaml.org/u/kit-ty-kate)
#### Post date: [September 14, 2021, 8:28pm UTC](https://discuss.ocaml.org/t/experimental-new-layout-for-the-ocaml-variants-packages-in-opam-repository/6779/8 "2021-09-14T20:28:20Z")
I’ve opened a meta issue for the issues raised here in: [Meta issue on the problems encountered wrt the new compiler variants layout · Issue #4838 · ocaml/opam · GitHub](https://github.com/ocaml/opam/issues/4838)
---
### Author:  [@dbuenzli](https://discuss.ocaml.org/u/dbuenzli)
#### Post date: [August 28, 2023, 6:26pm UTC](https://discuss.ocaml.org/t/experimental-new-layout-for-the-ocaml-variants-packages-in-opam-repository/6779/9 "2023-08-28T18:26:46Z")
I think we need better documentation for this. It took me a good 10 minutes on how to create a bytecode only switch.
For the interested reader that made it for me.
```auto
opam switch create ocaml.4.14.1+bco --packages=ocaml.4.14.1,ocaml-option-bytecode-only
```
---
### Author:  [@avsm](https://discuss.ocaml.org/u/avsm)
#### Post date: [August 30, 2023, 4:48pm UTC](https://discuss.ocaml.org/t/experimental-new-layout-for-the-ocaml-variants-packages-in-opam-repository/6779/10 "2023-08-30T16:48:17Z")
> [@dbuenzli](#):
>
> I think we need better documentation for this.
We really do. I’m adding guerilla documentation to individual options packages (like [ocaml-option-nnpchecker](https://ocaml.org/p/ocaml-option-nnpchecker/latest)) as I get a chance, but there’s no one place where all of these are documented at the moment. PRs from anyone towards this in the opam repository are most welcome.
---
### Author:  [@dbuenzli](https://discuss.ocaml.org/u/dbuenzli)
#### Post date: [August 30, 2023, 5:09pm UTC](https://discuss.ocaml.org/t/experimental-new-layout-for-the-ocaml-variants-packages-in-opam-repository/6779/11 "2023-08-30T17:09:19Z")
That’s certainly nice but that was not really my problem.
My problem was:
1. I know it is possible to install a specially configured ocaml package.
2. Where do I find the instructions to do so ?
I headed to the [opam manual](https://opam.ocaml.org/) secretly knowing that I wouldn’t find the answer – because it’s rather an encoding trick specific to the OCaml opam repository.
Then I headed to the [opam repository](https://github.com/ocaml/opam-repository/) where I thought maybe the encoding was described, it wasn’t.
Then I thought maybe I would find documentation on the OCaml websites where after convincing my brain that documentation could perhaps live under ‘Learn’ (rather than ‘Docs’) I found [this](https://ocaml.org/docs/install-a-specific-ocaml-compiler-version) which didn’t help.
So I eventually turned to this forum where after consulting a few messages I managed to come up with this invocation by piecing up together different, seemingly contradicting (`--compiler` or `--package` ?), cli invocations.
HTH
---
### Author:  [@gasche](https://discuss.ocaml.org/u/gasche)
#### Post date: [August 31, 2023, 7:38am UTC](https://discuss.ocaml.org/t/experimental-new-layout-for-the-ocaml-variants-packages-in-opam-repository/6779/12 "2023-08-31T07:38:17Z")
Maybe we could offer this documentation in the `description` field of the `ocaml` package, which currently contains:
> This package requires a matching implementation of OCaml,
> and polls it to initialise specific variables like
> `ocaml:native-dynlink`.
(this comment is in fact fairly obscure to me, I don’t understand what users are supposed to do from “This package requires a matching implementation of OCaml”.)
If you think of doing `opam show ocaml-variants` instead, this is better because the `depopts` listing shows all available options (… for some OCaml version).
Maybe the description of `ocaml.5.0.0` could say something like:
> See `opam show ocaml-variants.5.0.0+options` for information on how to configure the OCaml compiler.
and then `ocaml-variants.+options` would (list available options in its `deopts` field and) have explanation in the `description` for how to setup configuration options, with working examples for common configurations.
---
### Author:  [@dbuenzli](https://discuss.ocaml.org/u/dbuenzli)
#### Post date: [August 31, 2023, 8:54am UTC](https://discuss.ocaml.org/t/experimental-new-layout-for-the-ocaml-variants-packages-in-opam-repository/6779/13 "2023-08-31T08:54:59Z")
> [@gasche](#):
>
> Maybe we could offer this documentation in the `description` field of the `ocaml` package
As for the _where_ this should go this could be an idea. But the surrounding material still needs to point you there somehow (that is I personally wouldn’t have thought to `opam info ocaml`).
Taking a step back regarding the encoding of configuration options into packages names leaves me rather sceptical. What we really want here is a way to configure a package before installing it. I made [this proposal](https://github.com/ocaml/opam/issues/2247) 8 years ago, maybe I would do it differently nowadays (i.e. I didn’t re-read it), but it seems to me that there’s a real need for it.
---
### Author:  [@mobileink](https://discuss.ocaml.org/u/mobileink)
#### Post date: [September 5, 2023, 2:33pm UTC](https://discuss.ocaml.org/t/experimental-new-layout-for-the-ocaml-variants-packages-in-opam-repository/6779/14 "2023-09-05T14:33:54Z")
Maybe too late for this, but:
“Compilers-as-packages” is a good idea, but its an implementation detail. Every switch requires a compiler, so we should have `--compiler=` - the user does not care if that involves a “package” or not. The `` string should be just the version ID, without prefix (`ocaml-variants.` or `ocaml-base-compiler.` or whatever).
The current naming scheme is verbose and overly opaque. I think its a barrier to entry. For example, the `ocaml-` prefix does not tell us anything we do not already know. Such a prefix is only needed for non-standard compilers like `dkml-`.
The `+options` suffix - is it really necessary?
Ditto for `ocaml-options-vanilla` - if no “options” are provided, then the inference is that the compiler should be built with defaults.
The options themselves do not need the `ocaml-option-` prefix. GNU `configure` is widely-used and understood. Its convention is to use `--with-` for add-ons, and `--enable-` for togglable features already available in the build. Following that convention would result in much more readable commands, IMO:
enable-32bit - 32bit build (default: disable)
disable-64bit - do not build 64bit stuff (default: enable)
enable-afl - enable AFL support
disable-bytecode - default:enable
disable-native - default: enable
enable-unsafe-string - enable unsafe-string as default (default:disable)
enable-flambda - enable flambda
enable-fp - enable frame pointers
with-musl - use musl instead of libc
enable-nnp - enable no-naked-pointers mode
disable-flat-float-array - disable flat float arrays in the runtime
enable-spacetime (for 4.12 this option is not available, since spacetime has been removed)
disable-dynamic-linking - instead of `ocaml-option-static`, no dynamic linking
E.g. instead of `ocaml-option-bytecode-only` we would have `disable-native`.
For example, instead of:
```auto
opam switch create foo --package=ocaml-variants.4.12.0+options,ocaml-option-flambda,ocaml-option-nnpchecker
```
we would have:
```auto
opam switch create foo --compiler=4.12.0,enable-flambda,with-nnpchecker
```
---
### Author:  [@jhw](https://discuss.ocaml.org/u/jhw)
#### Post date: [September 5, 2023, 3:42pm UTC](https://discuss.ocaml.org/t/experimental-new-layout-for-the-ocaml-variants-packages-in-opam-repository/6779/15 "2023-09-05T15:42:43Z")
> [@mobileink](#):
>
> Every switch requires a compiler
This is not true. It’s reasonable to use OPAM for packages that do not require an OCaml compiler.
---
### Author:  [@mobileink](https://discuss.ocaml.org/u/mobileink)
#### Post date: [September 5, 2023, 3:44pm UTC](https://discuss.ocaml.org/t/experimental-new-layout-for-the-ocaml-variants-packages-in-opam-repository/6779/16 "2023-09-05T15:44:54Z")
I stand corrected. But I think the point still holds: for switches that do need a compiler, `--compiler` makes sense.