Can I use an "opam switch" as an immutable base of another one?

Ecosystem
1

I’m learning OCaml through various sources and trying out various libraries. Maybe it’s overkill but I’m creating a switch/sandbox every time I start a new tutorial/book; so I have to wait while these packages are recompiled each time.
I’m wondering, is it possible to have a base immutable switch with common packages: the compiler, utop, ocaml-lsp-server, ocamlformat, etc.; to be used as the basis of another switch?

In opam online docs I found something related: opam create switch my_project ocaml-system, to create a switch based an OCaml compiler installed in the operating system.

I found no hints on the opam man-page nor on the online manual that this is possible, but I’m still hoping. Can someone confirm, please?

4 Likes
2

I don’t know if this helps, for me
ocaml --version returns
The OCaml toplevel, version 4.13.1
And
which ocaml
/usr/home/x/.opam/4.13.1+options/bin/ocaml
I used an older ocaml “system”-compiler to compile and install a newer “user”-compiler.
The compiler that came with the OS. was:
ocaml-4.05.0_1

3

Hi @devosalain, thanks for the idea. That doesn’t seem what I’m looking for.
I am able to install a ‘switch’ with a new compiler. What I was hoping to do is to have a whole set of packages shared between different ‘switches’ so that I don’t have to recompile those packages every time. For instance, I’d like to have the latest compiler, utop, ocaml-lsp-server, and other packages used for development shared between ‘switches’.

To express the idea in a preudo-shell script:

opam switch create mybase-4.14.0 ocaml.4.14.0
eval (opam env --switch=mybase-4.14.0 --set-switch)
opam install utop ocaml-lsp-server ...etc
# then, in my imagination, I would issue the following command to create
#   a project-specific switch importing all the things in the previous one
opam switch create myproject --with-base=mybase-4.14.0
eval (opam env --switch=myproject --set-switch)
opam install my-project-specific-dependencies
4

I don’t know if it is a wise idea. Do you really need different compilers ?
opam switch list shows in my case an active recent one.
I think in general it is a bad idea to keep old compilers around unless there is a specific use case like old software-applications you don’t want to touch anymore, with future problems, which will arise over time.
You probably have done,
opam switch --help

1 Like
5

Opam can enforce a “switch invariant,” which can specify a set of packages that must be installed in the switch. I don’t think that there’s any ability to share artifacts between switches though.

You could probably make something like this work by creating a docker container for the base switch and layering your project specific stuff on top, but I expect it’d be a pain.

Maybe an opam expert can chime in here. This is a feature that I’d also like. At the moment, project specific switches are really unwieldy.

1 Like
6

I think there has been a misunderstanding. I don’t want to keep around old compilers. I want to share an arbitrary compiler (likely the latest one), and a number of utilities, between different switches by using a switch as the base one, in order to avoid recompiling such packages every time I start a new project.

It’s just for the sake of saving those minutes it takes to recompile those packages every time I create a new switch (and not questioning my compulsion of having a switch per project). As I’m in learning stage, I create many small projects, and having a blank slate every time is somewhat reassuring for me.

Does it make more sense?

7

Perhaps it’s worth questioning this compulsion? If you’re just starting-out, I would guess you’re not installing packages you write into OPAM-controlled directories, so what’s the upside of having separate switches per project? What do you gain?

Also, AFAIK, OCaml isn’t “relocatable” in this sense – you can’t build an OCaml with a certain “destdir” and then copy it elsewhere and have it work.

8

You should have a look at opam-bin.

1 Like
9

At the moment it’s just for my peace of mind. I don’t want to worry about running into dependency conflicts or documentation-implementation inconsistencies.

I also got the impression that one-switch per project is encouraged in OCaml-land from CS-3110 text book:

Create a switch for this semester’s CS 3110 […]

and from Real-World OCaml:

The next thing we need is a suitable environment for this project, with […] any […] dependencies available. The best way to do this is to create a new opam sandbox, via the opam switch create command.

I’m ok creating one switch per project even if it means recompiling every-time, I was just hoping there was a way to avoid recompiling package-versions that are already present in the opam installation (nixOS is a source of inspiration here).

10

You can get what you want by using esy.sh/. But of course it’s a different tool than opam. It works great though.

3 Likes
11

Is there a place to track this feature request? (Or turn it into an official feature request?) I am also quite interested in sharing build artifacts across switches, but more for space reasons than time reasons. I have nearly 100 different versions of Coq that I keep built so that I can debug issues for various projects across Coq versions, but I don’t have a spare 50–200 GB of disk space free for duplicating the base compiler and Coq dependencies a hundred times over.

(Does esy.sh support adding alternate remote repos, the equivalent of opam repo add coq-released https://coq.inria.fr/opam/released?)

12

I doubt that this is possible. But if you want this, have you considered using docker? The ability of docker to snapshot the filesystem at intermediate points might be enough to make this efficiently doable?

ETA: in OCaml’s defense, it’s pretty rare that any compiler toolchain+packaging system supports this. When it’s supported, it’s usually via assuming the toolchain is outside the packaging system.

ETA2: a different way of achieving this, is to use the ability of findlib to have multiple repositories on a path (the OCAMLPATH environment variable). Then you could install each version of Coq into a separate repository, and set OCAMLPATH when you wanted to use it. Probably though, opam doesn’t support this – so you couldn’t use “opam install” build/install Coq, but instead would have to do it “manually”. But that might not be so bad if it’s just Coq. If you need to install different versions of other packages in addition to Coq, for each of these variants, then … that could get painful.

13

Docker is pretty heavy-weight, I’d prefer not to have to do everything inside docker containers. (Plausibly the most difficult thing to set up inside docker would be getting a vscode server working? But also I imagine docker adds RAM overhead, and I’d rather not be running docker inside WSL inside Windows. Especially if I want to be running two versions of Coq at the same time to compare outputs.)

ETA2: a different way of achieving this […]

Right, right now the thing I have set up is that I use opam for managing ocaml versions, and manually compile, install, and manage the Coq versions with some symlinks. To switch Coq versions, I just change one symlink. This is not so bad when I don’t need anything that depends on Coq, but if I want to install a couple dozen Coq libraries with interlocking version constraints, it’s a bit of a pain to manage.

14

Ah, I would suggest that you should not use symlinks. Instead, you should use environment variables. I’ll offer an example. I use this myself, but have never made it public, so …YMMV: GitHub - chetmurthy/Sandboxes: Perl tool for dealing with many Sandboxes

The idea here is: I have a bunch of “sandboxes”. A sandbox is defined by a directory, and then a bunch of environment variables that get set or updated. So I can add to PATH, set OCAMLPATH, etc. This happens in bashrc scripts. So the sandbox fzf is defined (in rc.yaml):

  fzf:
    directory: '$HOME/Hack/FZF'
    rc: '$HOME/Hack/FZF/dot.bashrc'
    varname: FZFROOT

which tells you an rcfile to source before firing up whatever program you want to run in it. And the sandbox Opam is:

  Opam:
    directory: '$HOME/Hack/Opam-2.1.2/GENERIC'
    rc:
     -  '$HOME/.sandbox/bashrc/aliases.bashrc'
     -  '$HOME/Hack/Opam-2.1.2/GENERIC/dot.bashrc'
     -  '$HOME/.sandbox/bashrc/ocaml-base.bashrc'
    varname: CAMLROOT
    description: "Opam 2.1.2 installation"

So now I can do:

$ RUN -s fzf,Opam bash

and that’ll run the scripts for these two sandboxes, and then fire up an interactive shell. So I can have a ton of such sandboxes. I also have “composite” sandboxes, e.g.:

  OCaml:
    - fzf
    - Opam
    - OCaml-Python-VENV

so RUN -s OCaml bash will run the scripts for all three of these, in order.

Anyway, I find that this is a pretty convenient way to manage a large number of somewhat-related environments, without the danger of somehow mixing them up in untoward ways.

The RUN script also sets the prompt and some magic string that gets blurted by bash to set the title-bar of the window in x-windows and Chromeos, so I can see what the current sandbox (or sequence of sandboxes) is in the titlebar.

ETA: of course, if I enter a sandbox, viz.

$ RUN -s fzf bash

then in that interactive shell I can enter another, viz.

$ RUN -s Opam bash

and that will be the same as if I did

$ RUN -s fzf,Opam bash
1 Like
15

This probably will not help the majority of users, but if you can use a rather sophisticated copy-on-write filesystem like btrfs on Linux, you can actually achieve this: Working with Btrfs - Snapshots - Fedora Magazine

Btrfs snapshots work different than file copies: They keep references to current and past inodes instead. When you appended the change to the file, under the hood Btrfs allocated some more space to store the changes in and added a reference to this new data to the original inode. The previous contents remain untouched.

So basically,

  • Create a new btrfs subvolume. This appears on the system like a directory.
  • Set up the base switch inside this directory. Install whatever packages you want in this base switch.
  • Create snapshots of this subvolume for each new project. Each snapshot will start off with the exact same installed packages. Then install anything else you want in the individual projects. The original subvolume remains unchanged but they all share most of the disk space because it’s a copy-on-write filesystem.

EDIT: yeah, I forgot that the OCaml compiler is not relocatable, so the above probably won’t work :sweat_smile: Afair it’s mostly because the full path to the compiler is hardcoded into bytecode executables.

1 Like
16

To answer the original question: no, opam does not currently support sharing build artifact across switches. One reason why this is not obvious to implement nicely is that the OCaml compiler is not relocatable by default, its behavior depends on the path it was installed into, so removing or updating the “parent” switch could break “children” switches that reused the parent compiler – even if the build artifacts were copied when creating the children switches. (Esy works around this problem with a path-rewriting hack. It is also possible to play with the filesystem to pretend that all switches are at the same path.)

@dra27 has been working on making the OCaml compiler relocatable, see Relocatable compiler work , which would open a way forward to binary caching without hacks or restrictions.

2 Likes