Relocatable OCaml

[dra27]

It is my great pleasure, 3 years to the day since I finalised the first full demonstration of it in Ljubljana for the OCaml Workshop in 2022, to announce the general availability of Relocatable OCaml for testing and review.

For example, this creates a 5.4.0 switch (the good old-fashioned slow way):

$ time opam switch create --repos=relocatable=git+https://github.com/dra27/opam-repository.git#relocatable,default original-compiler ocaml.5.4.0
real    1m27.646s

But, having run that, subsequent switches are a just a tad quicker:

$ eval $(opam env)
$ ocamlopt -where
/home/opam/.opam/original-compiler/lib/ocaml
$ time opam switch create . --repos=relocatable,default ocaml.5.4.0
real    0m5.105s
$ eval $(opam env)
$ ocamlopt -where
/home/opam/myproject/_opam/lib/ocaml

And, um, that’s it really!

Oh, there are a few other things bundled into the relocatable branch on dra27/opam-repository:

• It’s available for all versions of OCaml since 4.08. They install using their “trunk” versions - i.e. the version number of what would be the next release of OCaml (4.08.2, 4.09.2, 4.10.3, 4.11.3, 4.12.2, 4.13.2, 4.14.3, 5.0.1, 5.1.2, 5.2.2, 5.1.3, pre-release 5.4.0 and, for completeness, trunk OCaml is present as 5.5.0)
• Relocatable versions of ocamlbuild and ocamlfind are included, along with patches for omod and stdcompat
• The ocaml-option- packages work for all of the older versions, too
• They all work on Windows (including 4.08-4.12, which aren’t yet available in opam-repository)
• They all work on Apple silicon (including 4.08, 4.09 and 4.11)

I have literally just opened the PRs on ocaml/ocaml (ocaml/ocaml#14243, ocaml/ocaml#14244, ocaml/ocaml#14245 and ocaml/ocaml#14246). In the interests of those of us who maintain libraries and tools which have to target multiple versions of OCaml, there will be some kind of story for keeping these older versions available, but the focus for now will be reaching a consensus for merging the feature itself into OCaml 5.5.

Please do test and report any issues at GitHub · Where software is built. Over-zealous lock-file systems notwithstanding, the compiler should be as “drop-in” a replacement as updating to any “point release” of the compiler (i.e. if you have something which works in OCaml 5.3.0 but doesn’t work in Relocatable OCaml 5.3.1, that’s a bug, and I’d love to hear about it a.s.a.p., please!)

[bentxt]

This seems like a big undertaking. I would be interested in a little more context, such as how and why did came to this project, how can it be used, now that it exists? Thanks.

[dra27]

There’s some more context in Relocatable OCaml - from concept to demo to PRs | Notes from the Windows corner, which includes a link to the RFC.

@bentxt

how can it be used, now that it exists

I’m not sure what to add beyond the instructions above - if you add that repository to your opam switches, and create switches based on OCaml 4.14.3/5.3.1/5.4.0 (etc.) then they’ll be using it. Ultimately the aim is that it’ll just be “OCaml”

[arvidj]

For what it’s worth, I tried relocatable OCaml with 4.14.3 and 5.4.0 and it worked exactly as advertised with an appreciable speed improvement. Thanks for the hard work!

[lindig]

What is the use case? Opam is very much designed to create a personal developer environment so where does the need to move libraries around come from? Or is this more about the executables being produced?

[dra27]

@lindig

What is the use case? Opam is very much designed to create a personal developer environment

The personal developer environment is the use case If you have more than one switch with the same compiler, the compiler is built from sources for each one of those switches. With this, it’s only built once.

@lindig

where does the need to move libraries around come from

Quoting the RFC:

This can be readily seen in opam today:

# Create an opam local switch
dra@Tau:~/work$ opam switch create . ocaml.5.3.0
...

# Report the Standard Library location from both builds of ocamlc
dra@Tau:~/work$ opam exec -- ocamlc.byte -where
/home/dra/work/_opam/lib/ocaml
dra@Tau:~/work$ opam exec -- ocamlc.opt -where
/home/dra/work/_opam/lib/ocaml

# Rename the directory
dra@Tau:~/work$ cd ..
dra@Tau:~$ mv work new-work
dra@Tau:~$ cd new-work

# Repeat the previous test
dra@Tau:~/new-work$ opam exec -- ocamlc.byte -where
Fatal error: exception /usr/local/bin/opam: "execvpe" failed on /home/dra/new-> work/_opam/bin/ocamlc.byte: No such file or directory
dra@Tau:~/new-work$ opam exec -- ocamlc.opt -where
/home/dra/work/_opam/lib/ocaml

[lindig]

I agree that switches are expensive and it’s great to make them more space efficient. In the past I was never sure how safe or unsafe it is to move OCaml files likes .o .cmx around because of potentially embedded paths.

[nojb]

Another use-case is to make binary distributions of the compiler package, instead of having to compile from source. (At LexiFi we have done this for many years with a home-made patch to the compiler, and it works very well; not having to rebuild the compiler each time saves a lot of time… especially on Windows!). opam-bin opam - opam-bin also did this, with its own patch to the compiler, which supposedly now will no longer be needed.

Cheers,
Nicolas

[sid]

How “deep” is the relocatability ?

Say I create a opam switch at /home/abc/mydir and then compile some ocaml packages/libraries in the switch. Can I create a copy of the switch (cp -r) and opam list to see the same libraries installed in the copied directory or is the relocatability only limited to the compiler executables and some internal compiler libs when creating a new switch ?

[nojb]

@sid

is the relocatability only limited to the compiler executables and some internal compiler libs when creating a new switch

The announcement here concerns only the compiler package. Other packages will need to be adapted as needed (many packages that do not depend on compile-time paths already are). In practice this means that now that the compiler is relocatable, some switches will be fully relocatable as well (so that you could copy them en masse), but not those containing packages which are themselves not relocatable.

However, as long as the compiler was not relocatable no switch could ever be relocatable, which greatly diminished the motivation to work on relocatability of individual packages. Now that the compiler is relocatable, the race is on to adapt all other packages to do the same.

Cheers,
Nicolas

[dra27]

From a libraries perspective, the Dune package management project had only found ocamlfind and ocamlbuild needed additional patches, though there may be some other tools in opam which do as well.

I haven’t put too much information out about making user programs relocatable partly because it only affects programs which are installed into switches (which isn’t that common) and partly because the precise details may of course during the review of the relevant PRs (-set-runtime-default et al in #14244 and -runtime-search in #14245).

The most likely way a program ends up with an absolute path embedded - at least because of OCaml rather than its own doing - is because it links with compiler-libs and so has Config.standard_library embedded. To be made relocatable, the program simply has to have -set-runtime-default standard_library_default=../lib/ocaml added to its link flags (and -runtime-search always if it’s a bytecode executable).

If a program needs to do computations of its own to find directories in the switch, it’s a fairly simple call to a new primitive - ocamlbuild is a good demo (it needs to find its runtime library), with a draft at Relocatable ocamlbuild by dra27 · Pull Request #2 · dra27/ocamlbuild · GitHub

[Leonidas]

@dra27

From a libraries perspective, the Dune package management project had only found ocamlfind and ocamlbuild needed additional patches, though there may be some other tools in opam which do as well.

That’s not entirely the whole story. There are a few more packages with relocatability issues but we submitted patches to some (some of which were merged), but ocamlfind and ocamlbuild were by far the ones with the most reverse dependencies. The other failing packages were in many cases rather obscure packages, some of which I have never heard before. So overall the relocatability situation when it comes to OCaml libraries is fairly decent actually.

[sim642]

Perhaps I’ve missed this, but it seems to me that there’s actually two interesting things here.
One of them is the relocatability of the compiler, which allows it to be moved and copied around without breaking.

The other one is opam actually doing the copying/cloning from some existing switch to the new one. The relocatable branch of opam-repository fork is full of some cloning-related variables in the opam files.
Where is this coming from?

Because this makes me wonder the obvious: if the compiler can be cloned into a new switch, then other packages compiled with a relocatable compiler should also be cloneable (assuming they are relocatable as well). Or does this logic not really work for some reason?
And if it works, how would other packages get this ability?

[vlaviron]

@sim642

Because this makes me wonder the obvious: if the compiler can be cloned into a new switch, then other packages compiled with a relocatable compiler should also be cloneable (assuming they are relocatable as well). Or does this logic not really work for some reason?
And if it works, how would other packages get this ability?

You might want to check opam-bin, which does (more or less) that already, and which should benefit from the relocatable work (removing the need for patches on the compiler).

[dra27]

@sim642

Where is this coming from?

The fourth PR in the set addresses this part for the compiler, in particular tools/opam/generate.ml which extends OCaml’s build system to generate a script which can be used to clone the compiler (the script process.sh is then used in the opam package itself).

The packaging solves the compiler’s cloning - a lot of packages are fairly well served by Dune’s caching on top of that.

@vlaviron

You might want to check opam-bin, which does (more or less) that already, and which should benefit from the relocatable work (removing the need for patches on the compiler).

I think so, yes - the nice thing here is that a system like opam-bin provides the infrastructure for precompiled binary distributions. I feel the relocatable work hopefully allows opam-bin then to be a distribution rather than, sort of, a “fork” (because of the patching).

My hope is that with the compiler (the “first” package) being properly relocatable, there’s more incentive to make all packages work that way by default and adopt distribution systems which capitalise on it (which is pretty much what @nojb says above, too)

[SkySkimmer]

Rocq can also be relocatable since 9.1 (although I think the opam package won’t use the relocatable mode until 9.2), this was mostly for the sake of getting cache dune hits across workspaces without thinking about separate opam switches.