Improving interoperability between ocaml, ocamlfind and dune

Ecosystem
,
1

In the OCaml library ecosystem, there are two important pieces of software that are being relied on for managing our code:

  • ocamlfind, lead by Gerd Stolpmann has been the standard for managing OCaml libraries for many years. It provides a convention for storing libraries on the filesystem, and a META file format for describing the properties of libraries. There is a CLI tool that interprets META files and makes it easy to embed within build systems.

  • dune, lead by @jeremiedimino, is an specialised build system for OCaml that aims to be fast and expressive. Dune’s functionality complements but doesn’t fully replace ocamlfind – it can interpret META files as installed by ocamlfind, but it also now has newer features such as virtual libraries which cannot be expressed by ocamlfind. Dune installs a dune-package file with every library that contains its full metadata, and translates the compatible subset into the META format as well.

This has lead to some incompatibilities in recent months, since independent build systems now need to handle two separate formats (META and dune-package). Therefore, the authors of both systems have gotten together to make the following changes to improve the consistency of our community:

  • ocamlfind has moved to github.com/ocaml/ocamlfind from its former home at GitLab. Gerd will work on migrating links and merge requests from the old repository to the new home. This also belatedly recognises ocamlfind as a very important piece of the OCaml Platform and ecosystem, by hosting it at the ocaml/ GitHub repository alongside the compiler, opam, dune and the ocaml.org website.

  • There is a new ocaml/ocaml-library-standard repository where we will work on a documented standard for both dune and ocamlfind to use for the purposes of clean interoperability. This is just starting up this week, and so we will give Gerd and @jeremiedimino some months to establish the common ground between the projects. If anyone has specific asks for features that are not covered by ocamlfind and dune at the moment from a packaging perspective, then this is a good time to get in touch. Note that new feature requests may not make the first version of the spec – we are keen to ensure that the existing status is reflected initially to solve the immediate problem, and so the focus is on documenting the existing interactions between the OCaml compiler, ocamlfind, and dune.

16 Likes
2

It’s really good news.

Opam is obviously another key OCaml tool.
I should admit that I’m not 100% clear about how the packages are exactly handled at the lowest level, especially when coming to the cooperation between dune, ocamlfind and opam.
See Connection between libraries in Opam, Dune, and Findlib
Especially about the "1-1 mapping between opam packages and findlib libraries enforced by dune to make the difference (between packages and libraries) manageable ".

Obviously, the ideal solution would be to get only one package&libraries file for each Ocaml program in order to feed opam and dune with reliable information (for package and build management, respectively).

Can you tell us if the ongoing effort you present will solve that?
Or if another effort should be made, and give us hints about how to contribute?

Thanks.

1 Like
3

Hi @Luc_ML,

dune itself has features these days to generate opam files from dune metadata – see ocaml-github for an example project that uses this functionality (or of course, dune itself). This all greatly helps with reducing the amount of boilerplate we have to maintain, and is an incremental step on the way to reducing the number of namespaces we have without breaking backwards compatibility.

But please bear the above in mind. We’ll first be working on getting ocamlfind and dune to play well together, and then look at further improvements after that. Each step is quite delicate and needs to not break any existing software.

4

I understand that you focus on solving the immediate problem caused by ocamlfind/dune incompatibilities.

In parallel, is there any current discussion about future requirements regarding an “integrated package&build OCaml solution”, that could feed a kind of candidate roadmap?
Do you have an idea of a next must-have requirement in that field? (I’m not talking about doing it now, just planning smth)

Thank you for that useful pointer.

5

We’re planning this in the context of adding namespaces to the compiler and tools, but this is not yet ready. We’ll likely run it through an OCaml core developers meeting sometime next year and then post about concrete plans for integration then.

6

As the author of a few released (odig and omod) eco-system tools and a few yet unreleased ones (e.g. brzo and b0caml) I have given the subject quite a bit of thought and have been compiling OCaml software without additional metadata or ocamlfind for quite some time now without any problem by relying on the fact that OCaml compilation objects are largely self-describing.

While these techniques work, they can certainly be made more efficient and reliable by redefining a bit how OCaml libraries get installed and adding a tiny bit of new metadata to library archives.

I’m largely convinced that we do not need an additional metadata format for the simple task of using libraries and that this problem should be solved upstream – since it then improves the usability globally from using the bare compilers to the toplevel.

I have gathered my thought about this and a possible simple plan for action in
this gist. The proposed scheme has the advantage of putting the eco-system on a good conceptual and naming diet with each of the names involved appearing at the file system level which is good for usability (e.g. which library name should I specify to use this module can be answered by looking up the file system).

8 Likes
7

Indeed, I have also wondered if ocaml can use/cannot use similar mechanism as CLASSPATH mechansim of java. I am not sure what additonal functionality ocamlfind provides in addition to finding installed ocaml libraries since these days I only just interface with dune/opam. However, for the task of discovering and finding installed ocaml libraries, couldn’t a proposal like this(Implement `OCAMLPATH` by dbuenzli · Pull Request #8946 · ocaml/ocaml · GitHub) be enough? Not sure how far off the implementation of namespaces proposal is, but after reading it I sense the OCAMLPATH proposal is in similar spirit and part of it.

8

In fact reading @dbuenzli 's gist I found myself wondering–what if libraries were just modules. The way I see it, we don’t really need a new concept–modules already provide hierarchy and namespacing. Then the question of:

…can be answered by the fully-qualified module name itself.

1 Like
9

That’s a good theory. But reconciling it with reality is another story ;-).

10

The proposal sounds good to me, the only thing is:

$(opam var lib)/stublibs/dllMYLIB.so # Or wherever appropriate

which goes a bit against the model of package management of Nix/Guix (each
package in its own directory, and then setup environments with selections of
packages). Maybe there should be another standardized environment variable for
this?

(I don’t use Nix for OCaml stuff, but usually making things Nix-friendly brings
good “cleanliness benefits”)

11

@smondet in theory you can put that anywhere. However there will then be some kind of environment variable you will then have to extend with the location of each new library which is not that nice.

See the OCaml manual for details.

12

Is this API exposed for writers of configurator-like dune extensions?

13

Do you know any particular prior work that explains the challenges with developing a packaging system that is derived from the module system?

1 Like
14

Note that formally we are not talking about a packaging system here but rather install conventions or rules.

I don’t think there’s any particular challenge if you start from scratch. The problem is that the eco-system is already in place, the compiler already has its own way of working. You can’t break everything and upstream is especially (and rightly) sensitive about this.

As such it’s better to move in small reachable steps. For example there’s nothing in what is being proposed that prevents to eventually get to something like @lpw25 namespace proposal. I would rather say that it paves the way to it.

The proposal is really nothing new or revolutionary. It’s just constraining the installs and adding the missing metadata bit to library archives that makes it easy to do the only thing that ocamlfind does for you in 99% of the cases directly in the ocaml compiler.

Doing this upstream has the effect that it becomes possible to use libraries directly with the compiler itself and in the toplevel, without the need for third party tools (personally I have no love for these crufty #use "topfind" or #use "omod.top").

Astonishingly the ocaml compiler was never able to this except manually which of course doesn’t scale. The only reason for this is that it lacked a notion of dependency between library archives (which is currently provided to you by ocamlfind's requires field in META files) and this is the main gist of the proposal.

But the attentive reader will also realize that the proposal manages to collapse opam package names, ocamlfind names, install directory names and library archives names to a single name which is good for the usability of the system (some will complain we lost in flexibility, I argue this flexibility is not needed in practice and harmful from a usability point of view).

9 Likes
15

From a cusrory look at the description it looks like dune’s virtual libraries are a special case of ocamlfind subpackages. Is this how it is implemented?

16

What dune calls virtual libraries and implementations is nothing new. They have existed in the eco-system for almost as long as ocaml has (e.g. the threads and (now gone) vmthreads libraries).

What is new is the ability to automatically select implementations via a variant tag which you can see as icing on the cake. This is not supported by ocamlfind nor by my proposal (I don’t think it would be hard with more metadata, but I also think it might be a bit premature).

17

That is true. We never fundamentally change the way things work with Dune. It was already possible to have virtual libraries before Dune. But to be completely fair to the people who did the work to generalize and formalise virtual libraries in Dune, there were actually some real problems to solve and work to do. For instance, if you try to use virtual libraries without Dune, it’s likely that you will have to think carefully about the installation layout of your files and the final user will have to be careful about the order in which they list their libraries at link time. This is something Dune users don’t need to think about because Dune handles all these low-level details for them.

To be clear, the majority of Dune users shouldn’t care much about this discussion. The real feature Dune provides to its users is the Dune language; a simple and high-level language to describe their project. In this language, libraries and executables are first class citizens and this allows developers to reason at a higher-level without having to think about the low-level details. Compilation commands, META files, layout of installed files, etc… simply form the assembly the Dune language compiles down to. In this regard, Dune has succeeded and it is now possible to write complex OCaml projects without knowing what is a .cmi file or what is the difference between ocamlc and ocamlopt.

Now, there are a few things we need to acknowledge. First of all, Dune is relatively young. It is only 3 years old and there were other tools and standards that existed long before it such as ocamlfind. Dune also has flaws. For instance, if your project doesn’t fit in the Dune language, then you just can’t do much. Your best bet is to come and discuss with us to see how we can move forward. While we have been thinking about this question for a while, and in particular how to provide users some sort of extensibility this is simply something we have not prioritized so far. If you are a seasoned programmer and like to have a lot of power over how your project is built, Dune can feel a bit invasive.

Finally, there are also cases where Dune projects need to be used as part of a bigger system, such as big bazel projects. In such cases, it can be valuable to have a few more tools such as ocamlfind to tie things together.

This is what this effort is all about. Acknowledging the history, bringing together people who have worked on these problems in the past and agree on a standard to make life easier for OCaml developers. And to be clear, if this standard could be just Dune then that would be great from Dune’s point of view and would make our life much easier, but that’s effectively not today’s reality. While Dune is now widely used, there are still a lot of projects that don’t use it.

3 Likes