Connection between libraries in Opam, Dune, and Findlib

Learning
,
1

I would like to understand the connection between libraries and packages better and would be interested in plans to close the gap between them.

Dune knows which OCaml libraries to make available during compilation of a project and it can report them with dune external-lib-deps globally or per package (-p) for a source tree. The names it reports can be fed to ocamlfind query to locate them. However, the names only approximate the corresponding Opam packages. For example, threads belongs to Opam package base-threads. So the connection between a library and the Opam package that installs it is not obvious (to me). Is there a way to establish a connection or plans to establish it? For example, could the META file for a library optionally name the Opam package that installed it?

Addendum. These recent discussions seem to be relevant:

7 Likes
2

Hi,

In dune files, everything that is in (libraries ...) refers to findlib names. That is the same as what as in package(...) in _tags with ocamlbuild.

This does not always involve findlib: within a project, dune will be able to resolve libraries by itself, for example when linking your test suite against your lib.

Strictly speaking, the library names (for findlib) and opam package names are not related. For packages using topkg or calling ocamlfind install directly, it is up to the package author to specify the exposed modules, the library name, and the package name.

Base libraries in particular do not use strictly the same name. That is also the case in some common libraries: for example the ounit package exposes a oUnit library which has a OUnit2 module.

dune is a bit less flexible: the library name is always the same as the package name. That is derived from the (public_name).

opam used to have some support for exposing the mapping between libraries and packages: findlib files. It seems that it does not exist anymore though.

To avoid the confusion, the best practice these days is to make packages expose a single library, which exposes a single module, all named the same (that’s what you get when using dune with the default (wrapped true) mode).

3 Likes
3

In summary, I’d say the situation is more confusing (for historical reasons) than one would like and hence the efforts to streamline this by Dune potentially generating Opam files.

4

I’d say that dune already keeps this difference manageable by enforcing a 1-1 mapping between opam packages and findlib libraries. Generating opam files might help, but in the linked proposal the dependencies in dune-project are opam names so this is a purely syntactic generation (at least for now).

5

Dune complains when a library is known to findlib and used in code but not imported by a jbuild file. But it does not notice when a library that is used is not mentioned in the corresponding Opam file and hence available only by accident.

6

This is true. As a stopgap solution I recommend using clean switches per project and only install packages via dependencies declared in the opam file and avoiding to call opam install … directly.

7

Dune should have enough information so that it could warn when a dependency is missing in an opam file. However, opam files are complicated to interpret. Putting the information in the dune-project file should help with implementing these checks.

Currently, the solution @Leonidas suggests is indeed the safest one.

8

Thanks for this wise recommendation that gives us light (and hope!)

Can you detail a little bit about how to do that?

Currently I’m using opam for installing packages “as recommended in all tutorials” (opam install <package>).
And also dune for building (so I know the syntax of a dune file).
Is the opam file you mention like the one detailed here? https://opam.ocaml.org/doc/Tricks.html#Provide-a-set-of-packages-for-a-group-of-users-to-install?

Do I need to use opam 2.0 to create “one clean switch per project” ? (I’ve just upgraded this afternoon, assuming it makes possible to create one switch per project/program - and possibly several switch with the same compiler - )

And which command is required to get the packages/libraries available (for coding and for compilation)? Or maybe dune simply does the job? (I can spend some time in the opam documentation but this point should be obvious for you)

Also, can you explain clearly and exhaustively what is a package and what is a library?
See example from @emillon:

the ounit package exposes a oUnit library which has a OUnit2 module

AFAIK what is an (Ocaml) module is clear for me: a Foo module can be represented by a file foo.ml ; a Bar module car be related to a Bar module declaration ( module Bar : BAR = struct … end ) inside the the foo.ml file of a Foo module, and accessed by Foo.Bar .

Thanks

EDIT:
I’ve read some opam documentation, especially http://opam.ocaml.org/doc/Manual.html#Local-configuration-files.
I upgraded to opam 2.0.0.
I did

$ mkdir ~/project ; cd project/
$ opam switch create ./ ocaml-base-compiler.4.06.1

<><> Error report <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
┌─ The following actions failed
│ λ build ocaml-base-compiler 4.06.1
└─
┌─ The following changes have been performed (the rest was aborted)
│ ∗ install base-bigarray base
│ ∗ install base-threads base
│ ∗ install base-unix base
└─

Inspired by https://github.com/ocaml/opam-repository/issues/12050 I tried with 4.02.3 but got the same error message.
Opam 2.0.0 has just been released last week and there may be some traps.

This opam local repository setup should be obvious for you. What do you recommend?

EDIT2:
I did again:
$ opam install opam-devel
and I got a clear message from opam that I didn’t have before (but thanks for that!):

# The ‘bwrap’ command was not found. Install ‘bubblewrap’ on your system, or
# disable sandboxing in ~/.opam/config at your own risk.

So I did the following that solved my opam 2.0.0 installation issue (if it may help s.o.):
$ sudo apt-get install bubblewrap

Then I did:
~/project$ opam switch create ./ ocaml-base-compiler.4.06.1

Done.
<><> (All…) installed successfully <><><><><>

And it seems that I have now a local repository switch for project.
As opam switch documentation tells:
$ opam switch create --help

DESCRIPTION
…
When creating a directory switch, if package definitions are found locally, the user is automatically prompted to install them after the switch is created unless --no-install is specified.

Now, to follow @Leonidas recommendation, I understand that I “just” need to write a file ~/project/opam and redo
~/project$ opam switch create ./ ocaml-base-compiler.4.06.1
“to get prompted to install them after the switch is created”.

I hope these details could be helpful for other people in my situation.

1 Like
9

It is confusing indeed. In a nutshell, modules are an OCaml abstraction, libraries are a findlib abstraction, and packages are an opam abstraction.

Modules: you got this. They are the larger unit of code in OCaml. They can contain types, values, exceptions, other modules, etc. You can define one using module M = struct ... end and everytime you compile a file named foo.ml, its contents correspond to the body of the module Foo.

Libraries: as you may have noticed, when you run the ocaml compiler, it creates some binary files, like cmi (compiled version of .mli), cmo (bytecode version of .ml), or cmx (native version of .ml). These files can be linked into an application. That’s how you get separate compilation of third party code (Core or Containers is not rebuilt every time you build your application). To link them with an application, these files need to be given a global name: this is the role of findlib. The name is their library name. This step is comparable to storing a .a file in a global directory.

Packages: libraries are a useful abstraction, but they’re not the whole story. Suppose I have an application that requires Containers. What is Containers? Where do I get it? How do I build it? Do I need some other packages to do so? This is the role of opam and packages. An opam package is a source tarball and a metadata file with all that info. The opam program can interpret it to install dependencies for a piece of software (executable or library), build it, and install it. In the case of a library, opam will install cmi, cmx etc files into the correct directory so that it can be found (via findlib) by other pieces of code.

Hope that helps. Let me know if I should expand on some parts!

20 Likes
10

This should be a FAQ answer on OCamlverse.

4 Likes
11

Are the efforts for this going forward?

Is there a document somewhere that says how to set up a new project like this? We’re starting one and everyone is just installing project dependencies with a bash script right now, but it would be nice to have a way to pin dependency versions (maybe with the .opam file, and then use opam to install dependencies even as we use dune to build?). Any tips or links here appreciated!

12 
git checkout …
cd project
opam switch create ./ ocaml-base-compiler.4.07.1

Will create a switch and automatically install all the dependencies declared in the opam file in the right versions.

4 Likes