Opam command for a local switch - How to use it

Learning
1

In short:
Do I use correctly Opam for managing a local switch? (in order to get “a clean switch per project in sync with the dependencies specified in the opam file”)
This follows the recommendation from @Leonidas. See https://discuss.ocaml.org/t/connection-between-libraries-in-opam-dune-and-findlib/2536/6

__
1/ Switching between local and global switches
Playing with Opam 2 switch commands, in order to understand them, I created an Opam switch local to the project called project2 and located in /home/test/Documents/project2/.

$ opam switch create ./ ocaml-base-compiler.4.06.1

This creates an _opam directory with all the required stuff:

$ ls
_opam

It seems that since there is this a new switch local to this project, the opam switch command doesn’t have the same behaviour:

$ opam switch 
#    switch                compiler                      description
→  ~/Documents/project2    ocaml-base-compiler.4.07.0    ~/Documents/project2
      4.02.3               ocaml-base-compiler.4.02.3    base-bigarray.base base-ocamlbuild.base
                                                         base-threads.base base-unix.base
                                                         ocaml-base-compiler.4.02.3
      4.06.1               ocaml-base-compiler.4.06.1    4.06.1
      4.07.0               ocaml-base-compiler.4.07.0    4.07.0

Then:

$ opam switch 4.06.1 && opam switch
(same result = the global 4.06.1 switch is NOT selected. The local switch is still selected.)

Finally, the following command worked for selecting a global switch:

$ eval $(opam env --switch=4.06.1 --set-switch)
[NOTE] Current switch is set globally and through the OPAMSWITCH variable.
       Thus, the local switch found at ~/Documents/project2 was ignored.

And it also worked for selecting again a local switch:

$ eval $(opam env --switch=~/Documents/project2 --set-switch)
[NOTE] Current switch is set globally and through the OPAMSWITCH variable.
       Thus, the local switch found at /home/test/Documents/project2 was ignored.

Q: Is there a way to keep things as simple as $ opam switch [name] from within /project2 directory where there is an opam (or project2.opam) file or from anywhere else, where [name] is the name of the global switch or the path to the project where there is an opam file ?
Q: Why is an opam file created whereas we see in many projects a foo.opam file (foo being the name of the project)?

__
2/ Installing the packages dependencies for a project and keep it synchronized
The command that finally makes sense for me is (from the local directory of the project):
$ opam pin edit project2
Because it fires up a text editor with an opam file template.
Then accepting the last [Y/n] choice triggers the installation of the packages defined in thi local switch.
Save the new opam file back to "~/Documents/project2/opam"? [Y/n] y
This opam file is saved here (resulting from an opam pin command):
~/Documents/project2/_opam/.opam-switch/overlay/project2/opam

EDIT:
To get the packages specified in the opam file, I use the following command within the project directory that seems logical:
$ opam install .

Q: Is this the right way to install dependencies for a project?
If I update the opam file, the installed Opam packages that become useless are kept in the switch. Instead of manually remove them, is there a sync command?

__
3/ Repairing a "broken switch"
When only working with these Opam commands, I encountered several times the following problem in some switches (local or global):

$ ocaml
Fatal error: cannot load shared library dllunix
Reason: /home/test/Documents/project2/_opam/lib/ocaml/stublibs/dllunix.so: undefined symbol: caml_ba_element_size

Q: What is broken? (I’m not enough familiar with Ocaml and Opam)
What caused this problem?
How can I repair this broken switch instead of removing and reinstalling it from scratch? ($ opam switch remove project2 && opam switch create ./ <package-or-version> This fixes the problem but gives no explanation)

2

Local switches use is intended to be streamlined so that when you enter a directory where there is a local switch, opam will automatically switch to it. If you go out of the directory then you can switch between your global switches again normally (but not your local switches). I think this is using the assumption that when you’re working on a project (in this case project2) you would rarely need to switch to another switch and always use the local one. That’s why you’d need to “force” opam to switch with the --set-switch flag if you really want to.

At which step was the file created? I don’t think I’ve encountered this before.

We usually just write opam files by hand since it’s oftentimes quite bearable for new projects (the fields are defined here, and the minimum required opam file could be as simple as this). If you have the opam file ready, then yes, opam install . is correct. You can also install only the dependencies and not the package itself, by using opam install --deps-only ..

Syncing pinned packages can be done by running opam update <pkgname> && opam upgrade <pkgname>. This is the same as if you want to upgrade any dependency.

I’m not sure, but can you confirm that you’re not using a switch called system (i.e. using OCaml not installed via opam, but via OS package manager)? That often causes issues.

3

I understand how Opam is intendend to behave. But this is not the case in my context: when I go out of a directory where there is a opam file, I can’t no more enable a switch.

Would this env var be the cause?

 $ printenv OPAMSWITCH
4.06.1

I create the opam file with the following command (looking for a template-based command):
$ opam pin edit project2
I understand that you manually create a foo.opam file for the foo project.

BTW: as if I use Dune, I should get rid of this template part generated by the opam pin edit command that seems useless?

$ cat opam
...
build: [
  ["./configure" "--prefix=%{prefix}%"]
  [make]
]
install: [make "install"]

By syncing, I mean not only installing the dependencies (as with a global switch) but ALSO removing those that became useless. The only mechanism I see is by manually removing them:
$ opam remove <p1 p2> . #from the project directory where there is the local switch
$ opam remove <p1 p2> #when a global switch is selected

Is there a real sync mechanism that strictly sync the installed dependencies with what is specified in the opam file (by removing not specified packages)?

No I,m not using the switch “system”.
I tried one time to select the system compiler with the $ opam switch system command, which has no interest when using Opam, then made the tests I reported with freshly installed switches where I had this problem.

4

Additional tests:

From the projet root directory with a valid opam file there, and with a global switch selected, the following command tries to install in the global switch the dependencies specified in the projet2/opam file:
~/Documents/project2$ opam install --deps-only .
In my context, the opam command doesn’t behave in the streamlined way you described.

I tried to cancel what I did ( $ opam pin . then $ opam pin edit project2 ) with the following command:
$ opam pin remove project2

Then I tried intentionally something weird:

$ opam switch .
[ERROR] Can not set external switch '~/Documents/project2' globally. To set it in the current shell use:
         eval $(opam env --switch=~/Documents/project2 --set-switch)

That’s this Opam hint I used to get out of my situation where I couldn’t selecte a global switch anymore.

$ opam install .
project2 is now pinned to file:///~/Documents/project2 (version 0.1)

So I see that pinning is used and should make sense.

Does it help you to see what I’m doing wrong, or if we can identify a lack in the Opam documentation or suspect an error in Opam code?
Thanks.

5

Just answered most of this on opam’s tracker: Trouble when using Opam local switches · Issue #3573 · ocaml/opam · GitHub

To answer some of your additional questions:

That’s this Opam hint I used to get out of my situation where I couldn’t selecte a global switch anymore.

yes, OPAMSWITCH actually locks the switch in use, which is why --set-switch isn’t a good idea by default. unset OPAMSWITCH is enough to get out of there.

Q: Why is an opam file created whereas we see in many projects a foo.opam file (foo being the name of the project)?

It’s historical, dune recently enforced the choice of foo.opam, but dune didn’t exist and just opam was the default at the time :). This makes sense for single-package source trees, but we should probably update to unify this.

Ha, we could probably skip the second part of the message when the two are actually the same…

Q: Is this the right way to install dependencies for a project?
If I update the opam file, the installed Opam packages that become useless are kept in the switch. Instead of manually remove them, is there a sync command?

opam install . will always sync. Packages that have been removed will be unpinned, as well as removed pinning dependencies. Then opam remove -a should remove everything that is no longer necessary.

6

Just gave the result of your tests commands: https://github.com/ocaml/opam/issues/3573#issuecomment-426398738

Ok. fixed.

$ opam pin ./ #within a foo/ directory
then
$ opam pin edit foo
calls a text editor with an opam file template then save it as afoo/foo.opam file and triggers an opam update.
Even if these commands I used no more make sense, Opam should be changed to generate a correctly named opam file. foo.opam

@bobbypriambodo told us he uses neither template nor tools:

Test: I added a package in the the opam file then did $ opam install . and it installed it.
Then I removed this package from the opam file and it stayed in the switch as shown with $ opam list.
So it means that Opam automatically installs required packages. But it doesn’t automatically remove useless packages. So that’s not sync (between the packages available in the switch and the Opam file “depends” field specification).
What do you think about changing that to get a real sync? (it means adding packages auto remove).
Of course some people who installed other packages in the switch and wants to keep them could say that it’s not a good idea to remove them.
But a real sync looks very simple and clear: the opam file drives the switch configuration via a simple $ opam install . command.

7

PS
Can you clarify what opam pin is and is not?
And in which situations it is useful?

8

opam pin creates and manages “pins”, which are custom package definitions local to a given switch. To quote the manual:

   This command allows local customisation of the packages in a given
   switch. A pinning can either just enforce a given version, or provide a
   local, editable version of the definition of the package. It is also
   possible to create a new package just by pinning a non-existing package
   name.

I believe the confusion arises from the new opam install DIR command. What is important to understand is that this command is actually a pin underneath: it was added for convenience, and does a little bit more (always sync the package definitions, and, obviously, install), but for simple cases with a single package definition, opam install . and opam pin . are basically equivalent.

opam pin subcommands can then be used to manage the pinned packages, try e.g. list.

Another thing that is important to understand is that pinning (and install DIR) are completely orthogonal to local switches: they play well together, and were conceived in synergy, but they are two independent features.

Opam should be changed to generate a correctly named opam file. foo.opam

The file created is correct. Well, it won’t be recognised by dune, alright, and as I said we should work on unifying this, but opam will handle it just fine.

1 Like