Struggling with Dune and Opam (how to meet dependencies and install then globally)

Hi,
I try to build a project template (1) that uses Dune and Opam. After circa 5 minutes of my PC fans spinning full steam (git seem to be really busy), Opam has placed a _opam folder in my project that is over 5 GB in size! So basically my question is: Is there a way those deps can be installed globally in order to save space and time?
[1]: GitHub - melange-re/melange-opam-template: A simple project template using Melange with opam

here is the opam command that seams to install all the stuff:
opam switch create 4.14.1 -y --deps-only

While searching for solutions I stumbled over the following articles:

  • So in one article (2) they where talking about --set-switch in order to install globally:
    opam env --switch=4.06.1 --set-switch
    [2]: - Opam command for a local switch - How to use it

  • The official Dokumentation talks about opam switch set but that doesn’t work either.

I think one reason why I struggle is because I don’t really understand the concept of switches.

Ok, the other problem is that I cannot meet the requirements. Here is the error:

dune  build @react @node
File "dune-project", line 1, characters 11-14:
1 | (lang dune 3.8)
               ^^^
Error: Version 3.8 of the dune language is not supported.
Supported versions of this extension in version

I tried to install dune 3.8 manually but I how can I use it here?

Sorry for sounding so complicated, but nothing seems to work.

1 Like

I don’t actually know how local switches work (so your knowledge might be the complement of mine grin). Instead, I have only worked with global switches. B/c they’re so straightforward.

[as an aside: I don’t think you can install anything globallyand then work with local switches: a switch is self-contained and cannot share anything with another switch: after all, each is based on a compiler/runtime/toolchain build]

Maybe try starting from-scratch? Blow away your opam root directory, redo opam init, also blow away any _opam directories, and then create a few switches? I routinely blow away all my switches and do opam switch create X.Y.Z. I also have a bash alias

function refresh_switch {
 eval $(opam env)
}

so I can do

opam switch X.Y.Z
refresh_switch

and my environment is updated.

Don’t knoow if this helps, and if not, I’m sorry. I really do not know how local switches work.

Why using --deps-only ?
Use global « switch » as recommended by Chet (see many comments about local switches). Keep it simple.

Concerning dune…, well, if the sw is configured with dune AND fully tested by the author, it should/may work!
But you are not forced to use dune to compile and link your OCaml program. A Makefile does it very well without doing weird things in your back; and there are even tools to generate it, if needed.

Can you tell us the exact commands that you ran and give the exact outputs of those commands? Or at least a reasonable extract of the output if it is very long. It’s very difficult to help if we don’t have the correct information.

Hey Chet,

thanks for your advice: I will start from scratch …

To be honest, I’m a little confused about your other comments (local vs global switches)

To avoid any misunderstanding, here is what I would like to do: I’d like to have dependencies built on my (global) ~/.opam directory, and not locally in my project under _opam. Does that makes sense?

[Again, I don’t use local switches at all, but …]
From what I understand, the presence of a _opam directory indicates that there is a local switch in there. This is why I’d suggest you blow away your ~/.opam directory, all _opam directories, and start from-scratch. [BTW, since you might have copied files into /usr/local/lib, be sure to remove those, too].

Then recreate your opam environment, and then your switches, and then you can install packages into those switches.

P.S. just to be really anal, it might be a good idea to logout and log back in, so that all environment variables are erased. Heck, after you blow away all the directories, you might want to reboot, to blow away any system-level caches. I don’t know what MacOS does in this regard, which is why I’m suggesting to be so … anal.

I think I found a simple way to reproduce the problem with dune:

> ocaml --version
The OCaml toplevel, version 4.14.0
> opam install dune
dune --version
3.7.1
opam install melange
> opam list -a | grep  melange
melange                                         0.3.2       Toolchain to produce JS from Reason/OCaml
melange-compiler-libs                           0.0.1-414   Compiler libraries for Melange, a toolchain to produce JavaScript from OCaml
> find ~/.opam/ -type d -iname '*melange*'
/Users/ben/.opam//default/.opam-switch/packages/melange-compiler-libs.0.0.1-414
/Users/ben/.opam//default/.opam-switch/packages/melange.0.3.2
... sources ...
/Users/ben/.opam//default/lib/melange-compiler-libs
/Users/ben/.opam//default/lib/melange
/Users/ben/.opam//default/doc/melange-compiler-libs
/Users/ben/.opam//default/doc/melange

> cd ~/melangetest/

> cat dune-project
(lang dune 3.9)
(using melange 0.1)
> dune build
File "dune-project", line 1, characters 11-14:
1 | (lang dune 3.9)
               ^^^
Error: Version 3.9 of the dune language is not supported.
Supported versions of this extension in version 3.9 of the dune language:
- 1.0 to 1.12
- 2.0 to 2.9
- 3.0 to 3.7

PS: Which version of opam are you using and how did you install it?

opam --version
2.1.4

I think I installed it with homebrew

I don’t know what melange is. It’s probably complicated. Could I suggest that you start back-tracking, to find the smallest set of packages you can install, that produces the problem. So for instance, if you create a switch and install camlp-streams, can you then compile a program that uses them? If the answer is “no”, that would be useful to know.

Thanks. Can I ask where you got (lang dune 3.9)? The latest version of dune is 3.7.1. So it doesn’t make sense to use 3.9. Right?

This is what I use for my day to day workflow for managing local switches. I like to think of them as isolated sandboxes of packages on a per git checkout basis. Nothing is shared so there are no accidental cross dependencies. It does waste disk space but there’s currently no good solution for that. The relocatable OCaml compiler patches will help but they’ve not landed yet.

Create a bare local switch as, substituting whatever OCaml version you are after:

opam switch create . 5.0.0 --no-install

Then for each *.opam file you can install the dependencies for that using

opam install ./<package_name>.opam --deps-only --with-test -y

or install them all together as

opam install . --deps-only --with-test -y 

This should give you everything you need to dune build the project or dune runtest.

Hi @bentxt, thanks for trying this project, I help maintain this opam template.

About opam switches:

Local switches might be useful when you work with a lot of projects in the same computer. As the number of projects increases it might be harder to resolve the dependencies for all of them under a single switch, as opam only allows a single version of each package to be installed. But as you mention global switches are much more efficient in terms of storage.

It is very important to know which switch is selected before running any opam command. To do so, one can run opam switch. In the list of results, the active switch (local or global) will appear highlighted, and with an arrow on its left side. In this list, local switches name is the path to the switch in the file system, while global switches name is the version of the OCaml compiler they use.

About Melange and Dune:

One relevant thing to note is that the latest version of Melange is only compatible with Dune 3.8, which has not been released in the official opam repository yet. It seems it could be released very soon.

As Melange depends on this version of Dune, a new version for the melange package will also be released a little bit after that.

I understand you want to build Melange using your opam “global” switch. To do this before the official releases happen you would need to use opam pin to install the unreleased version of both packages. So instead of doing opam install <package> (which installs the officially released version), for now one has to use opam pin add <package>.dev --dev-repo, e.g.:

opam remove melange # make sure no issues happen during the next command
opam pin add dune.dev --dev-repo # --dev-repo installs from latest commit in github
opam pin add melange.dev --dev-repo

As a side note, the way this “pinning” happens in melange-opam-template is by using opam pin-depends field in the project opam file.

Finally, there is a short guide about Dune in the Melange docs site, which includes steps to build a simple project, either using OCaml or Reason syntax. Another section that you might find interesting is the package management one.

Hope this helps clarifying. This process will be much more straightforward in a few days / weeks.

4 Likes

Thanks for that useful detailed answer.

I’m astonished…
How can this be possible?…

IMHO, writing this requirement on the package webpage is a minimum.
In addition, a clear traceability on a package being published is necessary.

I had and still have headaches with using dune or its so-called documentation. I’m fine with using opam for a while, even if some side-effects can be triggered if you don’t use it « normally ».

In the past, I already mentioned weird behaviors of the opam publishing process from a not so beginner standpoint.
But who am I for telling this? Especially when talking to people responsible for tooling design, implementation and maintenance (« in the past you had to manually install and compile all packages, so enjoy what exists »…).

One thing seems clear (at least to me):
A beginner as well as a normal OCaml user or an occasional OCaml user must be protected from this kind of weird and discouraging behavior (of tools and people).
If you add to this problem the difficulties for setting up tooling for the first time and the lost-in-dune[-documentation] experience, you can understand what beginners or not so beginners can feel and how they can react. While OCaml is a very nice and satisfactory PL.

Knowing the exact state of any OCaml object/artifact one needs to handle seems a minimum and vital requirement.

Except enforcing basic and documented rules (correctly and explicitly implemented by tools), can you see another solution?

There are multiple factors for this:

  • Melange is a relatively new project
  • Melange is not a simple library, but a backend for the OCaml compiler that emits JavaScript
  • Before 3.8, Melange projects would use Dune in a more primitive way. During the last year, we have worked to deepen this integration to leverage the full power of Dune in Melange projects. Unfortunately, this deeper integration involved changes in both Dune and Melange, which means we can’t provide backwards compatibility with older versions of Dune.

This is mentioned in https://melange.re/build-system/#creating-a-new-project. In the Dune page that documents Melange stanzas there is a warning to mention that the Melange package is not officially released yet. We will update this as new versions of both dune and melange packages are released.

I agree. We use opam pin-depends in the template .opam file to get all this complexity hidden under a single command. There are still the trade-offs between local and global switches that this thread discusses. I guess the ideal solution would be having local switches sharing a global cache of packages, so there is no distinction about local and global switches anymore, kind of like esy does.

Thanks for your answer.

I’m sure you’ve worked a lot for producing Melange.

But I personnaly dislike needing to read Dune’s documentation to get the information I need while I’m confident in the author of a sw.

These information must at least be available in the webpage of Melange with a BIG RED warning : « this will NOT working until dune 3.8 is released !!! Pls. wait a few days (automated publishing process). See Dune’s documentation there… ».
I’ve searched for it (small phone screen) and did not found it.

Again, I’m pointing a « hole » in tooling that lets programmers/authors do what the honestly find sufficient from their standpoint. But from beginner/not so beginner standpoint, it’s often an unnecessary and bad experience.

I hope people competent or responsible for OCaml mainstream tooling (i.e. opam and dune now increasingly intricated) - even if opam&make do the job - will react and discuss and elaborate some plan to help enforce key rules that respect users and increase their satisfaction (in terms of usability and reliability).

Is there an option to set pin-depends within the dune project itself? Especially if I want dune to generate my opam files?

No, you have to use a template file which dune will then merge together with its output opam file.

EDIT: e.g. GitHub - yawaramin/dream-html: Generate HTML markup from your Dream backend server

1 Like

My goodness. I’ve been looking for this for days! Thanks as always @yawaramin. The doc page itself is so obvious to find now Packages — Dune documentation.

1 Like

Thanks alot for your general explanations on Opam and your specific help.

For me personally it would be tremendously helpful if more of this conceptual documentation would be available. The official usage centric documentation is not enough for me.

Concerning Melange, I could install the dependencies with those “pinnin” ocaml commands of yours:

And then I tried to run dune wich gave me the following error:

> dune 
# ...
File "src/dune", line 7, characters 7-22:
7 |   (pps reactjs-jsx-ppx))
           ^^^^^^^^^^^^^^^
Error: Library "reactjs-jsx-ppx" not found.

So I tried this which helped

 opam pin add reactjs-jsx-ppx.dev --dev-repo 'https://github.com/melange-re/melange.git#ff0ca826c7b2175ff9208f626c68d960cfcdfcd3'

Is this the right way to do it?