I have grown to like OCaml, even though initially it seemed difficult, while I was learning to build a compiler.
Now that I have become little bit better in coding OCaml, I wanted to build other projects, specifically a terminal code editor (like Helix, NeoVim, etc)
However, trying toopam install notty nottui lwd etc. libraries to manage the terminal emulation has shown that OCaml / Opam’s build system wants to downgrade and recompile 200 different or so other libraries, and even the compiler and lsp itself, before I could install the libraries, and do meaningful work.
$ opam install notty nottui lwd
The following actions will be performed:
↘ downgrade ocaml-compiler 5.4.0 to 5.3.0 [required by ocaml-base-compiler]
↘ downgrade ocaml-base-compiler 5.4.0 to 5.3.0 [required by ocaml]
↻ recompile ocaml-config 3 [uses ocaml-base-compiler]
↘ downgrade ocaml 5.4.0 to 5.3.0 [required by lwd, notty]
↻ recompile seq base [uses ocaml]
↻ recompile ocamlfind 1.9.8 [uses ocaml]
↻ recompile ocamlbuild 0.16.1 [uses ocaml]
↻ recompile num 1.6 [uses ocaml]
↻ recompile dune 3.20.2 [uses ocaml]
↻ recompile cmdliner 2.1.0 [uses ocaml]
↻ recompile base-effects base [uses ocaml]
↻ recompile base-domains base [uses ocaml]
If using libraries causes so much friction (rant), then we can not use OCaml for anything serious, that could take a long time to complete, or has to be maintained for a long time.
Maybe there is some other way we can avoid such issues, but most users would want to opam install what they want and expect to be able to start their work.
Your example is a good outcome. It is dependency warm drinking water, rather than dependency hell. You’re trying to install a package that’s incompatible with the version of OCaml that you have, on which many packages depend, and opam gives you some actions that still make that work.
The only way that this much friction goes away is for other people to fix it before you notice it, especially by people going around and asking package maintainers to make a package work with the next unreleased version of OCaml, as is happening right now. That didn’t happen or the work wasn’t done for notty and lwd, so those need an older version. As often happens with languages that aren’t super popular, some problem would be solved by more popularity.
The maintained version of notty is actually notty-community. Maybe opam could suggest a compatible fork in an upgrade plan like this, when the alternative is more work.
opam’s relatively good here for giving you a plan that works, and OCaml needs it to be good because good OCaml style is relatively brittle to upgraded dependencies: if you have an exhaustive match over the cases of a dependency’s ADT, and the dependency adds a new one, then your match no longer compiles.
This seems to happen a lot for many libraries, that’s the issue I’m trying to bring up.
Using notty-community does not solve the issue, because nottui and lwd also have the same conflict, and they have no available alternatives. Even if the library maintainers update their dependencies, it will only be a temporary solution, because after few years, the code will again have conflicts.
Makes me ask, How do others use OCaml to do anything, if libraries conflict like this?
If you care about this particular toolset, perhaps you could be part of the solution.
No one should expect “to opam install what they want and expect to be able to start their work” unless they are purely consumerist parasites of the commons.
Is this really an immediate blocker for you though? Just follow the install plan, let it downgrade to 5.3–which is a fairly recent version–and go about your day? If the 5.4 restriction really bothers you, send a PR to notty to update their dependency bound?
Edit: as someone said earlier, notty is unmaintained, and the opam repository maintainers have published a new package notty-community to replace it. Unfortunately, this forces the entire community to churn to replace a package name everywhere. This is a legitimate problem. Imho the opam community maintained package should take over the name of the unmaintained package, not force everyone to use a new name. Opam-repository has the right to control the package names published through them. Imho they’re being too conservative here by using a new name.
Golang packages eagerly depend on newer versions of the compiler, and become unusable by old versions of the compiler, so if you are stuck on an old version for any reason (like using Kubernetes images that are a few versions behind) you may find yourself reading release notes to find the right interim version to manually upgrade to, to keep your versions somewhat up-to-date.
Dependency hell really happens, and what you’ve run into really isn’t ideal, but you’re underestimating hell by a lot here.
Libraries can conflict, but you don’t have an example of that here. Opam is showing you how to have all of your libraries work for you - just, on a slightly older version of OCaml.
What you can do in general, without trying to solve bitrot yourself, is to make more use of opam switches, as it doesn’t matter if separate applications built from separate switches have actually incompatible constraints. I had to have a switch just for coqfmt for this reason.
dune seems to be also be developing in a direction that’ll make this more natural, but I’m not following it closely.
Others have pointed out that one of the packages you were trying to install was not available at the compiler version in your switch (5.4.0), so opam was trying to downgrade your compiler. Which is …. gonna be painful, but is also almost never what you want.
This has happened to me also when I’m working on upgrading one of my packages to a new compiler version, and so I’ve taken up the habit of pinning the compiler version, viz.
Then if I try to install a package that isn’t available at that compiler release, I get a nice succinct complaint instead of an interminable list of packages that I need to recompile/reinstall.
But then if you assume never downgrading the compiler is the right default, the next topic you get on discuss is “opam is too stupid to figure out that by downgrading the compiler I could have installed the package set I wanted”.
True enough. I have scripts that (in parallel) drop-and-then-recreate en masse N different switches for N different (pinned) compiler versions, then install a bunch of packages in each switch, so that when it comes time to do development, I can choose which compiler version I want to work with. For sure, sometimes one might want opam to go ahead and downgrade the compiler, and heck, it’ll do so if you don’t pin the compiler. If you don’t want that behaviour, it’s straightforward to forbid it.
And it’s good that you have that choice – that opam doesn’t force one or the other behaviour.
improve the tools a bit, send a PR to the project, these are open source projects, they come with no guarantees of usability but they come with the ability to make some changes you need and either send them upstream or make your own fork
and yes, maybe you don’t have time to maintain a fork of one or the other of your dependencies… that’s ok. just don’t assume other people have all the time in the world to maintain the upstream project
In this particular example, help is on the way. lwd and nottui have merged the dependency on notty-community, so this problem will go away on the next release.
I can’t say that I have experienced the problems “doing things” that you have. I use opam switches to handle this. Local switches (opam switch create . 5.3.0) and local aliases (opam switch link …) can be particularly useful for this.
Once you have compiled an OCaml executable which is native, or which is a bytecode executable compiled with the -custom option, the executable will run irrespective of the current selected switch.
As the situation currently stands, all of them will have to be updated to depend on notty-community. This seems like a suboptimal approach. Imho it would have been better to publish notty-community as just ‘notty’ in opam-repository as I mentioned earlier.
I hope opam-repository will consider a package takeover policy in coordination with ocaml-community to avoid this kind of churn in the future.
when someone runs into an issue with a dependency, the response here is “it’s open source, you can contribute!” or “feel free to send a PR.”
I get the intention, but think about it from the perspective of someone learning OCaml who just wants to build a small CLI tool. They’re not in a position to debug a dependency they barely understand yet, and that was never their goal. They came to build something, not to maintain someone else’s library. We should meet people where they are.
Accepting that the root issue is lack of usage of those libraries and lack of maintenance energy.
Not to be the ai-bro, but eventually using some help from LLMs can get you pretty far on both learning the language and fixing someone else library. Not that I want PRs from LLMs, but I pushed some fixes about CI/Windows that were really out of my zone.
Incidentally, I wonder if the subject would have come up at all if OPAM could deploy relocatable binary artifacts (deployed from the cloud or even purely locally), which would mean that such mass-recompilations actions would be much faster to perform.
It is funny that you mention this, because OCaml (note: OCaml, not random libraries from the internet) is one of the languages with best track records anywhere in terms of backwards compatibility, to the point that program written in OCaml 20 years ago could still be compiled relatively easily with present-day OCaml.
In my personal experience, people building serious software in OCaml generally take a rather sober approach to their use of third-party libraries (as they should), precisely because of this issue: you don’t control the third-party library and no amount of effort in the UX of the package manager will shield you from breaking changes/abandonment/etc. I don’t think any other language has discovered a magic solution to this basic fact.
There is something strange with your experience. Opam has this concept of invariants, which are formulas that a given switch should always enforce. By default, such an invariant contains the version of the compiler, which means that opam install should never try to modify the version of your compiler (unless you pass an option such as --update-invariant).
$ opam switch 5.3.0
$ opam switch invariant
["ocaml-base-compiler" {= "5.3.0"} | "ocaml-system" {= "5.3.0"}]
$ opam install ocaml-compiler=5.4.0
[ERROR] Package conflict!
* No agreement on the version of ocaml-compiler:
- ocaml-compiler = 5.4.0
- (invariant) → ocaml-base-compiler = 5.3.0 → ocaml-compiler = 5.3.0
You can temporarily relax the switch invariant with `--update-invariant'
No solution found, exiting
Did you mess with the configuration of your switch to remove the default invariant?
It depends how you create your switch. For example, if you do opam switch create . to create a local switch, the invariant is "ocaml" {>= "4.05.0"}, which allows compiler downgrades.