Between spin, drom, and esy, there are a lot of ways to create and manage OCaml projects (not to mention plain dune+opam). This has made it hard for me to get up and running on OCaml projects and understand best practices. Even among the most popular libraries, there seems to be a lot of variety in how the project’s packages are managed, how the project is built, code style, etc.
I think having the OCaml core team adopt and promote a comprehensive tool, and then publish extensive documentation on how to get up and running, best practices, etc. using that tool would go a long way to increasing OCaml productivity and adoption. Such a tool could still use opam+dune under the hood but be more opinionated about how dependencies and builds work and be “officially” sanctioned as the recommended way to create OCaml projects.
I am less concerned about which tool gets chosen, but I do want to call out Python’s Poetry. Poetry builds on other Python tooling by using virtual environments and pip under the hood. In fact, I think Poetry would be a great model for an OCaml project management system given the similarities between Python’s and OCaml’s long history and diversity of tooling. Poetry also does a great job wrapping up all the different pieces it builds on into a nice, easy-to-use package.
Why isn’t just dune+opam sufficient? It’s well documented, and pretty much works out of the box for most OCaml usecases (standalone binaries, libraries, js_of_ocaml, foreign libraries).
Also, the dune init ... subcommand family makes it easy to setup new projects as well.
The three tools you mention serve different audiences. That’s why there are three different tools: consider their varying approaches to metadata and project lifecycle management. That’s also why there is variety among projects: their age (older projects tend to be more conservative in shifting toolchains), their audience (application or library?), and their target (native, embedded, javascript) all dictate differences.
To address the title of your post, the “official” package manager for OCaml is the opam package manager. This is primarily the metadata format (as found in ocaml/opam-repository), and the libraries that handle that. The opam CLI is also distributed along with the opam libraries, and goes to great pains to be backwards compatible to maintain stability across versions (the OPAMCLI variable for example). More concretely, we will not accept packages into the ocaml.org package management ecosystem unless the package is accompanied by an opam file, since that is the common mechanism we use to relate packages to each other and identify compatible subsets.
There is no official “project” manager, as the diversity of choice is still settling as people build over opam+dune for their various usecases. I encourage you to contribute to the discussion about how you find the use of spin/drom/esy/dune+opam for your own application rather than seeking answers from the core OCaml team. Consensus will emerge from the usecases you (the community) are all applying the tools to, at which point best-known-practises for each of them can be documented on the central website. Unlike Highlander, there can be more than one project management tool!
Incidentally, comments are welcome on a draft chapter I’m working on for RWOv2: The OCaml Platform - Real World OCaml. It’s not intended as an exhaustive survey, but more to get a newcomer going while working their way through the book. (comments can go on Issues · realworldocaml/book · GitHub).
I currently prefer to use dune + opam over the other options. But there are some niceties I miss. E.g., to add a dependency I have to:
add dep to dune-project
run build to generate the opam file
invoke opam (one way or another) to install the dep
then add the library to the relevant dune file(s)
This isn’t super difficult to do — tho it’s tougher for newcomers, obviously — but it is repetitive, and seems like a good target for automation
I think there are a couple such routines that are less frequent, but they’re not coming readily to mind!
The other thing people seem to like is project templates. I am skeptical of this, since it usually means embracing boilerplate, but it is kind of tedious to track down the various dune stanzas for, e.g., configuring a menhir/ocamllex pipeline, and I don’t set them up often enough to have them memorized. So I can definitely see the rationale here.
I respect everyone’s opinions here and I don’t think anyone is wrong, per se. But without an opinionated, comprehensive tool the barrier to entry to OCaml is high, and it is especially high when you compare it to newer languages like Rust, Go, Elixir, etc.
I don’t think I fully understand the diversity in tooling argument, especially considering that many languages have a single package/project manager (e.g. cargo) that works for 99% of use cases. Sure, some projects fall into that 1% of weird use cases that need to be able to do something different, but those 1% of situations are usually strange in other ways and the authors are prepared to handle them (they aren’t beginners).
Diversity in tooling also makes it hard to learn from existing projects since popular projects are doing things in different ways. It adds just one more barrier to getting up and running with OCaml.
I am seeing first-hand how adopting Poetry is increasing Python productivity at work. People feel more comfortable using Python and run into fewer issues, which in turn encourages people to use Python more. That is a desirable situation and I don’t think OCaml has achieved that.
With the release of OCaml 5, many devs are going to be spending time either learning OCaml for the first time or trying it out again. People are going to be excited about learning effects and exploring OCaml 5’s approach to concurrency and parallelism. But I am concerned that without a straightforward, coherent, and comprehensive tooling story, those devs won’t stick around.
I think you have a good point. In fact you can search the forum and see that this tooling argument has come up multiple times. Poetry is a good example. Go with its single ‘go’ tool is another one. I think we can learn a lot from how simple it is. Start using a dependency library, run ‘go mod tidy’, and it’s downloaded, checksummed, and cached in one shot.
There is definitely something to be said for a single integrated tool for using a language. On the other hand the OCaml tooling maintainers have never had this philosophy, choosing to instead ‘let a thousand flowers bloom’. Something to keep in mind
I think you’re right on point. Right now learning OCaml must be incredibly hard compared to other languages, where you generally need to write (at most) one file to add a dependency or configure your project. Then you need only one command to build (and possibly run) your program; that includes downloading packages, building them, etc. People are now used to this level of comfort (and I know I am that impatient when I look at other languages!).
Compared to that, even dune+opam is super clunky and there’s a lot of basic errors one must learn to cope with. Even the mismatch between findlib names, opam names, and modules, is tough.
That said I have little hope that the core team will ever bless any tool besides ocamlopt so I think the best chance is that dune grows and slowly subsumes the rest of the tooling.
edit: the one-line version is that, if I didn’t know OCaml already I probably would struggle to get going with it, and probably use rust/scala/kotlin/haskell/… instead.
I think the bottom line is that our competitors don’t sit still. OCaml has gotten much better in its tooling, but the rest of the world keeps moving. At the same time, as a heavy python user, I didn’t even know about Poetry and was still using pip, which shows that diversity of tooling is in fact an advantage.
The question in my minds is whether any of these tools really gets rid of the tooling pain points of OCaml, which is the reliance on multiple esoteric metadata formats. I’m guessing that the answer is no. esy at least can do a little better since it entirely replaces opam, but any other extra tool must add yet another metadata format to those of opam and dune.
I didn’t know about poetry either. I don’t use pip and usually use conda when I need a python environment. The fact that didn’t prevent Python from becoming the most popular language is contradictory to the premise of this entire thread.
I think you’re right on point. Right now learning OCaml must be incredibly hard compared to other languages, where you generally need to write (at most) one file to add a dependency or configure your project. Then you need only one command to build (and possibly run) your program; that includes downloading packages, building them, etc. People are now used to this level of comfort (and I know I am that impatient when I look at other languages!).
Do you think people would be deterred if the process was just:
$ opam install
$ dune build
Is the problem the fact that we have two binaries or that binaries themselves have some rough edges that need to be ironed out?
Compared to that, even dune+opam is super clunky and there’s a lot of basic errors one must learn to cope with. Even the mismatch between findlib names, opam names, and modules, is tough.
Perhaps you could describe how other languages do it better. I’d love to improve this, but I don’t know how. In rust for example, I see the exact same problems with crate names, library names, and module names.
That said I have little hope that the core team will ever bless any tool besides ocamlopt so I think the best chance is that dune grows and slowly subsumes the rest of the tooling.
Thankfully they’re far too wise to try and impose something. instead of a mandate, we could try and evolve towards something that is more usable. The community already proved that it will move to something new if the benefits outweigh the costs like in the case of dune.
The fact that didn’t prevent Python from becoming the most popular language is contradictory to the premise of this entire thread.
That was maybe true in the early aughts, but expectations have risen
now. This is probably one of the most critized aspect of python in 2021.
A lot of things used to be possible and are not anymore (for example, a
proprietary language or even a language with proprietary compilers;
that worked for mathematica but prevented Ada and D from becoming
popular, for example). The point here is that unified, easy tooling is
also table stakes for a new language.
I think it’d be better, but we’re far from that. Right now you need to create a local switch, configure your shell to invoke opam env, call dune build to generate opam files from dune-project (if we assume that’s the one autoritative file), then use something like opam install . --deps-only, then dune build finally works. Even then you might struggle to find the actual binary since it’s not in an easy place like target/release/foo.exe.
Oh and I didn’t even look for the proper invocations that produce and use a lockfile.
well with cargo for me it’s a lot easier:
add the dependency to Cargo.toml (one line; you can install cargo-edit if you don’t even want to edit by hand)
cargo build --release
et voilà, the transitive dependencies are downloaded, added to lock file, and the new binary is produced. From within my code I just have to use the new crate.
I’m not saying cargo is perfect but it’s a lot more streamlined. Coming back to project after a few years and having them build without any fuss is also a very nice experience (made possible by lockfiles and strictly immutable packages).
This criticism is a lot more fair, but the argument changed from dune and opam not being a single binary into opam and dune has these particular flaws.
Even then you might struggle to find the actual binary since it’s not in an easy place like target/release/foo.exe .
I can’t see how this is easier than $ dune exec -- foo.exe, but if there’s enough people to argue that this is a good idea, I don’t see why we can’t do the same.
I agree with the issues regarding opam. It is needlessly complex because it tried to be a general purpose package manager like apt, rather than a project and language specific project manager like other tools mentioned in this thread. The end result is that the more important use cases like project local sandboxes and lockfiles get a second class UI, compared to installing global packages.
A few minutes weren’t enough for me to understand how Go does it. Maybe you could take a few minutes to describe how Go manages its various identifiers? In OCaml, we have:
Packages - contain multiple libraries and executables.
Libraries - contain multiple modules.
Modules - introduce toplevel identifiers usable in source code
In rust, there’s crates, libraries, and a namespacing system to cover these three points. How does Go do it?
How does one depend on a binary then? For example, if I need to build a code generator and then run it to generate code for another package?
Are there any naming restrictions for modules, packages, files? Simon’s complaint was that one had to remember package, library, and module names. It seems like in Go one has to remember 3 sets of identifiers as well.
We already know that main is the executable package in GoLang. Hence a module containing the main package is the executable module. The main package will contain a main function that denotes the start of a program. On installing the module having the main package it will create an executable in the $GOBIN directory.
I.e. if you have a module containing a package main that will be built into an executable.
So as noted above, the main package name has a special meaning. Other than that, no restrictions (keeping in mind that I’m still learning).
In Go file names are not meaningful in the language. All files in the same package share the same namespace. The package is the unit of namespacing.
How this could translate to OCaml: we could have tooling that mandates that a module named Main would be built into an executable. Other module names would have no special meaning. Of course we could have other special names e.g. mandate a module Test is built into a test runner.
I respect that the core team has a strong belief about this. But if the goal is to make OCaml more popular, then I do not think it is the right approach. I also respect also that there may be other goals, and that is okay, too (not every language has to be mainstream). Finally, people in this thread of made enormous contributions to OCaml and to PLT at large, so thank you all for your great work.
I’m a big fan of OCaml as a language so I hope that I am wrong and the core team’s approach does work out in the end. But in the meantime, it is hard for me justify investing more time into learning and contributing to the ecosystem as I do not believe it will have a flourishing future.
Thanks for clarifying. I don’t think we’re too far off from the system in Go then. The main distinction I see is that a library (package) in OCaml does not necessarily introduce a module named after the library. I guess that’s why we need the whole public_name vs. name business in dune. It’s actually not so easy to fix this problem, because many people prefer hierarchical public names such as pkg.foo. Such names clearly don’t map to valid OCaml identifiers.
How this could translate to OCaml: we could have tooling that mandates that a module named Main would be built into an executable. Other module names would have no special meaning. Of course we could have other special names e.g. mandate a module Test is built into a test runner.
That would be fine, but all you’ve done here is shave off the name field in an executable stanza in dune. Hardly a major change.
