@cdaringe this is extremely nice work, thank you! I’ve contacted you directly about moving it into a more collaborative document to help refine it together across the various platform maintenance teams.
I just wanted to quickly clarify this:
The development process for opam is driven by a simple principle that is inspired by OCaml itself: don’t needlessly break backwards compatibility without good reason, and when it is necessary, justify it. Our tools are embedded in projects that have lifespans measured in the decades, and we take compatibility seriously. That’s why we take pains to provide migration paths from (e.g.) opam 1.2 to 2.0, from 2.0 to 2.1 that are as invisible as possible, and metadata is versioned in both opam and dune.
This is entirely compatible with the need for graceful evolution, since there is a clean separation of metadata (opam files) from the clients that operate over it (the opam executable). I’ve made it clear over the years that we welcome new experiments in the realm of clients that operate over metadata. Where I push back is when metadata is duplicated, as it’s extremely hard to reconcile metadata evolution in thousands of downstream projects. That’s why (for example) the new ocaml-ci is “zero configuration” as it can operate sensible purely by analysing the opam and dune metadata for a project. To enter the OCaml/opam ecosystem, your project therefore must at least have an opam metadata file, since that is what defines what it means to be in the opam-repository (the relationship to other packages, and instructions on where to get yours).
For clients, new workflows such as esy, drom and the VSCode plugin are the perfect experimental vehicle to inform future opam client development. For example, one lesson from VSCode is that a prime source of confusion for users is that “opam install” doesn’t edit the dune and opam metadata! Addressing this requires rethinking the entire model of the client: @dra27 and I’ve been sketching out an alternative opam client design that is more declarative than the current imperative one. Instead of having any global state, an invocation of nopam would simply get the local environment to the state defined in the local .opam file. In this scenario, to install a package a user just has to edit the .opam file to add the dependency, and then an invocation of nopam would result in it being available.
Many of the discussions on GitHub issues on the opam development trackers reflect our immediate priorities – to maintain a stable and evolutionary release process for the opam 2.x series with the minimum of drama for our users upgrading. Please do not confuse this for a lack of desire to innovate – we absolutely welcome such inputs, and will factor that into the opam 3.x development plans. The only thing we’re very careful of is needless backwards compatibility breakage, which is why changes to the opam metadata format are so carefully scrutinised.
I’m overall delighted with the level of innovation happening around opam these days; lots of plugins emerging, new solvers, more analysis and so forth. Keep it coming! 