The OCaml Platform - a vehement dissent

Ecosystem
,
21

A classic case of confirmation bias. Who’s to say the “suffering” would not have been even greater had the Haskell Powers that Be decided early on to choose one product as the winner and actively promoted it, thereby discouraging innovation?

Regarding your previous comment about “tools that don’t even exist yet” etc. - by definition, all improvements involve bringing something that does not yet exist into existence. It’s hard read your comment as anything other than endorsement of the status quo and opposition to innovation. The way to support people who work to improve things is to make it easier, not harder, for them to do so. The Roadmap makes it pretty clear that Dune integration is a requirement for a tool to gain entrance to the One True Platform. Example: W17 Run Tests “…The editors also integrate with Dune…”. That eliminates the entire class of those of us who have no intention of integrating our tools with Dune. The problem here is stark: Dune is not extensible.

22

The problem is that “The OCaml Platform” is not a platform. It’s a product: a selection of tools. OCaml (by which I mean whoever controls e.g. ocaml.org) should not be in the business of picking winners in the tooling marketplace. Any more than they should be picking winners from among OPAM packages, which I don’t think is controversial. For example I assume nobody thinks it would be a good idea to declare a particular package is the One True JSON package for OCAML.

I think Clojure is a good point of comparison. For a long time Leinengen was the most popular build tool, but it was not the only option. Clojure faced all the problems OCaml faces with respect to getting started. Had somebody proposed that Leiningen be officially declared as “The Frontend of The Clojure Platform”, Rich Hickey would have (I assume) immediately rejected that, and would have been right to do so. Eventually Clojure came up with its own (superb) build tooling. That probably won’t (and should not) happen with OCaml, but the point remains: picking an early winner would have squashed innovation in the space.

Having said that, let me be clear: I am not arguing that work on the Platform should stop. I’m not even arguing that it should not use Dune. On the contrary, I think its a good project. But it is a product, not a platform, and it should not called “The OCaml Platform”. Rather, like any product, it should be given a distinctive name (trademarked by the OCaml Foundation). For example, “The X Toolsuite” where X could be “Dromedary” (or “ODromedary”), or Bactrian, or Caravansary - no shortage of camel-related terms. It should be promoted as one among many possible such toolsuites (or “dev environments”). It could even be officially supported (or even "endorsed*) by the Powers that Be, but only so long as it is made clear that such endorsement does not mean “recommended to the exclusion of alternatives”. It would just mean “you can trust this stuff to work pretty well if you decide to use it.”

The primary reason I oppose blessing Dune (plus OPAM) as the official build system for OCaml is because as build systems go it is just not very good. It’s very good indeed at somethings, but also very bad at some essential things (like extensibility). Furthermore, I expect that over the next year or so Bazel will become a practical alternative for any OCaml project. The rules already work (and have built largish projects like Mina and Tezos) and tooling to automatically convert from Dune to Bazel is pretty close to done. Bazel can build the compilers (I just built 5.1.0-rc3). The remaining hard problem is how to publish Bazel-based packages to OPAM. On the other hand, I’ve added support for the Bazel module system, which means that for a fully Bazelized project OPAM is not needed. In any case, soon it will no longer be true that Dune is the best choice.

Furtherfurthermore, the Dune-to-Bazel conversion tooling uses a build-system agnostic intermediate language, which means that writing a tool to emit, say, ninja files should be feasible. Which opens up yet more tooling possibilities.

I’ll post details over coming weeks.

Regarding “platform”: to me a platform is a collection of public, standardized data schemas, APIs, and protocols. “The OCaml Platform” is not that. The major problem facing OCaml is lack of such (tooling) schemas, APIs and protocols (they’re not entirely lacking, just inadequate.) It is not the trouble users have getting started - those are just symptoms. Its the trouble toolmakers have building the tools that would make life easier for beginners. Dune is about as unfriendly to toolmakers as it gets. The build protocol for the compilers is poorly documented. Many tools in the ecosystem fail to follow good tooling practice - for example, hardcoding file paths for inputs and outputs, inadequate command-line options, etc. Lots of small things that add up to a lot of headachery. Addressing those issues, especially making the build protocol clear, might well have a bigger impact than blessing a set of current tools as the One True Platform.

3 Likes
23

I understand that you as someone writing a tool/a build system has this perspective, but someone that is not doing is not really going to care. Maybe its gonna result in better tools after some time, but some of the complexity is inevitable. Odoc for instance has a very complex command line, for good reason, and I am glad dune takes charge of this for me.
Also, having a one true platform is not going to prevent people from improving tool APIs.

I dont think the plan of the platform is to prevent anyone from making a new build system. As far as I know, the plan for the dune-opam integration is make opam usable as a library, which means that anyone will be able to write an OPAM frontend in ocaml, maybe there would be a way to use it in bazel.

9 Likes
24

I’ll second @EmileTrotignon’s words. Most of us don’t want to bother with the tooling around OCaml. We want to use OCaml to make stuff. OCaml spent most of its history mired in solving the question of “how to build and document OCaml programs”, and it’s time we moved on from that. dune is a nice, standard, competent tool that does the job efficiently and allows the community to move on from worrying about tooling. It covers most of the use-cases we have as a community, bringing us closer to the competition (other new languages). The developers are constantly listening to requests for specific use-cases that haven’t been covered yet.

The Platform is an attempt to present the complete, ready-to-go, off the shelf, easy-to-adopt OCaml distribution – one that can compete with Rust, Haskell, Go etc. If you think you have an idea for a better tool than dune, than by all means present it and what use-cases it covers better. If the community agrees with you, they’ll switch to it.

14 Likes
25

In fact, you’re mistaken and we’ve had rather spirited debates on this very forum regarding whether OCaml.org should “bless” a set of recommended packages.

There are people on both sides that make good points regarding the benefits and problems of standardization. It’s kind of our form of bikeshedding.

I don’t think that we’ll come to any consensus soon. As long as people aren’t forced to use the standard platform, I think that people will be happy. And I do think that the people working on these efforts very much don’t want to lock anybody in. The concern is that is that it feels like we may be moving in a direction where that becomes increasingly the case. (c.f., Linux).

26

And the more everything is integrated (with and into Dune), the harder it will be for people who want to bother. Because they will have to know the innards of Dune (or their plugin system, or whatever) too and integrate with it, to be able to have other people use your tool or frontend or whatever.

The hardest part of writing the test extensions was getting any information on how and when the inline test runners (ppx_inline_test) are being build by Dune and having to hack my way around the lock in watch-mode (which maybe won’t be necessary some time in the future, when the Dune RPC clients can actually “see” the output of the running command).

27

Firstly, your dissent is noted, and appreciated (thanks for engaging with the discussion!). We have had many, many build systems for OCaml in its rich history, and almost all of them still work (ocamlbuild, oasis+ocamlbuild, omake, make, cmake, all lurk within the older corners of opam-repository and are regularly bulk built).

I’m entirely supportive of documenting alternative recommended build systems for OCaml, iff:

  • the recommendation is meaningfully tested across the architecture/OS combination supported by the OCaml compiler.
  • it has some users who are motivated to keep it working, and step up to do the work.
  • it has an interoperability story (even if manual) with publishing to the opam repository, which is the gathering space for the OCaml community.

There will never be a requirement to use Dune to participate in package set in the OCaml.org website. The only requirement is the presence of an opam file in your package to indicate how to build your contribution, and how it can depend on other packages from the community via dependencies, and how it should be used (via its license).

There were some related talks (such as Meta’s buck2 for OCaml) at the recent OCaml Workshop. I, and I’m sure many others, would be delighted to see more writeups and presentations on improved build approaches in our various gatherings.

In the meanwhile though, Dune is a clear frontrunner for the default recommendation, as objectively counted by the package set in the opam repository. This does tell me that for the dune team, it’s important to prioritise negative feedback to make sure they don’t accidentally stomp on a smaller set of users. For example, @xavierleroy (and others) have noted that the default warning flags in development mode are too expansive and diverge from the compiler defaults. I’ve created dune issue #8645 to track this; please do create others as you come across papercuts or interoperability problems with your own developments.

24 Likes
28

I started with OCaml somewhat recently. I remember being a bit confused, but it might be due to the way I got started. I took a course in “advanced programming languages”. The instructor had put all of the the build commands into a makefile or something so that we would only have to modify the code files and rebuild without worrying about build details. I read the makefile and there were a bunch of commands in there. I wondered why it was so complicated. (It turns out that it is not very complicated in simple cases, but I wouldn’t know that from the instructor’s makefile.)

As someone who has mostly written C and C++ in my professional career and even before that, my bias is toward the simplicity of opening a text editor, writing some code, then typing “cc” or “gcc” and executing the resulting output. I use IDEs and build systems too, but when starting to learn a new language, I don’t want to think about choices like what build system to use or what IDE to use before I even start trying out some code.

I don’t remember what the OCaml docs were like in that regard, now. I remember that I eventually settled on using dune, even though the main project where I work is using Makefiles that call ocamlbuild. I was a little wary at first, but it turned out to reward just a little effort, at least for some of things that I wanted to do that I’m not yet sure I how I will do in the Makefile project: e.g., build subprojects (is that what I should call it?), and easily compile an individual file (with all of its dependencies).

I also tried learning Rust from the Rust “book”. I was slightly annoyed that it had me learning Cargo in order to build “Hello World”, but I guess that is the way of the world now.

2 Likes
29

I also found it somewhat difficult to find in the Rust docs how do something like compile without the runtime, which one is occasionally interested in doing with systems programming.

30

Note that this is absolutely not specific to learning a new language. It’s exactly the same when you start toying with a new idea or design. The amount of boilerplate one need to setup in the “way of the world now” just to try the tiniest spark of an idea is totally off-putting.

5 Likes
31

I use the OCaml (or Rust or Python or Coq) Jupyter kernel for that. OCaml Jupyter | An OCaml kernel for Jupyter notebook
Multiline input works better than with Utop and using Opam packages works, an example:
https://github.com/akabe/ocaml-jupyter/blob/master/notebooks/word_description_from_DuckDuckGoAPI.ipynb

And whenever I read “Akinori Abe” I have to think about Akinori Abo: https://www.youtube.com/watch?v=6HMa5tofqps

2 Likes
32

This actually defeats your point. Leiningen is the one that won at the “tooling marketplace” by being preferable to both prior Java tooling and later competitors like Boot. Then the “powers that be” decided to release their own build system in the Clojure CLI and asked the community to move to it, not on merit but because it’s official.[1]

So I agree our Leiningen is Dune, but it isn’t becoming the Clojure CLI. Rather than pushing on us a new thing, OCaml Platform is just adopting the current winner of the “tooling marketplace”. You should see this as an opportunity, actually: It means future winners may also have a chance of being adopted.

  1. Even then, it’s taken 5 years for it to match Leiningen’s market share. And while it’s more flexible, it takes multiple libraries and more configuration to recreate common workflows with it; I and many others are still conflicted. ↩︎

5 Likes
33

People did indeed suggest, propose, beg, and plead for Leiningen to be delcared official/blessed, for years. And yes, Rich et al. did ignore those calls, until the core team produced their own build workflow, a decade+ after Clojure was being used productively in the wild. Through that entire period (and still, from what I hear), a top-line complaint from new users was “where is the blessed, full-featured build system equivalent to cargo/maven/pip/etc?”. In this way, I see the platform proposal as a way to avoid what was the Clojure experience for so long.

If a suitable dune alternative were to develop in the coming years, I see no reason why it wouldn’t eventually be promoted as such within “The Platform”, especially if it were to satisfy some use cases for which dune perhaps isn’t naturally suited. Until such an alternative is available, quibbling about the semantics of “The”, or “platform”, etc. doesn’t seem like a productive pursuit.

Dune exists, it works, and is reasonably well-integrated with other tooling in the ecosystem. Criticizing it vs some unstated ideal seems incredibly unfair.

13 Likes
34

It looks like we penned our Leiningen-related retorts ~simultaneously. It’s kismet. :rofl:

4 Likes
35

Exactly! In the end, it is each individual developer, project or organisation that chooses their tooling. The OCaml Platform can, at most, reflect the majority consensus, guide newcomers to “the happy path”, and help concentrate efforts towards the technologies defacto chosen by the community. If the community moves, so must the platform.

And to ensure it does, we need exactly these frank, honest and constructive conversations.

5 Likes
36

I think that’s a misreading of history, but this is a side issue so let’s not pursue it further here.

37

Completely agree, please let us know who these malefactors are so we can mock and insult them behind their backs! (That’s joke, people!) Seriously, I know you can’t be thinking of me or the others who have offered very specific, very concrete critiques of Dune. But that’s not really the issue. In fact I think Dune is a wonderful tool. I just think its the worst possible choice as the central point of integration.

38

Also, different build tools are still required for Java and JavaScript “backends” in the clojure/clojurescript ecosystem.

In OCaml, besides the usual ecosystem, the 2 JavaScript compilers - js_of_ocaml and Melange - also use dune as their build tool.

1 Like
39

But they (or at least jsoo) do not require Dune (rules_jsoo); some other widely-used packages do, in the sense that using them without Dune requires a lot of extra work.

40

That’s reasonable, given the fact that OPAM is genuinely the only (practical) option. But please keep an open mind wrt package management. Bazel has a package management system (unfortunately called “Bazel modules”, informally “bzlmod”) that is designed to support external package management systems via a “module_extension” mechanism. I’ve been adding bzlmod support to all my packages and it works beautifully. I’m not sure yet exactly how one would go about integrating bzlmod and OPAM, but it should be a fun challenge, maybe somebody will help out. In any case, just wanted to put that on your radar.