Hello, I’m surprised by this affirmation. Is it a personal opinion or does this reflect the dune team point of view?
2 Likes
If you build software as a profession, you should invest in a proper, generic build system; one that will no doubt require more work for the simple cases but will also allow you to build anything from anything.
What build system is that?
Early OCaml projects were often managed with make and it is not straight forward to capture dependencies, or to get parallel builds to work. A specific pain point is that a CMI file is either derived from an MLI file when it exists, or directly from an ML file when no MLI exists. This is difficult to express in make. Another complication is building variants, say a profiling or debugging variant. Working with dune is, in comparison, easier and faster. In response, more OCaml specific build systems were developed and dune won that contest. You can still combine dune and make in diverse projects but I would always chose dune for the OCaml part.
3 Likes
It seems I’m seeing a lot of historical revisionism on how dune came to be or how it “won” a contest.
First dune came out as jbuilder which was an ad-hoc system developped over a week-end to be able to build and distribute JaneStreet’s open source code dumps outside of JaneStreet (the week-end design and adhocness still transpires to this day in my opinion :–)
As for “winning” a contest, I don’t want to expand too much on that, but let’s say it did not win on technical, usability or adequacy grounds. It won, for good or bad, by political maneuvering.
dune is undoubtly the build system to use now if you want to use OCaml in a restricted setting but disregarding the problems that both @rixed and @threepwood mention is a disservice for current and future users OCaml.
The fact that basic usability problems like @threepwood mentions are still not resolved even though they have been mentioned for many years now constantly remind me that it was a good call to stay away from it. And yes as @rixed mentions the domain specific build system is a wrong assumption for the practitioner who does not live in an OCaml niche.
4 Likes
It does not reflect my point of view. I don’t know what were in mind initially, but in practice, we’ve mostly leaned towards a build system that is optimized for the common OCaml project made to be published to opam, while still able to accommodate non standard setups depending on one’s pain tolerance. It is certainly not optimized to single file scripts as some people have pointed out.
No doubt. But how does that preclude you from using language specific build systems when it’s convenient to do so?
While I certainly I believe you, I find it puzzling that you think your experience generalizes to the rest of us. Off the the top of my head, the programming languages you’ve listed do not include C++, Python, Java, C#, Go, Rust, Scala. Altogether those probably cover the majority of projects in the industry. Do you really think makefiles would be adequate for those as well? Have you considered that there’s some survivorship bias here?
I don’t fault you for not knowing every little bit of dune trivia, but I think there are some omissions here. The most important IMO being that dune’s set of primitives (the stanzas) was a carbon copy of an already proven scheme for defining OCaml build rules. Dune may have started as a weekend implementation, but it was a reimplementation of battle tested ideas. I can’t be certain, but I think it’s a much better explanation for its success than what you’ve offered.
There’s some good criticism in this thread, but your comment just comes off as bitter. I get that you have different ideas on what a build system should look like, but instead of focusing on promoting them, you’ve chosen to just repeat the same 2-3 talking points about dune for nearly a decade now.
The last word on build systems has not been written yet, and many new interesting projects are still popping up in this space. It would be more useful for dune and non dune users alike to hear them instead of the same old tired axe grinding.
6 Likes
As for “winning” a contest, I don’t want to expand too much on that, but let’s say it did not win on technical, usability or adequacy grounds.
I can only speak for myself, but I assume I’m not the only one who did move to jbuilder/dune for usability/adequacy (correct, fast builds) reasons.
8 Likes
And today I stumbled on Generating Makefiles with OCaml? - #20 by jeremiedimino specifically for how jbuilder/dune started as a Makefile generator for Base.
2 Likes
If you’re trying to get any global answers regarding “what’s optimal”, know that there’s no easy way to do that here. You’re going to get only a tiny representation of OCaml users answering here, generally the ones who are particularly vocal, active, or just currently looking into the language. The majority is silent, and the proof is in the pudding – the vast majority of users happily make use of dune, including myself.
Before dune, ocaml build systems were diverse, incompatible, complex and brittle. This was acceptable at a certain point in time, but the competition has advanced, and nobody wants to mess with complex build systems anymore. The competition includes dynamic languages that don’t even have build systems to deal with, as well as fully integrated solutions that make previously onerous work trivial. OCaml felt antiquated in this sense until dune came along and simplified things.
Personally, do I love the s-exp syntax? No, and I think it presents one more unfortunate barrier for OCaml adoption for people coming in from the outside. I would have preferred something more common in the industry, but on the other hand, it’s not too hard to learn and it does the job, and it’s also infinitely better IMO than the mess we had before. I can also appreciate the advantage of having a dedicated DSL, enabling strong analysis and and guarantees, rather than inserting a fully Turing-complete language like OCaml itself into the build system.
2 Likes
Be specific, please, if you think I misrepresented this.
That’s quite a funny comment to make. As far as I remember you were not around when the events I’m referring to unfolded.
I can reassure you I’m not bitter. For my own use I live in a quite better build space than what dune would provide me and I’m likely also better off without having users pestering me :–) The reason why this build space is not promoted (and will perhaps never be) is because I’m still unhappy with some aspects of it. It turns out that since 2017 I’m no longer paid to be working on build system research so that line of work went into a low priority process. Personally I don’t feel like pushing onto others what I do not feel adequate or marginally better to what exists now, I think it would be irresponsible to do so for the promotion of the language.
I also had hoped that perhaps dune would eventually turn into something better, given that it seems a lot of ressources are being poured into it, but given the way you chose to answer here rather than trying to answer the usability talking points (some of which are trivial), I’m not hopeful.
I already explained in my post how I think it “won”.
Let’s try to lower the temperature folks. While evidently not everyone agrees, I would like to think that everyone is doing what she or he believes is most appropriate, and doing so with best intentions in mind.
While I have heard people periodically complain about different aspects of how Dune works, often these complaints are made in unstructured form in the middle of random threads in various forums. This makes it rather difficult to convert them into concrete suggestions and/or proposals for improvements. A more constructive way to go about it would be to open an issue in GitHub - ocaml/dune: A composable build system for OCaml. (and/or, even better still, engage with the dev team and propose a fix).
As far as I know (happy to be corrected) most of these “resources” (funding from Jane Street and Tarides) are and were channeled mostly to people working on the Dune “engine” (the backend, which is used inside Jane Street) and on the “package management” stuff. Contributors working on the “frontend” of Dune (the rules and the Dune language), which is what Dune means to most users are few and far between and are doing it mostly on their spare time these days (is my feeling).
It is a personal opinion. Dune was never designed with education in mind. In fact, Dune’s “engine” is industrial-grade and scales to extremely large codebases (like those found inside Jane Street).
Thanks for mentioning this. Despite what is sometimes claimed, Dune is superior to the build systems that existed before, at least in some important aspects. Just to mention some: high level of correctness, excellent backwards-compatibility story, rock-solid Windows support, scales to large codebases, etc. This does not mean of course that it is perfect (and based on user reports, usability is not where it should be), but it is a bit disingenous to claim that there aren’t technical reasons for the widespread uptake of Dune by the community.
Cheers,
Nicolas
12 Likes
I find that Dune works pretty well for basic cases, but I am occasionally frustrated by portions of its interface that don’t anticipate simple variations to common workflows.
One of my recent posts provides an example of a workflow squarely within “the OCaml niche” that seems like it should be trivial, and probably would be trivial with a traditional build system. (I’m still looking for a solution, by the way. 
)
I do wish there were a larger middle ground between the chaotic, shell-driven workflows of Make and the clean-but-austere approach of Dune stanzas. The auxiliary Dune Libraries seem like a promising start (I found the Configurator easy to use), and I hope we’ll see more extension points like these in the future. Are there any plans @rgrinberg ?
Not that I care for myself. But note that some of these constructive steps have already been taken at least two years ago.
This is a recollection of some old discussion from the era when dune started to get traction. Maybe here, maybe on the mailing list. It made sense at the time, but I can’t find any trace of it now so maybe just a false memory.
I stand by my overall opinion though: Dune makes building libs and executables from OCaml source files easy and fast, but falls short when one want to build a more complex project. Dune is certainly not the only build tool to have this shortcoming, thus the rise in popularity of CI tools that are oftentimes used to do what traditional build tools used to build.
I don’t know how “safe” it is to recommand that devs learn a generic tools over specialized ones, but that’s an approach that certainly deserves to be voiced too. If you are happy with your tool by all means stick to it!
Some time ago, before specialized build systems became the norm, almost every project was build by some form of makefile (or makefile generator).
Were those build systems more complex and brittle than specialized ones? Certainly!
That’s a trade-off that should surprise nobody.
To be perfectly honest, my gripe with specialized build tools comes from my fear of eco-system comunities. I’ve seen how devs became more and more specialized into one language, one framework, one tool, and how mixing various tools to solve a problem has become frowned upon, and I regret the time when languages were just tools not cultural identities, and when it was OK to mix and match knowledge and technologies instead of defending one’s turf against another’s. If I have one bias to confess, that’s this one. And I’m totally open to the idea that I might be oversensitive to this topic and can come across as bitter. I’m trying not to.
4 Likes
This has stumped me on more than one occasion and unfortunately having an LSP hooked up makes it even more confusing because the LSP can correctly infer the type so everything looks right (and is correct) but it still refuses to compile.
1 Like
What do we call a complex project in this context? Is it about multi-language projects?
Dune seems to fit the current trend at that level. For example, Maven is the build tool used by millions of Java projects out there, and it’s terrible at doing anything else. For better or for worse, the vast majority of developers live with language-specific build tools and are happy that way.
I want to say that, as a former user of OMake and Oasis, it wasn’t hard to convince me to switch… Dune is greatly superior to what I had before, and I’m a happy user.
I do agree that there are too many files to manage. Drom has already proven we can have a single file describing the whole project, and we don’t need to spread that information across the codebase. I wish we could be moving in that direction.
And yes, please, make warnings warnings not errors, so I can stop running my tests with the release profile 
3 Likes
It’s kind of possible with just dune. Not completely moving to a single file, as there would still be a dune-project, but it’s already possible to have a single dune-file. By using the subdir feature. We tried but it’s not really pleasant, and I can’t say that it really scales.
Btw drom doesn’t really have a single file either, see for example this package.toml
Unfortunately this isn’t completely true. Janestreet is now using a heavily modified version of the engine from what is in the open source dune. And the open source version we know is kind of slow. Much better than ocamlbuild, but far from ideal and a real pain point day to day at scale. I actually think that it would be easier for everyone if the build system at janestreet had a different name, given that it doesn’t use the same rules nor engine (maybe I can suggest something like… jenga ). This would also ease some governance/ownership issues.
Shared pain here. If one is motivated to work on that topic, ahrefs is actually considering funding.
And to add to the list of pain points, I believe that the scripting story in ocaml is in a kind of “sad” state. It’s more or less possible to interpret a single file. But it doesn’t really scale further than that. And even for a single file, if there are external dependencies it becomes non trivial to manage.
10 Likes
I’m going to reiterate my oldest and most long-standing complaint about the OCaml language ecosystem, since I haven’t seen it discussed in this thread: the unnecessarily complicated mechanisms required for composing applications with their dependent library frameworks.
1 Like
That’s interesting. I didn’t know that, thanks for the clarification.
Cheers,
Nicolas