How about we all try a slightly less serious discussion about tooling for once. I invite you to respond with a list of most desired features, most hated bugs, and anything else you might like to see dune do differently.
To keep things somewhat on topic, let’s avoid complaints that aren’t specific enough. For example, if you think dune is too slow, describe which commands you would like optimized. If you don’t like the manual, point out the specific questions that aren’t good enough. Let’s also steer away from enormous projects such as rewriting the entire frontend to use your configuration language.
Given enough user support and a little bit of luck, the dune team might fix these issues in 2023.
I’d like to be able to run specific tests with dune runtest. A file at a time would be great, a test at a time would be better, and having a little language for selecting subsets of tests would be amazing.
I’d love to get rid of warnings-as-errors and instead have dune remember and replay warnings from the files that are already compiled. That would keep the warnings visible but not block my workflow.
The current Dune manual is mostly a reference manual, which is fine for experienced users. However, for many beginners Dune is their first entry into the world of OCaml and the current manual offers them a very poor experience.
In my mind, the manual should have a much lengthier tutorial section, introducing most basic features in tutorial style (see the SCons user’s guide SCons 4.4.0 for a fine example of this). Also, it should be explicitly targeted towards beginners in OCaml, and not focus solely on the Dune side of things.
By the way, this is an ideal task for someone who wants to contribute to the ecosystem in a impactful way, yet does not require a high level of technical proficiency in the Dune codebase.
Caveat: This might be out of scope for dune and this will only work for use-cases where everything builds using dune
These days a lot (if not all) libraries I depend on are built using dune. I also have been relying on dune’s support for defining package metadata when writing my own OCaml projects. I also use dune’s excellent vendoring support in a lot of use-cases. I’d be really happy if dune could gain some additional features in this area and also aid in managing dependencies without requiring opam or an external opam plugin like opam-monorepo.
Some of the platforms I use already package dune + pretty recent versions of the ocaml compiler. If dune could also help with downloading dependencies and managing local copies of sources it will allow one to get started with projects with dune + compiler without having to configure an opam switches etc. I’ve also really liked the vendoring workflow as I get full merlin/lsp support including within the dependencies with this whereas with the opam workflow merlin/lsp stop working the moment I jump to a location within the dependencies.
Is dune able to generate a list of opam dependencies used in the project along with the versions used? This would be helpful for reproducible builds. Ideally it should print the list in a dune-project format so it can be easily copied.
Is it my turn on Santa’s lap?
First of all thank you all for the amazing work being put into OCaml tooling, it keeps getting better at an awesome rate!
As for the coming year…
I want more soft docs for dune, with common workflows, currently it’s mostly listing stanzas…
I want dune build --release for leanest and meanest executables, dune debug dropping me in a debug session at breakpoints I define somehow, and dune bundle creating FHS-style tarballs for package managers.
I also want the ability to run tests selectively, seen it first using rust & vscode.
Now I know most of these are currently achievable, but the process isn’t “invoke this command and sit back and relax” I guess.
Sorry for being not very specific, but here is my story. When I first decided to use dune several years ago, I landed on the online doc: Overview — Dune documentation and I was like “oh wow, a full-featured doc using sphinx and everything!” But what turned out to be really amazing, is that (I honestly think) I was never able to find solutions to my questions using theses docs. Even until now. I always end up trying to copy things from existing projects. (thanks a lot, Sherlock!)
I guess this means, as other users said, that one needs a new user’s manual, in addition to the reference manual. But in fact even the reference manual is hard to navigate. It is currently a mixture of lists of stanza syntax with here and there some hints and tips whose status is not always clear (experimental feature? just a hint or a mandatory way?). It would benefit a clear separation (like the OCaml manual) between a gentle intro and a technical API). Thank you Santa and happy new year
Indeed my plan is to allow to pass tests directly to dune runtest. For example dune runtest foo.exe for (test (name foo)) or dune runtest foo.t for a cram test.
Warnings
As I’ve mentioned before, there’s no full solution here without some help from the compiler. We can start by replaying the stderr from compiler commands though.
Manual
I don’t know if we’ll finish this in 2023, but we can certainly start on this.
Package Management
Indeed this project is enormous but luckily the team already recognized the need for this. I can’t give any timelines, but definitely expect some updates in 2023 about this.
Opam Dependency Versions
Dune can probably list all the packages used as part of building libraries an executables. It does not know the packages of binaries. However, I’d like to solve the problem in a different way. Instead of listing the packages post-build, I’d prefer if dune didn’t let you use a package until an explicit dependency on the package was specified in the project file.
Generic Rules
It’s a shame that this seemingly simple feature requires some deep changes to dune itself. We’d like to work no these changes in 2023, but I’m not sure if it will bubble up to the frontend in time.
My wish would be for the decoupling of the dune files from the source tree. From a high level that opens up a couple use cases:
OCaml programmers, especially beginners, can use tools that could generate dune files based on implicit conventions, so they don’t have to litter their source tree with their own hand-crafted dune files.
Mixed language programmers can use build script generators to generate dune / ninja / Makefile files as appropriate to the domain. CMake is the build script generator I’m familiar with, but there are many others and it would be nice to have an OCaml-based entry there as well. One nice benefit for Dune maintenance is that Dune, like Ninja, can stay simple with a limited number of stanzas and actions; the higher-level build script generators can be scripted to provide any needed functionality.
As a motivating example today, my parameterizable, embedded Dune DSL tool requires a few lines of copy-and-paste boilerplate in a dune file in the source tree. And then it generates a dune.inc which has to be in the same source tree. It is a) unhygienic having generated files in a source tree and b) requires users to put in boilerplate.
One idea to implement this would be to allow specifying three directories:
The user code directory tree (the --root today). It contains user code and dune-project.
The build code directory tree containing generated dune and dune-workspace files
The build object directory tree (the --build-dir=_build today)
Today (1) and (2) are the same. But with (1) and (2) separated, a user could specify in their dune-project that they would like a higher-level tool like my Dune DSL (or CMake, Meson, etc.) to populate (2) without touching (1).
I don’t want to over-specify a solution. There are probably other things that could work like a multi-stage Dune. Central wish is that I can let tools of my choice create dune files without polluting the source tree.
The include stanza allows including the contents of another file in the current dune file. Currently, the included file cannot be generated and must be present in the source tree.
So I don’t think what I described (ie. generated dune files that are not present in the source tree) can be achieved. But I’d love if there was already a way! Do you have an example somewhere?
EDIT. Emphasis added to stop the reply chain: that are not present in the source tree
But I don’t think this is the same. First, in my development tree, I prefer my dependencies to be open-ended, for example sedlex>=3.0 , so I can upgrade, rebuild and test without editing the versions of each dependency a priori. But then I want to reproduce the versions exactly when building in a CI framework. Second, I actually prefer separating the concerns of dune and opam, that is I want to use dune only as a build system. There are a few reasons for that, one can be gleaned from my recent question here The upshot is that I don’t like to specify prerequisite packages in the project file at all - that is a packaging concern, hence better left to opam. If you proceed as you indicated, I’ll be stuck.
I have some tests that need to be run as superuser, would be nice to have support for it with sudo and doas. Could be simple enough, but it might require a settable timeout for the test, in case things were not engineered for sudo to not ask for a password and it’s not entered.
The first contact with dune is horrible.
The second contact as well. etc.
(re)giving dune a trial from scratch shows many ambiguities and makes the experience very uncomfortable or horrible. You may wonder what is rotten: the documentation, the sw or the programmer himself? Just with a basic program with a basic library module (lib)
Why calling bin/ what should be called source/??!! Why not crunch/ or foo/?…
Being forced to handle a bunch of dune files in the hierarchy is a trap. See herebefore how easy it is to fail.
Why is/are dune file(s) not generated from the program structure, defined by the main program and its modules (internal « libraries ») and (« external » depext) libraries?
There is often…no feedback. Or it’s cryptic.
Obviously, this software has not been designed from the user standpoint… while it’s supposed to be a key part of the OCaml infrastructure.
It also prevents beginners to do and understand what is compilation and linking of .mli/.ml and their subtleties, which are hidden by declarations (sexp).
I don’t see advantages over make (handwritten or generated Make files, which are readable and simply… work). Being faster is not enough; make is still a good choice until dune becomes usable.
Btw, where is the specification of that software? And the documents on which was made that spec.
Where is the roadmap?
Problem
Iirc, I had similar questions as @Luc_ML on my first starts with dune in regards to the naming conventions of bin/lib/test and why files in subdirectories couldn’t immediately compile. I remember speedrunning through the Examples and https://dune.readthedocs.io/en/latest/quick-start.html to try out some lib, missing some key details here and there haha.
Solution(s)
But maybe a respective readme should be generated with each dune init templates because dune’s quick start docs explain a handful of these questions really early on.
Maybe include_subdirs should be default for new init’ed templates. But now we unknowingly introduce users to the rules of include_subdirs. We do know the dune team is working on (include_subdirs qualified) - thread. So once this is shipped, defaulting new projects to have (include_subdirs qualified/unqualified) may be a great help.
I’m on team-make-startup-as-easy-as-possible because it makes experimentation for new users really easy and lets them continue down the ocaml rabbit hole. I expect software developers to be of the patient, debugging, doc-reading type, but having to dig into build docs when you want to experiment with a library is always a hassle.
@jbeckford generating dune files without touching the source tree is something we’re working on. We should have something ready with 2023 but with other limitations. For example, you might not get the friendliest error messages if the script you’re using for the generation will itself depend on dune file it is generating.
I’m curious about the comparison of dune and ninja though. The dune dsl is designed to be very high level so I’m not sure why you think it’s a good idea to wrap it. I understand there are some complicated builds that require dynamically generated rules, but this isn’t a common use case I imagine.
@nobrowser We’re not going to break the workflow for any existing users without at least providing an opt out so there’s no reason to worry. Note that the feature doesn’t require you to write package dependencies in the project file either. Dune can read .opam files just fine as well.
@haesbaert Yes, that would be quite useful. Something more general would be to add a timeout action. So you could write something like (with-timeout 2.0 (run my-test.exe)).
@Luc_ML Please read my initial post in this thread. I’m interested in feedback but not in low effort rants.
There is no specification for dune. I don’t think that spending time writing one would be a good use of time. A lot of software remains without a spec and is solely defined by an implementation - some interesting programming languages come to mind. I don’t see anyone complaining about those.
There’s no official roadmap and development is done at the discretion of the maintainers. In practice, we heavily rely on feedback from both industrial and open source users to decide what to work on.
The upshot is that I don’t like to specify prerequisite packages in the project file at all - that is a packaging concern, hence better left to opam. If you proceed as you indicated, I’ll be stuck.