Dune wish list for 2023

Community
21

And we can see the result of that way of designing something that is just “supposed to be” a key part of the OCaml infrastructure.

Let’s see how you will improve dune beginners’ experience from all these feedbacks.

22

Maybe spend time watching/gathering feedback from new people try out dune If not already done :slight_smile:

Preferably people who dont know ocaml. Maybe even without any compiled lang experience.

2 Likes
23

I don’t have a problem with dune per se. Having a build tool require a descriptive DSL to work sounds reasonable to me.

I think the dune experience could be improve by improving the tooling around discover-ability.

For instance, I find little value in having to read the doc to be able to switch from this form:

(executable
 (name my_exe)
 (libraries lib))

to this form:

(executables
 (names my_exe my_exe2)
 (libraries lib))

In the same way that, when writing OCaml code, I don’t want to have to read the doc of a particular module, just to learn about its function names. I may do just fine just by poking around and calling an obvious function by its name.

So some kind of auto-complete in the official vscode plugin (which beginners should use) would be very nice. This would enable dune maintainers to add, move or remove (and maybe deprecate?) options with minimal user friction I would presume.

I have a couple of other nitpicks with dune:

  1. Builds seem to be slow overall and frequently hang at the late stage of my build processes. Sometimes for up to 10s on my computer, even for very simple projects.

  2. I faced a few bugs here and there that required me to do a dune clean. I don’t quite remember the details but basically I was getting really weird error messages from my tooling and cleaning the build was the not very obvious solution. I suppose dune is a little buggy here.

I’m using v3.5.0 btw

3 Likes
24

This is an interesting example actually. Normally when we have s-expression syntax and a Lisp-like semantics we expect forms to work automatically with variable numbers of arguments, e.g. this is touted as one of the clear advantages of the syntax:

(+ 1 2)
; works the same way as
(+ 1 2 3 4)

So it’s surprising to have plural forms in dune file syntax.

6 Likes
25

I would love to see (include-subdirs qualified) become the default (with default file names like _module_.ml so it requires minimal thinking to see where the base module is described).

It’d also be nice if we could integrate dune file contents into dune-project (perhaps in a dune section with a base directory entry). This way, with one file (and include-subdirs), you have all your configuration done for you in a simple project and you don’t need to touch anything else.

3 Likes
26

Thanks @rgrinberg for the initiative! It’s a cool idea and brave :slight_smile:

My main gripe with Dune is that its usability is sometimes poor (both the experience of using it, and of reading the documentation to use it), and I think that the root cause is that none of the heavy contributors to Dune really cares about it. I don’t think that you are evil people intentionally writing confusing software, but when you decide what to work on, you repeatedly pick something else. This is what I describe as “you don’t care” – you don’t care enough to make a real effort. My wish for 2023 is that some of the frequent dune contributors start caring about usability – ideally all of them!

To give some examples:

More generally, it looks like I have reported 10 issues against Dune in 2020 and 2021, the vast majority of it being actionable usability issues, and basically none of them were fixed. I’m not of course suggesting that my issues should be given priority treatment, I believe the lack of prioritization applies to all usability issues reported. (But it’s hard to tell because you don’t have a “usability” label in your issue tracker. Please add one!)
I mostly stopped reporting usability issues when I encounter them with Dune or Dune’s documentation, because right now it feels like a lost effort.

(Note: I already made a comment to this general effect in March 2021 on the bugtracker, so this should not come as a complete surprise to the Dune team. This point remains at the top of my “Dune wish list”.)

11 Likes
27

It would be great if libraries will be able to install custom templates, that dune will be able to use using dune init mytempaltename

1 Like
28

I agree with many of the criticisms here, but I still find @gasche’s message a bit harsh. As far as I know, Dune has only two developers working on it as their main job: @rgrinberg and @snowleopard. @snowleopard (who works at Jane Street) focuses on the low-level “engine” of Dune and his priorities are aligned with those of his employer, which leaves only one general-purpose developer working full-time on Dune for all the other aspects (@rgrinberg), and his time is shared with other important projects for the community (eg OCaml-LSP).

There is a small group of people who contribute more or less regularly, but mostly on specific areas of expertise (@Alizter and @ejgallego for Coq, @hhugo for Jsoo, @dra27 for Windows, @bobot, myself from time to time) and Dune’s development is not their main job.

Given the limited developer capacity available, I think it is not surprising and rather understandable that documentation/usability issues have not been given top priority.

So it’s good to provide constructive criticism (and I do think that better documentation is sorely needed and the most obvious candidate for improvement that is coming out of this thread so far), but at the same time I feel it is not realistic to expect such a small team to focus on documentation and usability above all else (fixing bugs, implementing new features requested by the community, etc).

Ideally, more people would step forward to help with these issues, especially given that many of them can be fixed without much technical expertise on Dune’s internals. However, the fact is that this has not been happening so far.

Regardless of the above, I think the discussion in this thread will probably influence how Dune’s dev team prioritizes usability/documentation issues, but one should keep in mind that there are some structural manpower issues at play in the current situation, as well.

Just my 2c.

Cheers,
Nicolas

12 Likes
29

Dear Santa,

  • I would really like to use root_modules but can’t because of this issue and the fact every single one of my projects use Logs. I’m surprised I’m the only one hit by that :slight_smile:

  • I often pin packages to a semantic version range to ensure upgrades don’t break compilation, which ends up reading: (dependencies (some-package (and (>= 1.15.0) (< 1.16.0)))). I would like to be able to shorten this to (dependencies (some-package 1.15.x)) or (dependencies (some-package 1.15._)). Happy to submit a patch myself if there is some interest as this looks simple enough.

  • Nitpicking as it looks a bit tricky to implement given the benefit, but I often (include dune.inc) and it does not regenerate dune.inc before interpreting it, nor does it work if it is initially missing (the manual explicitly states this is not supported). Alternatively some templating/rule generation mechanism could alleviate the need for such includes, but this has already been mentioned in this thread.

Thanks for the great work on dune and happy new year !

1 Like
30

@benjamin-thomas the fact that you’re asking this question is a flaw in our documentation. The executables form is quite useful and is not there just to save you a bit of typing. In your example, my_exe and my_exe2 are allowed to use the same shared modules. For example, if you add a.ml, you will be able to access it from my_exe.ml and my_exe2.ml. If you were to use the singular form of executable stanza, you will need an intermediate library for module A to share it.

I’d love to add auto complete to dune but it’s quite a hard problem unfortunately. We have a parser that works like a compiler, but it’s hard to turn that into editor tooling.

  1. Could you benchmark your build with --trace-file. That’s the best way to get to the bottom of slow builds.

  2. Yes, doing incremental build correctly is hard. I would say most problems stem from incorrect build rules written by the user (missing dependencies are quite common). We have a sandboxing mode that we intend to make the default one day to make it easier to catch such bugs.

@bluddy _module_.ml was already discussed. Hopefully there will be a mode to enable it this year. I proposed adding a subdir stanza to allow users to write dune files inside the dune-project file. E.g.

$ cat dune-project
(subdir .
  (library ..))
(subdir test
 (test ...))

It wasn’t met with much enthusiasm.

@gasche Thanks for understanding. We are indeed not apathetic and would like to improve UX. Unfortunately there are resource constraints and organizational issues in the real world. I think the manual stagnates because the entire team agrees that it’s “bad”, and therefore has no incentive to improve something that will be discarded eventually anyway. We should probably adopt a more gradual plan for improving the manual.

I’ll try to look over all your other tickets this week and at least respond with an estimate of how hard would it be solve these issues.

Regarding compiler errors, to separate individual errors newlines would require us to parse compiler output, no? Sometimes the compiler emits more than one error/warning per compilation unit.

@Kakadu Yes, custom templates would be quite useful. I’m afraid they’re not important enough for me to tackle without additional help unfortunately.

2 Likes
31

oh dear, I wasn’t aware of that either; for the longest time I assumed that you couldn’t share modules between executables at all, which was indeed quite annoying. Great to know that it’s actually possible…!

Perhaps the error message for the case where several executable clauses share a module could be tweaked. The current wording points towards using a modules clause and reads as being a bit too definitive:

To fix this error, you must specify an explicit "modules" field in every
library, executable, and executables stanzas in this dune file. Note that
each module cannot appear in more than one "modules" field - it must belong
to a single library or executable.

I would tend to read this as asserting that there is just no way for executables to share a module.
Perhaps the message could, as a hint, suggest using executables?

3 Likes
32

Amazing initiative @rgrinberg

I have 2 use-cases that would be nice to support (and as far as I know there’s no elegant way of doing it, but I might be wrong):

  • Create rules that point to a glob (and have a way to specify to the file globbed)

  • Possible to add a custom dependency to a library. I have seen a few places and empty action to add a deps file to a target.

33

@mefyl

  • I thought I replied in that issue what the problem was. Weird. The gist of the problem is that dune has to reverse engineer the toplevel modules of non dune libraries. It does so using a heuristic that fails if there’s more than one library per directory. So in this case, the issue is most easily fixed by changing Logs to follow this convention

  • I have nothing against this syntax but I would like some approval from the opam devs before blessing it. I think it would be weird if we were to diverge too much from opam here. @rjbou ?

  • We’ll get this one done in some form for sure!

@davesnx

  • Acknowledged. As I’ve mentioned before, globs are quite problematic, but we could introduce other forms that help with the boilerplate.

  • What does it mean to have a “custom dependency” in a library? In particular, which build action is supposed to used this dependency? Is it preprocessing? compilation? linking?

2 Likes
34

You did in the reproduction PR :slight_smile: The reason why is clear, I just not sure @dbuenzli will be keen on changing the layout of the project to accommodate dune.

35

If I understand correctly this is about the install layout, not the source project layout.

In that case, in fact I’m very keen. Mind you I wrote a whole RFC about making that install convention a standard :joy:

My recent packages should follow it and some older packages were migrated to it, but I did not check if all were as it’s kind of news to me that this is actually a problem.

That though will not happen if no proper issues are filed in the issue trackers of the offending packages.

2 Likes
36

That’s great news then, thanks! I’ve created the relevant issue.

1 Like
37

Thanks @nojb for your reply. I am not familiar with the Dune development dynamics – I’m not surprised to learn that contributors are a scarce resource, but I know you are a regular contributor, thanks!
(Nitpick: I’m not sure that counting full-time people is the best metric – for example, as far as I know the compiler distribution also has just 2, Florian Angeletti and Sébastien Hinderer.)

I’m not asking for anyone to focus full-time on usability, or to place usability “above all else”, this is of course just one dimension of the project. I am only asking that people spend a small fraction more of their time on usability issues. Without an increase in development resources, this does unfortunately mean doing a bit less of the other things, so a balance has to be found – my wishlist item is to move the current balance a bit. And yes, I think that people that focus on other things / have other jobs should also consider lending a hand in improving usability, if people agree that this is an important dimension of the project.

There are also a lot of things that Dune gets right, in particular the compatibility story is impressive! (I also have the impression that the development process is well-organized, with frequent open meetings etc. The team may be small but it does a very good job at welcoming contributors, it seems.) Thanks everyone for the good work.

6 Likes
38

I did think this is a somewhat harsh take on the dune maintainers, especially given your own recent post on core compiler bottlenecks – the same bottleneck applies here.

It might be useful to contextualise where we are with a historical perspective: it was just five years ago that we were struggling with absolutely abysmal build times and usability with the layered “combo attack” of ocamlbuild, camlp4, and oasis. This really was the worst of all worlds: you could run arbitrary code in the build system, and had to deal with the slowness of dynamic dataflow graphs (preventing good scheduling), and not particularly optimal build rules. I was recently looking over the epic ocamlbuild rules that @samoht and I wrote for MirageOS 2.0 back in 2011, and was sad for the wasted months of our lives we’ll never get back from that wrestling match.

Since then, dune (nee jbuilder) came onto the scene with one main innovation that changed everything: allow builds to be described as static files for the vast majority of builds (no arbitrary OCaml code), and make compatibility an important goal: since the static files are versioned, future versions of dune are able to build older build descriptions without breakage. Every single rule in dune is versioned, and with decent error messages if you use a new feature with an older dune. This is so much more user friendly than what came before, and to me a sign of maintainers that really do care about the build experience.

The last couple of years have seen a very rapid migration to dune (>91% in the user survey), and a proportionate burden of bug tails. Is it more important for large projects to successfully migrate to dune, or to facilitate usability improvements? There’s no good answer here, but I personally think it’s more important to ensure that projects that have already committed to dune are not blocked, and the team has done an excellent job there. It would, of course, also be useful to fix usability issues, and perhaps a sprint or hackathon of some sort would help kickstart such movement.

My own wishlist?

  • Minor: fix dune#4121 so that I can finally vendor C libraries in my monorepos (and work with @yallop to merge the ctypes/dune port properly). This unblocks the ability to build large projects (like moby/vpnkit with millions of daily users in Docker on macOS and Windows) easily.
  • Major: to have a standalone library to interpret dune files and allow for the generation of rules, in the style of opam-0install-solver for opam files. This would help with integration into Bazel or other third-party build systems.
13 Likes
39

Ping me when you get around to it; perhaps I can help. Thanks!

(I only included Ninja as an existence proof that build tools don’t need to grow more complex when it is easy for other high-level tools to interact with it. Implicit is an assumption that high-level generator tools like autotools/Meson/CMake can interact with high-level build tools like GNU Make and IDE project files. I’m not suggesting that Dune should cease being a high-level tool.)

40

I was a little surprised by this phrase. Is the vscode plugin really more “official” than the Emacs mode I use? And is there really a good reason for beginners to prefer it?

–
Ian