Dune wish list for 2023

[benjamin-thomas]

@nobrowser

And is there really a good reason for beginners to prefer it?

I assumed I would get a better “experience” out of the box with the vscode plugin because of this quote from the section Configuring Your Editor on ocaml.org.

For Vim and Emacs , we won’t use the LSP server but rather directly talk to Merlin.

Granted, I haven’t looked into setting up another editor so I’m not sure if that’s correct.

[n4323]

Maybe one could just redefine name as names? Then (name foo bar) would be accepted which is a little unusual, but otherwise things should just work?

[n4323]

@rgrinberg

I think the manual stagnates because the entire team agrees that it’s “bad”, and therefore has no incentive to improve something that will be discarded eventually anyway. We should probably adopt a more gradual plan for improving the manual.

One option may be not to improve the manual (which is “bad”) but instead start a new tutorial and user guide (“good”) from scratch. After that’s in a good state, in a second step the manual could then be either rewritten as well, or liberally refactored as a pure reference manual.

Ideally, someone new to dune would test drive the user guide. I have made the experience that after learning the idiosyncrasies of some tool I have trouble pinpointing where exactly it fails in terms of user experience, although when using it first I had real and annoying issues.

[talex5]

@Armael

To fix this error, you must specify an explicit "modules" field in every
library, executable, and executables stanzas in this dune file. Note that
each module cannot appear in more than one "modules" field - it must belong
to a single library or executable.

I would tend to read this as asserting that there is just no way for executables to share a module.
Perhaps the message could, as a hint, suggest using executables?

Yes please! I didn’t know about this either (having also been misled by the error message). Updating my build now…

[olafhering]

I think this question suggests that “dune” should know about “opam”. Instead it has to be the other way around: “dune” should offer some way to extract the used dependencies within the dune files and provide them in a machine-parsable format. Then other tooling, like “opam”, will be able to translate this namespace into its own namespace.

There used to be the command dune external-lib-deps, which was removed in dune 3.0. This command did not provide info about the required ppx packages, for example. A more complete replacement for this functionality is pending, there is an issue open in the bug tracker. Hopefully the replacement will handle libraries, ppx, external binaries and also conditionals (to hide dependencies for the current environment) properly.

The key point is: “actual library filenames”, “ocamlfind names”, “opam package names”, and the “system package management names” are four separate namespaces.

For rpm packaging and pkg-config, a virtual namespace pkgconfig(value) = version exists to represent capabilities of a package. This pkgconfig() string can be used in Provides/Requires/BuildRequires, the Provides are automatically populated during build, based on the *.pc files. For the SUSE ocaml packages I simulate the same concept with a separate ocamlfind() namespace, which is also populated from existing META files during package build. IMO such functionality is missing in opam to translate at least between “ocamlfind” names and “opam” package names.

[grayswandyr]

First, thanks for your work on dune!
Recapping what I wrote somewhere else, I think newcomers, students and even developer of simple projects would really benefit from:

• specifying a single dune (or dune-project as you propose) file (e.g. with the subdir stanza you’re talking about)
• having a simple stanza and way to call dune such that warnings are kept as plain warnings but reshown at every new invocation of dune (untl the source files are fixed), while not inhibiting tests
• having no upwards exploration of the source tree by default (once again, we have several students with errors every year because of a missing dune-workspace)
• I don’t know if much can be done to improve the situation but fresh invocations of dune utop are slow, even for a simplistic project, apparently due to the linking phase
• auto-formatting (in VS Code at least, by far the most used editor among students) doesn’t work as long as an .ocamlformat file (even empty) isn’t present

[BikalGurung]

@olafhering

The key point is: “actual library filenames”, “ocamlfind names”, “opam package names”, and the “system package management names” are four separate namespaces.

Yes, exactly. There is this ocamllib RFC that was meant to solve this issue but alas it was not to be.

[gasche]

@rgrinberg

Regarding compiler errors, to separate individual errors newlines would require us to parse compiler output, no? Sometimes the compiler emits more than one error/warning per compilation unit.

This is a good question, we have to decide how to deal with this user wish and the responsibility may lie with the compiler, or upstream, or both.

My impression is that:

1. Dune makes decisions about how to collect and later report several messages together, so it should participate
2. The compiler is only affected for (non-fatal) warnings, not for errors (it stops after the first error).

Given dune’s default to turn all warnings into errors in development mode, I think that a solution that adds blank lines between each error in the Dune output would work well in practice – it sounds simple and gives satisfying results. On the compiler side, a low-hanging-fruit approach would be to add a blank line after each warning or error, including the last one. This would also work (assuming Dune keeps this trailing blank line in its concatenated output), but it would result in useless trailing blank lines after the last warning or error (if any), so it feels less nice.

(Hopefully this sort of things will be easier once the compiler can provide a more structured output format to compiler-calling tools.)

[UnixJunkie]

Here is my dune wish list for 2023:

• Do one thing, and do it well.

Let other tools take care of the rest.

[mobileink]

Align Dune syntax with Scheme syntax so we can use Scheme implementations to work with Dune files. Specifically: disallow a single dot in lists. Currently it is not unusual to find (foo blah blah .), e.g.

(action (with-stdout-to %{target} (run ../src/test.exe --kind bad .))

which makes Scheme readers choke. Require ‘./’ in such cases.

Dune’s syntax is almost Scheme as it is. I’ve got Scheme code to translate dune files to Bazel, which I’ve tested against a lot of code, and this is the only problem I’ve run up against.

[hhugo]

More paralellism for inline_tests when available. See Run inline tests in parallel · Issue #1516 · ocaml/dune · GitHub.

[mimoo]

• better integration with opam (I want to list my dependencies and their versions in a single place, I don’t want to deal with global switches or to make my project unbuildable due to installing/updating/removing a dependency) (also, why do I need an .opam file for a project at the root of the project? We end up having hundreds of .opam files at the root to make our dune build work)
• friendlier commands (dune test <PATH>, dune build <PATH>, dune check <PATH>)
• strong conventions based on directory structure (minimal configurations)
• merge dune and dune-project files (why have 2 cfg files per project?)
• implicit_transitive_deps set to true by default (this is so much saner)
• detect presence of rule.ml (or build.ml) and execute its content before building anything else, potentially promoting files (so that I don’t have to write my rules in this lisp language) (this is what cargo does with build.rs)
• a file extension for dune files, so that my IDE can syntax highlight and format automatically
• a command to start a new project (dune new <PATH>)
• a command to run specific tests (or specific inline tests) (dune test <TEST_NAME>)
• better documentation (omg, I spent so much time in there and the documentation needs to be rewritten from scratch :D)
• homogenous naming for modules. I’m still confused about MyModule_submodule vs MyModule.submodule vs another one I can’t remember
• clarify workspace vs project (actually, why do we need a dune-project file to begin with? Can’t that just be a dune file like Cargo does? (a workspace file and a project file are always just a Cargo.toml file) (also, if I’m in a git submodule of an ocaml project, running a dune command seems to detect the dune-project of the parent folders, dune should not do that imo unless a dune project/workspace explicitly lists this package as part of its member (this is what cargo workspace do as well))

[mimoo]

I remember wondering what “stanza” meant when reading the docs and not being able to find a definition anywhere

[JohnJ]

I would like an easier way to use Menhir’s messages feature. I have a project that uses several Dune rules to validate that the error messages are complete and then compile them into an OCaml file. (My rules are similar to the ones in this Menhir demo.) These rules feel very hacky and fragile. If there’s an an error (e.g. I edit a Menhir rule which requires me to change the messages but I don’t change them yet) then debugging and fixing that is a hassle. I feel like easing this friction would make the Menhir messages feature a lot more attractive to new users

[mimoo]

same with target/targets, and dep/deps, It’s unnecessary to have two keywords for essentially the same thing

[rgrinberg]

@grayswandyr

The warnings situation is acknowledged. The problem of having no upwards exploration is that it makes it impossible to run dune from sub directories. It’s quite a nice feature to be able to cd to a directory and run dune build @check inside it.

I have no idea what utop is slow, but could you see if you can reproduce the issue with the normal toplevel?

The issue with formatting is a decision ocamlformat made. It’s not related to dune.

@mobileink dune’s syntax is scheme inspired but leans towards human readability rather than compatibility with scheme. Would it help if we had a small library to help parse dune stanzas?

@hhugo that sounds like a good feature. How would dune compute the partitions? Or would there be a way for the user to provide them?

@mimoo Opam integration is on the timeline. It should be possible to move the .opam files to an opam/ sub directory. Nobody has done that work yet.

dune {test,build} <PATH> already work. I’m a bit hesitant to add something similar to check as it’s nothing more than alias for dune build @dir/check.

Do you have some examples of information that dune can infer from the directory structure?

dune and dune-project files are parsed differently. In particular, the first line of the dune-project file is special and sets the version of the dune language for the entire project. If we ever choose to move away from sexp, we would need to read the first line of the dune-project file (in whatever syntax) to detect how to parse the rules in the project

implicit_transitive_deps leads to some very poor messages in some situations. Once there’s first class support for it in the compiler, we’ll surely make it the default.

It’s already possible to write a dune file in ocaml syntax. Other Topics — Dune documentation It’s not a feature we recommend though

Your IDE should be smart enough to detect file types without extensions. Does it not work for makefiles for example?

There’s dune init although it receives very little love.

A command to run specific tests was mentioned earlier in this thread. We’ll add one.

We need concrete suggestions to improve the manual. “It sucks” doesn’t really help us improve it.

@JohnJ better menhir integration would be good of course. Could you make an issue and we’ll take a look? We would appreciate some help with features like this. I’m not familiar with advanced features of menhir so I need that someone could explain and/or help out with the development here.

[zozozo]

For reference, I also had the same problem (e.g. using menhir’s messages feature, and making sure the messages file stays up to date), and I managed to write some rules that give me a reasonable solution, that you can find here.

Basically the trick is to:

• have a messages file containing your error messages (syntax.messages in the dune file linked)
• have some rules to generate an updated version of your messages files. To do that, the dune file:
  • generates a new messages files from the .mly file (this is new.messages), this contains empty/dummy error messages for all the states from your parser
  • create an updated version of the current messages file (mainly to update state number and if I recall correctly, remove messages for now non-existing states)
  • merge these two files using the dedicated menhir feature. This results in a messages file that contains all of the original messages, plus empty/dummy error messages for new states, and minus the now non-existing states
• lastly, have a dune test that diffs the current messages files (the one commited in the repo), with the one generated by the step above. That way, when running tests, you’ll get an error if some states are missing from your message file, and you can easily use the promote feature to update you message file, and then update the message for the new states by hand (following the diff that was printed by the test).

[mobileink]

@rgrinberg

@mobileink dune’s syntax is scheme inspired but leans towards human readability rather than compatibility with scheme. Would it help if we had a small library to help parse dune stanzas?

Probably not much, thanks. Don’t really need a parser once the ‘.’ has been changed to ‘./’ - the Scheme reader already parses everything. Well, at least the once I’m using does, I haven’t tested other implementations. In any case, my thinking is just that we might see more tools if Scheme implementations could read dune files; the only thing blocking that (to my knowledge) is the dot ‘.’ in lists.

[JohnJ]

@rgrinberg

@mobileink dune’s syntax is scheme inspired but leans towards human readability rather than compatibility with scheme. Would it help if we had a small library to help parse dune stanzas?

It would be a small feature, I personally would be grateful if a dune file parser existed. There have been times where I would have liked to be able to extract dune-project data programmatically (e.g. the version number, license name, etc.). This was never for anything important, but it would have been convenient. Although I’m not a fan of JSON, YAML, or any of those other configuration formats, I appreciate how you can easily read them with basically any scripting language.

@zozozo Thanks for sharing this! Your rules are much more comprehensive than mine, and may be a good model for the kind of thing Dune could do natively. I’ll file an issue with the Dune repo soon.

Edit: it looks like there’s already an open issue about Menhir messages: Support for menhir's `.messages` files · Issue #3284 · ocaml/dune · GitHub

[edwin]

Is there interest in generating system-independent Makefiles from dune?

I did some experiments on modifying the output of dune rules --makefile: xen/tools/ocaml/dune2makefile at xen-builds5 · edwintorok/xen · GitHub.
The idea was to remove absolute paths, and otherwise system specific configuration from it so that the makefile could be reused on another machine, running another distro, that doesn’t have ‘dune’ installed.
Although this may be brittle and not guaranteed to work for long it’d only need to hold together for a transition period.
OTOH doing these changes in ‘dune’ itself might be easier (rather than parsing the output of dune…), but I wouldn’t want to impose the additional maintenance burden for what might be a “one-off” feature used for a few months/years and then not used again.

Having makefiles generated from ‘dune’ would allow a more gradual transition from hand-written – and often incorrect – Makefiles to dune in projects such as Xen, where a sudden transition isn’t really possible due to part of the CI running on Debian oldstable which doesn’t have a suitable version of dune available.
OTOH a simple shell script can compile the entire OCaml code in Xen a lot more quickly than the recursive Makefile, and similarly modifying the current recursive Makefile to have a single hand-written Makefile with no incremental compilation that just builds everything in a single rule is still quite fast, comparable in time to how long make clean takes (which has accumulated sufficient cruft along the years that it is quite slow), and avoids all the problems with the old makefile: tools/ocaml: simplify and speed up build system · edwintorok/xen@96ef0e3 · GitHub
But obviously that Makefiles will then need to be maintained by hand alongside the ‘dune’ files for a transition period.