Dune wish list for 2023

  • better integration with opam (I want to list my dependencies and their versions in a single place, I don’t want to deal with global switches or to make my project unbuildable due to installing/updating/removing a dependency) (also, why do I need an .opam file for a project at the root of the project? We end up having hundreds of .opam files at the root to make our dune build work)
  • friendlier commands (dune test <PATH>, dune build <PATH>, dune check <PATH>)
  • strong conventions based on directory structure (minimal configurations)
  • merge dune and dune-project files (why have 2 cfg files per project?)
  • implicit_transitive_deps set to true by default (this is so much saner)
  • detect presence of rule.ml (or build.ml) and execute its content before building anything else, potentially promoting files (so that I don’t have to write my rules in this lisp language) (this is what cargo does with build.rs)
  • a file extension for dune files, so that my IDE can syntax highlight and format automatically
  • a command to start a new project (dune new <PATH>)
  • a command to run specific tests (or specific inline tests) (dune test <TEST_NAME>)
  • better documentation (omg, I spent so much time in there and the documentation needs to be rewritten from scratch :D)
  • homogenous naming for modules. I’m still confused about MyModule_submodule vs MyModule.submodule vs another one I can’t remember
  • clarify workspace vs project (actually, why do we need a dune-project file to begin with? Can’t that just be a dune file like Cargo does? (a workspace file and a project file are always just a Cargo.toml file) (also, if I’m in a git submodule of an ocaml project, running a dune command seems to detect the dune-project of the parent folders, dune should not do that imo unless a dune project/workspace explicitly lists this package as part of its member (this is what cargo workspace do as well))
6 Likes

I remember wondering what “stanza” meant when reading the docs and not being able to find a definition anywhere :joy:

9 Likes

I would like an easier way to use Menhir’s messages feature. I have a project that uses several Dune rules to validate that the error messages are complete and then compile them into an OCaml file. (My rules are similar to the ones in this Menhir demo.) These rules feel very hacky and fragile. If there’s an an error (e.g. I edit a Menhir rule which requires me to change the messages but I don’t change them yet) then debugging and fixing that is a hassle. I feel like easing this friction would make the Menhir messages feature a lot more attractive to new users

2 Likes

same with target/targets, and dep/deps, It’s unnecessary to have two keywords for essentially the same thing :frowning:

@grayswandyr

The warnings situation is acknowledged. The problem of having no upwards exploration is that it makes it impossible to run dune from sub directories. It’s quite a nice feature to be able to cd to a directory and run dune build @check inside it.

I have no idea what utop is slow, but could you see if you can reproduce the issue with the normal toplevel?

The issue with formatting is a decision ocamlformat made. It’s not related to dune.

@mobileink dune’s syntax is scheme inspired but leans towards human readability rather than compatibility with scheme. Would it help if we had a small library to help parse dune stanzas?

@hhugo that sounds like a good feature. How would dune compute the partitions? Or would there be a way for the user to provide them?

@mimoo Opam integration is on the timeline. It should be possible to move the .opam files to an opam/ sub directory. Nobody has done that work yet.

dune {test,build} <PATH> already work. I’m a bit hesitant to add something similar to check as it’s nothing more than alias for dune build @dir/check.

Do you have some examples of information that dune can infer from the directory structure?

dune and dune-project files are parsed differently. In particular, the first line of the dune-project file is special and sets the version of the dune language for the entire project. If we ever choose to move away from sexp, we would need to read the first line of the dune-project file (in whatever syntax) to detect how to parse the rules in the project

implicit_transitive_deps leads to some very poor messages in some situations. Once there’s first class support for it in the compiler, we’ll surely make it the default.

It’s already possible to write a dune file in ocaml syntax. Other Topics — Dune documentation It’s not a feature we recommend though

Your IDE should be smart enough to detect file types without extensions. Does it not work for makefiles for example?

There’s dune init although it receives very little love.

A command to run specific tests was mentioned earlier in this thread. We’ll add one.

We need concrete suggestions to improve the manual. “It sucks” doesn’t really help us improve it.

@JohnJ better menhir integration would be good of course. Could you make an issue and we’ll take a look? We would appreciate some help with features like this. I’m not familiar with advanced features of menhir so I need that someone could explain and/or help out with the development here.

3 Likes

For reference, I also had the same problem (e.g. using menhir’s messages feature, and making sure the messages file stays up to date), and I managed to write some rules that give me a reasonable solution, that you can find here.

Basically the trick is to:

  • have a messages file containing your error messages (syntax.messages in the dune file linked)
  • have some rules to generate an updated version of your messages files. To do that, the dune file:
    • generates a new messages files from the .mly file (this is new.messages), this contains empty/dummy error messages for all the states from your parser
    • create an updated version of the current messages file (mainly to update state number and if I recall correctly, remove messages for now non-existing states)
    • merge these two files using the dedicated menhir feature. This results in a messages file that contains all of the original messages, plus empty/dummy error messages for new states, and minus the now non-existing states
  • lastly, have a dune test that diffs the current messages files (the one commited in the repo), with the one generated by the step above. That way, when running tests, you’ll get an error if some states are missing from your message file, and you can easily use the promote feature to update you message file, and then update the message for the new states by hand (following the diff that was printed by the test).

Probably not much, thanks. Don’t really need a parser once the ‘.’ has been changed to ‘./’ - the Scheme reader already parses everything. Well, at least the once I’m using does, I haven’t tested other implementations. In any case, my thinking is just that we might see more tools if Scheme implementations could read dune files; the only thing blocking that (to my knowledge) is the dot ‘.’ in lists.

1 Like

It would be a small feature, I personally would be grateful if a dune file parser existed. There have been times where I would have liked to be able to extract dune-project data programmatically (e.g. the version number, license name, etc.). This was never for anything important, but it would have been convenient. Although I’m not a fan of JSON, YAML, or any of those other configuration formats, I appreciate how you can easily read them with basically any scripting language.

@zozozo Thanks for sharing this! Your rules are much more comprehensive than mine, and may be a good model for the kind of thing Dune could do natively. I’ll file an issue with the Dune repo soon.

Edit: it looks like there’s already an open issue about Menhir messages: Support for menhir's `.messages` files · Issue #3284 · ocaml/dune · GitHub

Is there interest in generating system-independent Makefiles from dune?

I did some experiments on modifying the output of dune rules --makefile: xen/tools/ocaml/dune2makefile at xen-builds5 · edwintorok/xen · GitHub.
The idea was to remove absolute paths, and otherwise system specific configuration from it so that the makefile could be reused on another machine, running another distro, that doesn’t have ‘dune’ installed.
Although this may be brittle and not guaranteed to work for long it’d only need to hold together for a transition period.
OTOH doing these changes in ‘dune’ itself might be easier (rather than parsing the output of dune…), but I wouldn’t want to impose the additional maintenance burden for what might be a “one-off” feature used for a few months/years and then not used again.

Having makefiles generated from ‘dune’ would allow a more gradual transition from hand-written – and often incorrect – Makefiles to dune in projects such as Xen, where a sudden transition isn’t really possible due to part of the CI running on Debian oldstable which doesn’t have a suitable version of dune available.
OTOH a simple shell script can compile the entire OCaml code in Xen a lot more quickly than the recursive Makefile, and similarly modifying the current recursive Makefile to have a single hand-written Makefile with no incremental compilation that just builds everything in a single rule is still quite fast, comparable in time to how long make clean takes (which has accumulated sufficient cruft along the years that it is quite slow), and avoids all the problems with the old makefile: tools/ocaml: simplify and speed up build system · edwintorok/xen@96ef0e3 · GitHub
But obviously that Makefiles will then need to be maintained by hand alongside the ‘dune’ files for a transition period.

1 Like

As a slight tangent, not sure if this is a problem that should be fixed, more of a comment, but when working with beginners I’ve noticed some getting tripped up by the fact that while most dune commands are specific to the current directory (such as dune build, exec etc.), dune clean is project wide, and deletes all generated build artefacts.

This came up in the context of encoding a rather complicated build recipe (with Coq files that take minutes to compile) into dune and thus doing dune clean in one directory ended up requiring several minutes of compilation work to be done on the next compile.

Admittedly the reason we were using dune clean was because our build rules weren’t accurate recording their dependencies, and a good way of weeding this out was by running dune clean and then rebuilding particular subdirectories.

Insisting one last time :wink:
I get that but I think it’s counterintuitive to have this as a default behavior. It’s quite rare to find commands that go upwards by default. Feels like a global side effect. As a teacher and sporadic developer of research tools, I’d rather have:

  • at most upwards exporation until meeting a dune-project file (not dune-workspace)
  • subdir-like clauses in the said file to even avoid adding dune files anywhere (for simple projects; more complex projects may need specific dune files)
  • a clause in the dune-project file to allow going even more upwards (to accomodate the tool to the nice feature you describe)
1 Like

git and mercurial do that (and it is very convenient).

1 Like

I’ve updated Run inline tests in parallel · Issue #1516 · ocaml/dune · GitHub to explain how it could work.

Git tracks files that you have explicitly staged (or perhaps you’re speaking of other aspects of Git?; I don’t know Mercurial). Dune compiles files upwards if you don’t explicitly prevent it to do so. Even old files, small experiments that you made, old dune projects… and then you get dozens of errors completely unrelated to the project you think you’re compiling: speaking as an OCaml teacher, it’s not newcomer-friendly at all. Hence the middle-ground I’m proposing above for simple projects (typically those a newcomer will encounter).

1 Like

git and mercurial do that (and it is very convenient).

Not sure if we are discussing the same issue I had, but one major difference is that in the case of dune changing something in the directory above can change the behaviour for the current directory. This can give weird confusing issues, things that stop working if you move a dune project on your disk, etc. I have this “global side effect” feeling with dune, but not with git. A git repo inside a git repo behaves as you expect it.

BTW we’ve been running with (implicit_transitive_deps false) for like 10 months in our whole codebase and it’s been so challenging that it looks like we’re going to revert this back apparently. It’s a shame as the absence of it was also a major pain as well. I’m wondering if something can be done upstream

Something can surely be done upstream. There’s a proposal for it in fact Add `-Ihidden` in addition to `-I` for avoiding transitive dependencies in the initial scope by bobot · Pull Request #31 · ocaml/RFCs · GitHub

This has been included in dune 3.8

2 Likes

It would be nice if the dune init templates were updated to prevent this issue that many newcomers runs into: Unbound module for library if library name the same as test name · Issue #778 · ocaml/ocaml-lsp · GitHub

ppx_inline_test will run inline_tests in parallel starting with version v.0.16.1.

2 Likes