Plan for Dune 3.0

Community
,
1

The Dune team has started planning the next major release and we would like to share and discuss the current plan with the community. We expect to release Dune 3 in August 2021.

Please have a look and give us some feedback: Plan for Dune 3.0

This plan is not set in stone. If you don’t like it, let’s discuss and see how we can improve it.

22 Likes
2

Better support for warnings sounds intriguing, but I’m wondering how that will actually work in practice. Is there a sketch for what functionality you’d aim to provide?

One thing I could imagine is for warnings to show up as Merlin squiggles, but not to otherwise show up in build feedback.

3

Could you provide some more details on this please?

Calling ocamlfind at runtime. When running Dune outside of an opam environment, Dune invokes ocamlfind if it is present to figure out where to look for installed libraries. This is bad for reproducibility and doesn’t feel useful anymore, so we plan to stop doing that. Dune will however continue to use the OCAMLPATH variable, as well as the search paths hard-coded at build time by the optional ./configure script.

What does it mean for people running Dune outside of an opam environment? Does it mean that running Dune without opam will no longer be supported? Or is it still supported, but limited, by requiring that search paths be known at the time of building the Dune executable from source?

Thank you.

4

The current implementation of scoping has wrong semantics for nested vendored projects: a public library in the inner vendored project escapes its scope and is globally visible. This loophole caused problems for users, and we are going to remove it.

There seem to be two conflicting uses of vendoring in dune. One is to make a library private, and the other (which I tend to use) is an alternative to opam pins during development. When using vendoring-as-pinning, the library should still be considered public. It would be good if there were a way to specify which kind of vendoring you wanted.

5 Likes
5

We have a few ideas, but no concrete proposal yet. In a nutshell, we’d like the presence of warnings to mark the overall build as failed but not make it fail immediately. Also, during incremental compilation warnings should be displayed until they are fixed. Right now, when warnings are not treated as errors they are displayed during the first compilation and not subsequent ones.

/cc @bobot who was interested in this

2 Likes
6

Could you provide some more details on this please?

Sure. When looking for a library that is not defined locally, Dune 2.x looks for in the following directories:

  • first it looks in directories specified in the OCAMLPATH environment variable
  • if it doesn’t find it there, it checks if a set of directories was hard-coded at configure time
    • if yes, it looks in these directories
    • if no, it checks if OPAM_SWITCH_PREFIX is defined
      • if yes, it looks for it in $OPAM_SWITCH_PREFIX/lib
      • if no, it checks if ocamlfind is present in the PATH
        • if yes, it calls ocamlfind printconf path (unsetting OCAMLPATH) and looks for it in the directories reported by ocamlfind
        • if no, it looks for the library in $(which ocamlc)/../../lib

This means that when running outside of an opam environment, the behaviour of Dune depends on whether ocamlfind is installed or not. This is flaky given that there is no dependency between the two packages. So Dune 3.x will skip this step.

Concretely, ocamlfind printconf path uses the following source of informations:

  • the OCAMLPATH variable
  • the findlib.conf file, whose path is hard-coded at configfure time (or can be passed via OCAMLFIND_CONF)
  • a set of directories hard-coded at ocamlfind’s configure time

So, if your tooling/package manager is:

  • setting OCAMLPATH, there is nothing to do to adapt to Dune 3.x
  • hardcoding a set of directories in ocamlfind, then it should do the same for Dune (we checked and Debian does that for instance)
  • doing neither of the above and relying on the findlib configuration file, then we should probably discuss

What does it mean for people running Dune outside of an opam environment? Does it mean that running Dune without opam will no longer be supported? Or is it still supported, but limited, by requiring that search paths be known at the time of building the Dune executable from source?

This is a good time for us to learn about these use cases. In which environment do you use Dune?

1 Like
7

What is your workspace layout? Only workspaces using nested vendoring will be affected.

8

I had a similar concern, since I use Nix to set up my OCaml development environment. Sounds like $OCAMLPATH is being used, though, so I think this doesn’t affect my use case.

9

Just to confirm, Dune will still be able to read and generate META files, correct? Internally we have packages built by both Dune and other build systems. We rely on META files for interop between them.

1 Like
10

do you guys have plans for a generic release-mode?
I’m mostly interested in optimized & stripped executables when I invoke dune install without having to write custom rules.

1 Like
11

What is your workspace layout? Only workspaces using nested vendoring will be affected.

We do sometimes nest (e.g. ocaml-ci vendored ocluster, which vendored obuilder):

The latest master is only one-level deep, though.

12

I believe dune-release is more specific to a github-based workflow?

13

Yes, dune will still read and generate META files.

1 Like
14

Yes, we’re aware of the overloaded terminology and I agree that it’s a good idea to distinguish between package management and vendoring. We discuss it in a little more detail here: Vendored_dirs and release mode · Issue #3911 · ocaml/dune · GitHub

3 Likes
15

Thank you @jeremiedimino for the detailed answer. As @hongchangwu mentioned, we’re using dpkg for packaging.

Since @rgrinberg clarified that:

Yes, dune will still read and generate META files.

We think we should be fine with removing the dependency on the ocamlfind executable being available, because we do set an OCAMLPATH where the relevant META files exist.

1 Like
16

By “stripped”, do you mean executables compiled without -g? I’m afraid we don’t have any plans to add something specific here, but we do let users customize flags. Is that not enough?

17

Well, the compiler doesn’t have a flag corresponding to gcc’s -s. So my usual workflow involves specifying ocamlopt flags without -g and like so: -ffunction-sections -compact -ccopt -Os -ccopt -s -ccopt -flto -ccopt -Wl,--gc-sections (ocaml >= 4.10) followed by a custom rule that runs strip -s on the resulting executable just to be extra sure.

The shrink in size vs a default dune install is around 30~50% and reaches ~75% for smaller executables.

4 Likes
18

For the warnings, a first task is just to look at how people customize warnings, and if the documentation is clear on those points. We could have specific ways to specify those compiler flags. Perhaps using the new named warnings from the compiler.

The second is indeed as you propose to see if we can show the warnings at different times. Warnings specific to the editor is a good idea. Another possibility is for some warnings such as unused value to delay them to runtest or a runcheck, which would print the warnings delayed (e.g logged separately) and fail if it is not empty. But one design question is what is the set of compiled file considered.

The third is to clarify what we do for new compilers with new warnings.

19

I was just thinking about a release mode since it’s a common thing among build tools. Many distros packaging software (and I can imagine they’re not the only use case) opt for turning on extra optimizations, minimizing executable size, and stripping executables in favour of separate debug symbols.

2 Likes
20

(Since dunes plan to remove automatic creating dune-project in 3.0)

How is your current solution to prevent running dune in a directory?

It occurs when I have code/pj1 code/pj2 but carelessly run dune at code where code is just a folder to put my projects.

3 Likes