[ANN] Dune 3.0.0

Community
,
1

On behalf of the dune team, I’m delighted to announce the availability of dune 3.0.

The team has been working on this release for over 6 months, and there’s a bunch of new work to report. I’ll only highlight the some of the interesting new developments:

  • The watch mode has been rewritten from scratch to be faster and more scalable. We also no longer rely on any 3rd party tools such as fswatch. If any of you still have a dune workspace dune is still struggling with, we cannot wait to hear from you.

  • The watch mode now also starts an RPC server in the background. This RPC protocol is going to be the basis for other tools to interact with dune. Watch out for announcement on the LSP side to see how we’ll be making use of it to improve the editing experience.

  • The dune cache has been rewritten as well. It is now simpler and more reliable. There are still some components missing, such as distribution of the artifacts over the network. Nevertheless, we welcome you all to experiment with this feature and give us feedback.

  • We’ve addressed one of our oldest feature requests: high level rules for ctypes projects. This feature is still experimental, so we need feedback from real world projects before declaring it as mature.

Of course, there are many other fixes, enhancements, and only a few breaking changes in this release. We hope you have an easy time upgrading.

Happy Hacking.

3.0.0 (11/02/2022)

  • Remove uchar and seq dummy ocamlfind libraries from dune’s builtin
    library database (#5260, @kit-ty-kate)

  • Add a DUNE_DIFF_COMMAND environment variable to match --diff-command
    command-line parameter (@raphael-proust, fix #5369, #5375)

  • Add support for odoc-link rules (#5045, @jonludlam, @lubegasimon)

  • Dune will no longer generate documentation for hidden modules (#5045,
    @jonludlam, @lubegasimon)

  • Parse the native_pack_linker field of ocamlc -config (#5281, @TheLortex)

  • Fix plugins with dot in the name (#5182, @bobot, review @rgrinberg)

  • Don’t generate the dune-site build part when not needed (#4861, @bobot,
    review @kit-ty-kate)

  • Fix installation of implementations of virtual libraries (#5150, fix #3636,
    @rgrinberg)

  • Run tests in all modes defined. Previously, jsoo was excluded. (@hhugo,
    #5049, fix #4951)

  • Allow to configure the alias to run the jsoo tests (@hhugo, #5049, #4999)

  • Set jsoo compilation flags in the env stanza (@hhugo, #5049, #1613)

  • Allow to configure jsoo separate compilation in the env stanza. Previously,
    it was hard coded to always be enabled in the dev profile. (@hhugo, #5049,
    fix #970)

  • Fix build-info version in jsoo executables (@hhugo, #5049, fix #4444)

  • Pass -no-check-prims when building bytecode for jsoo (@hhugo, #5049, #4027)

  • Fix jsoo builds when dynamically linked foreign archives are disabled
    (@hhugo, #5049)

  • Disallow empty packages starting from 3.0. Empty packages may be
    re-enabled by adding the (allow_empty) to the package stanza in
    the dune-project file. (#4867, fix #2882, @kit-ty-kate, @rgrinberg)

  • Add link_flags field to the executable field of inline_tests (#5088,
    fix #1530, @jvillard)

  • In watch mode, use fsevents instead of fswatch on OSX (#4937, #4990, fixes
    #4896 @rgrinberg)

  • Remove inotifywait watch mode backend on Linux. We now use the inotify API
    exclusively (#4941, @rgrinberg)

  • Report cycles between virtual libraries and their implementation (#5050,
    fixes #2896, @rgrinberg)

  • Warn when lang versions have an ignored suffix. (lang dune 2.3.4) or (lang dune 2.3suffix) were silently parsed as 2.3 and we know suggest to remove
    the prefix. (#5040, @emillon)

  • Allow users to specify dynamic dependencies in rules. For example (deps %{read:foo.gen}) (#4662, fixes #4089, @jeremiedimino)

  • Sandbox infer rules for menhir. Fixes possible “inconsistent assumptions”
    errors (#5015, @rgrinberg)

  • Experimental support for ctypes stubs (#3905, fixes #135, @mbacarella)

  • Fix interpretation of binaries defined in the env stanza. Binaries
    defined in x/dune wouldn’t be visible in `x/*/**/dune. (#4975, fixes #4976,
    @Leonidas-from-XIV, @rgrinberg)

  • Do not list private libraries in package listings (#4945, fixes #4799,
    @rgrinberg)

  • Allow spaces in cram test paths (#4980, fixes #4162, @rgrinberg)

  • Improve error handling of misbehaving cram scripts. (#4981, fix #4230,
    @rgrinberg)

  • Fix foreign_stubs inside a tests stanza. Previously, dune would crash
    when this field was present (#4942, fix #4946, @rgrinberg)

  • Add the enabled_if field to inline_tests within the library stanza.
    This allows us to disable executing the inline tests while still allowing for
    compilation (#4939, @rgrinberg)

  • Generate a dune-project when initializing projects with dune init proj ...
    (#4881, closes #4367, @shonfeder)

  • Allow spaces in the directory argument of the subdir stanza (#4943, fixes
    #4907, @rgrinberg)

  • Add a %{toolchain} expansion variable (#4899, fixes #3949, @rgrinberg)

  • Include dependencies of executables when creating toplevels (either dune top or dune utop) (#4882, fixes #4872, @Gopiancode)

  • Fixes opam META file requires entry for private libs (#4841, fixes #4839, @toots)

  • Fixes dune exec not adding .exe on Windows (#4371, fixes #3322, @MisterDA)

  • Allow multiple cinaps stanzas in the same directory (#4460, @rgrinberg)

  • Fix $ dune subst in empty git repositories (#4441, fixes #3619, @rgrinberg)

  • Improve interpretation of ansi escape sequence when spawning processes (#4408,
    fixes #2665, @rgrinberg)

  • Allow (package pkg) in dependencies even if pkg is an installed package
    (#4170, @bobot)

  • Allow %{version:pkg} to work for external packages (#4104, @kit-ty-kate)

  • Add (glob_files_rec <dir>/<glob>) for globbing files recursively (#4176,
    @jeremiedimino)

  • Automatically generate empty .mli files for executables and tests (#3768,
    fixes #3745, @CraigFe)

  • Add ocaml command subgroup for OCaml related commands such as utop, top,
    and merlin (#3936, @rgrinberg).

  • Detect unknown variables more eagerly (#4184, @jeremiedimino)

  • Improve location of variables and macros in error messages (#4205,
    @jeremiedimino)

  • Auto-detect dune-project files as dune files in Emacs (#4222, @shonfeder)

  • Dune no longer automatically create or edit dune-project files
    (#4239, fixes #4108, @jeremiedimino)

  • Warn if dune-project is not found (fatal in release mode) (#5343, @emillon)

  • Cleanup temporary files after running $ dune exec. (#4260, fixes #4243,
    @rgrinberg)

  • Add a new subcommand dune ocaml dump-dot-merlin that prints a mix of all the
    merlin configuration of a directory (defaulting to the current directory) in
    the Merlin configuration syntax. (#4250, @voodoos)

  • Enable cram tests by default (#4262, @rgrinberg)

  • Drop support for opam 1.x (#4280, @jeremiedimino)

  • Stop calling ocamlfind to determine the library search path or
    library installation directory. This makes the behavior of Dune
    simpler and more reproducible (#4281, @jeremiedimino)

  • Remove the external-lib-deps command. This command was only
    approximative and the cost of maintainance was getting too high. We
    removed it to make room for new more important features (#4298,
    @jeremiedimino)

  • It is now possible to define action dependencies through a chain
    of aliases. (#4303, @aalekseyev)

  • If an .ml file is not used by an executable, Dune no longer report
    parsing error in this file (#4330, @jeremiedimino)

  • Add support for sandboxing using hard links (#4360, Andrey Mokhov)

  • Fix dune crash when subdir is an absolute path (#4366, @anmonteiro)

  • Changed the implementation of actions attached to aliases, as in
    (rule (alias runtest) (action (run ./test))). A visible result for
    users is that such actions are now memoized for longer. For
    instance:

    $ echo '(rule (alias runtest) (action (echo "X=%{env:X=0}\n")))` > dune
$ X=1 dune runtest
X=1
$ X=2 dune runtest
X=2
$ X=1 dune runtest

    Previously, Dune would have re-executed the action again at the last
    line. Now it remembers the result of the first execution.

  • Fix a bug where dune would always re-run all actions that produce symlinks,
    even if their dependencies did not change. (#4405, @aalekseyev)

  • Fix a bug that was causing Dune to re-hash generated files more
    often than necessary (#4419, @jeremiedimino)

  • Fields allowed in the config file are now also allowed in the
    workspace file (#4426, @jeremiedimino)

  • Add options to control how Dune should handle stdout and stderr of
    actions when then succeed. It is now possible to ask Dune to ignore
    the stdout of actions when they succeed or to request that the
    stderr of actions must be empty. This allows to reduce the noise of
    large builds (#4422, #4515, @jeremiedimino)

  • The @all alias no longer depends directly on copies of files from the source
    directory (#4461, @nojb)

  • Allow dune-file as an alternative file name for dune files (needs to be
    enabled in the dune-project file) (#4428, @nojb)

  • Drop support for upgrading jbuilder projects (#4473, @jeremiedimino)

  • Extend the environment variable BUILD_PATH_PREFIX_MAP to rewrite
    the root of the build dir (or sandbox) to /workspace_root (#4466,
    @jeremiedimino)

  • Simplify the implementation of build cache. We stop using the cache daemon to
    access the cache and instead write to and read from it directly. The new cache
    implementation is based on Jenga’s cache library, which was thoroughly tested
    on large-scale builds. Using Jenga’s cache library will also make it easier
    for us to port Jenga’s cloud cache to Dune. (#4443, #4465, Andrey Mokhov)

  • More informative error message when Dune can’t read a target that’s supposed
    to be produced by the action. Old message is still produced on ENOENT, but other
    errors deserve a more detailed report. (#4501, @aalekseyev)

  • Fixed a bug where a sandboxed action would fail if it declares no dependencies in
    its initial working directory or any directory it chdirs into. (#4509, @aalekseyev)

  • Fix a crash when clearing temporary directories (#4489, #4529, Andrey Mokhov)

  • Dune now memoizes all errors when running in the file-watching mode. This
    speeds up incremental rebuilds but may be inconvenient in rare cases, e.g. if
    a build action fails due to a spurious error, such as running out of memory.
    Right now, the only way to force such actions to be rebuilt is to restart
    Dune, which clears all memoized errors. In future, we would like to provide a
    way to rerun all actions failed due to errors without restarting the build,
    e.g. via a Dune RPC call. (#4522, Andrey Mokhov)

  • Remove dune compute. It was broken and unused (#4540,
    @jeremiedimino)

  • No longer generate an approximate merlin files when computing the
    ocaml flags fails, for instance because they include the contents of
    a file that failed to build. This was a niche feature and it was
    getting in the way of making Dune’s core better. (#4607, @jeremiedimino)

  • Make Dune display the progress indicator in all output modes except quiet
    (#4618, @aalekseyev)

  • Report accurate process timing information in trace mode (enabled with
    --trace-file) (#4517, @rgrinberg)

  • Do not log live_words and free_words in trace file. This allows using
    Gc.quick_stat which does not scan the heap. (#4643, @emillon)

  • Don’t let command run by Dune observe the environment variable
    INSIDE_EMACS in order to improve reproducibility (#4680,
    @jeremiedimino)

  • Fix root_module when used in public libraries (#4685, fixes #4684,
    @rgrinberg, @CraigFe)

  • Fix root_module when used with preprocessing (#4683, fixes #4682,
    @rgrinberg, @CraigFe)

  • Display Coq profile flags in dune printenv (#4767, @ejgallego)

  • Introduce mdx stanza 0.2, requiring mdx >= 1.9.0, with a new generic deps
    field and the possibility to statically link libraries in the test
    executable. (#3956, #5391, fixes #3955)

  • Improve lookup of optional or disabled binaries. Previously, we’d treat every
    executable with missing libraries as optional. Now, we treat make sure to
    look at the library’s optional or enabled_if status (#4786).

  • Always use 7 char hash prefix in build info version (#4857, @jberdine, fixes
    #4855)

  • Allow to explicitly disable/enable the use of dune subst by adding a
    new (subst <disable|enable>) stanza to the dune-project file.
    (#4864, @kit-ty-kate)

  • Simplify the way dune discovers the root of the workspace. It now
    stops at the first dune-workspace file it encounters, and fails if
    it finds neither a dune-workspace nor a dune-project file
    (#4921, fixes #4459, @jeremiedimino)

  • Dune no longer reads installed META files for libraries distributed with the
    compiler, instead using its own internal database. (#4946, @nojb)

  • Add support for (empty_module_interface_if_absent) in executable and library
    stanzas. (#4955, @nojb)

  • Add support for %{bin-available:...} (#4995, @jeremiedimino)

  • Make sure running git or hg in a sandboxed action, such as a
    cram test cannot escape the sandbox and pick up some random git or
    mercurial repository on the file system (#4996, @jeremiedimino)

  • Allow %{read:...} in more places such as (enabled_if ...)
    (#4994, @jeremiedimino)

  • Run each action in its own process group so that we don’t leave
    stray processes behind when killing actions (#4998, @jeremiedimino)

  • Add an option expand_aliases_in_sandbox (#5003, @jeremiedimino)

  • Allow to cancel the initial scan via Control+C (#4460, fixes #4364
    @jeremiedimino)

  • Add experimental support for directory targets (#3316, #5025, Andrey Mokhov),
    enabled via (using directory-targets 0.1) in dune-project.

  • Delete old promote-into, promote-until-clean and promote-until-clean-into
    syntax (#5091, Andrey Mokhov).

  • Add link_flags in the env stanza (#5215)

  • Bootstrap: ignore errors when trying to remove generated files. (#5407,
    @damiendoligez)

44 Likes
2

Is there an upgrade checklist somewhere or is it a drop-in replacement for 2.9?

3

Apart from the occasional missing dune-project file (now mandatory) and missing file dependencies to some custom rules you may have, it is supposed to be a drop-in replacement for 2.9

5 Likes
4

Hi @rgrinberg, I’m a bit puzzled, because $ dune build bin/foo.exe used to tell the opam commands to install missing packages in v2.9.3 but not so in v3.0.2.

Is this intentional or am I missing something?

5
  • Remove the external-lib-deps command. This command was only
    approximative and the cost of maintainance was getting too high. We
    removed it to make room for new more important features (#4298,
    @jeremiedimino)

Seems covered by this changelog entry

2 Likes
6

uh, thanks for the clarification – that was a life-saver for me when I started 2 years ago because I knew what to do and could go on.

1 Like