[ANN] Dune Developer Preview Updates

[ostera]

Hi folks!

Just wanted to share some of the work the Dune has been up to lately re: the Developer Preview we announced here – we’ll be using this thread to share more updates as things go.

As always, we hold our Dune Developer meetings in public and you’re more than welcome to subscribe to our public Calendar (Google, iCal)

---

Getting ready for the Public Beta

As we prepare for the public beta, we’re ramping up the DX interviews and ensuring the first few users will have a fun, productive experience with the developer preview.

If you signed up for the Dev Preview back in May, check your inbox for a link and instructions to schedule your DX interview with us.

Here’s a sample video (Mastodon or X) where you can see me building the Riot project on a machine that does not have OCaml installed. It is pretty neat!

Seriously, big shoutout to the Dune team at Tarides[0] and Jane Street[1] who have been doing a phenomenal job

So here’s what getting started with OCaml looks like today with the Dune Developer Preview as of today (August 19 2024):

1. get dune from our binary distribution – we’ll soon make this public!
2. run dune pkg lock in your favorite project
3. run dune build

That’s it. No need to install anything else, Dune will see that lock file, fetch, and build all necessary dependencies.

These are some strong step towards the OCaml Platform vision for 2026, that we are actively working towards. If you have any thoughts or feedback please let me know!

There are more improvements coming that will help remove friction to get started and creating a delightful experience. Both of these things we strongly believe will help onboard new users to the OCaml world.

Here’s a few in the works:

• Various DX improvements – from new outputs to simplified workflows, we want to make using Dune just delightful.

• Bundled support for dev tools (ocalmformat, odoc, lsp) – the default toolset will be available without any extra steps! just call dune fmt, and it works. No need to manually install anything else.

• Automatic dependency locking – when building, and even on watch mode, Dune will lock your dependencies by default and keep the lock up to date.

• Cross-project Caching – by default we’ll enabled a local Dune cache that across the system, so you never rebuild the same dependency even across projects.

• Signed binaries with certificates of origin – we care deeply about security and want to make sure that any binary we ship will be easily verified and tracked back to its sources.

Stay tuned!

PS: here’s a longer video (Mastodon, X) showing you the setup for OCaml from zero, creating a new project, and adding a dependency, all within ~5 minutes

[0] @emillon @Leonidas @gridbugs @tmattio @maiste . Ambre Suhamy, Alpha Diallo
[1] @rgrinberg

[c-cube]

This looks absolutely incredible. Congratulations to the dune team!!

[lthms]

Pretty great work!

@ostera

run dune pkg lock in your favorite project

Quick question about that. We are using opam lock at $WORK and we ran into something a bit particular about it: for recent enough packages of OCaml (3mo and less), OCaml now comes with system/arch dependent packages. Which means we need to slightly modify the lock file generated by opam for it to be portable.

Not the end of the world, but still a bit annoying.

I think I read dune is “opam free” but do you take this into account already?

[yawaramin]

See this thread Dune lockfiles are not cross-platform. How should we approach checking them into version control?

[ostera]

Another small DX update

• the output of builds is now much easier to scan by being aligned in columns
• and the colors look good on light themes too!

[ostera]

@lthms

Quick question about that. We are using opam lock at $WORK and we ran into something a bit particular about it: for recent enough packages of OCaml (3mo and less), OCaml now comes with system/arch dependent packages. Which means we need to slightly modify the lock file generated by opam for it to be portable.

Not the end of the world, but still a bit annoying.

I think I read dune is “opam free” but do you take this into account already?

Hi @lthms! As @yawaramin pointed out, there’s a thread where we are discussing and gathering community inputs on how to make lockfiles work better. We hope to end up with a solution that is automated and portable, but it may take some iteration to get there! (or even work outside of lockfiles themselves)

I’d love to look at your dep graph and learn about these manual updates you’re doing, so feel free to reach out to me over discord (same username)

[ostera]

Hello folks!

Call for Feedback

We’d like to welcome everyone to try and play with the Dune Developer Preview!

This experimental nightly release of dune includes a lot of improvements, including the much expected package management features, and it can be installed from that website or by using the new installation script:

$ curl https://dune.ci.dev/install | bash

In a few seconds you should be ready to OCaml by calling dune:

You can also watch and share this demo on X and Mastodon.

Please try it out and let us know what you think

You can book a feedback call with us here

You can submit feedback using this form

You can submit issues to Github on ocaml/dune

Changes since last update

The Dune shared cache has been enabled by default. We’re starting off by caching all downloads and dependencies.

We have improved support for dev tools. We’re working to streamline this but in the latest binary you can:

• Configure your LSP (in Neovim, Vim, Emacs, etc) to call dune tools exec ocamllsp to get LSP support for your projects out of the box – this may take a little bit the first time it builds the LSP for a compiler version, but its pretty much instant afterwards.

• Call dune fmt to get your project formatted – remember to add an .ocamlformat file if you don’t have one yet. An empty one is enough.

• Call dune ocaml doc to get documentation built

What’s next?

We’re looking forward to streamlining the DX, working on better dependency locks, and looking into supporting Windows.

In particular, we’re considering work on a few things:

• dune create <repo> – to let the community create templates that can be easily used from within dune
• dune pkg fetch – to prefetch packages and prepare a repository for working in offline mode
• dune build @deps - to build all dependencies, useful for staged builds in Dockerfiles
• dune pkg add <name> - to make adding packages straightforward
• a short-hand syntax for pins on github
• and more!

If you’ve got any ideas, we’d love to hear them, so please open a feature request on Github

Other updates

FunOCaml Presentation

At FunOCaml we had a last-minute opportunity to present the work being done on Dune and we used it to introduce the Developer Preview to the community, and even tested Package Management live with suggestions from the audience (thanks @anmonteiro and Paul-Elliot for participating!) – you can watch it on Twitch.

New design

We’re working with @Claire_Vandenberghe on redesigning the Developer Preview website so that it’d feel like a seamless extension of OCaml.org – in this current iteration we’ve made it easier to get started and we’re putting the FAQ front and center.

We’ll be iterating on this design in the coming weeks until it fits perfectly within the OCaml.org design system

You can check the new website here: https://dune.ci.dev

Upcoming Blog posts

In the near future we’ll be publishing blog posts about the Developer Preview and Package Management, which we’re working on with @professor.rose

[cdaringe]

There is serious polish here . Perhaps the “ocaml tooling <complaint>” hackernews tropes are nearing their end ! Excellent work!

[bhoot]

Great work! I was able to re-build my old projects with the nightly dune without a hitch!

The Dune shared cache has been enabled by default. We’re starting off by caching all downloads and dependencies.

Where’s the cache located? Nightly dune doesn’t seem to use ~/.opam. Within the project dir, dune.lock/<dep>.pkg contains a package’s declaration. But I couldn’t find its cache.

[Frederic_Loyer]

I have it worked well with a project with multiple dependencies.

But now, a trivial try:

loyer@ak-serveur3:~$ dune init project essai2 
Entering directory '/home/loyer/essai2'
Success: initialized project component named essai2
loyer@ak-serveur3:~$ cd essai2
loyer@ak-serveur3:~/essai2$ dune pkg lock
Solution for dune.lock:
- base-threads.base
- base-unix.base
- dune.3.16.0
- ocaml.5.1.1
- ocaml-base-compiler.5.1.1~rc1
- ocaml-config.3
loyer@ak-serveur3:~/essai2$ dune build
Error: Multiple rules generated for
_build/_private/default/.pkg/ocaml-base-compiler/source:
- dune.lock/ocaml-base-compiler.pkg:24
- dune.lock/ocaml-base-compiler.pkg:13
-> required by - package ocaml-config
-> required by - package ocaml

The version of the base compiler is weird (why a rc1) ?

[ostera]

@cdaringe

There is serious polish here . Perhaps the “ocaml tooling <complaint>” hackernews tropes are nearing their end ! Excellent work!

Thank you! I’ll pass the praise on to the team. And yes, we’d love it if the next time OCaml shows up on the orange website is because people are amazed at the onboarding experience.

@bhoot

Great work! I was able to re-build my old projects with the nightly dune without a hitch!

Thank you! Glad it Just Worked – I appreciated the responses you left on the Lobste.rs post too

@bhoot

Where’s the cache located? Nightly dune doesn’t seem to use ~/.opam. Within the project dir, dune.lock/<dep>.pkg contains a package’s declaration. But I couldn’t find its cache.

The Developer Preview doesn’t rely on opam the tool, only on the opam registry. So instead of ~/.opam we are currently using the XDG_CACHE directory, which should be ~/.cache/dune for most people.

We may move this to ~/.dune since that’s a very common / expected place to find these dirs.

@Frederic_Loyer

I have it worked well with a project with multiple dependencies.

But now, a trivial try: […]

The version of the base compiler is weird (why a rc1) ?

Since we’re pulling the latest opam-registry when resolving dependencies, if there’s a new one coming up that matches the constraints of the project, it’ll get picked up. For some projects that sometimes meant that OCaml 5.4 got chosen, which was unexpected!

I’ll bring this up with the team. In the meantime can you try constraining this to a strict 5.1.1 in your dune-project? I’m otherwise available for help on the dune channel of Discord

(package ...
   (depends 
     (ocaml (= 5.1.1))
     ...))

[Frederic_Loyer]

My bad…

My .profile added the opam directory in front of the PATH.

I have succeed in using dune after a fresh source .bashrc

And I forgot to do it again (thinking it is automatically done with a new shell).

With the right dune, it works.

But dune install fails (even in a root account and the new project create by dune int project).

[c-cube]

please do keep the XDG conventions, it’s nicer than cluttering ~ :-).

[bhoot]

The Developer Preview doesn’t rely on opam the tool, only on the opam registry.

Oh. I had in my mind that dune would use opam underneath.

To be clear, will opam the tool go out of use leaving behind only the registry, once the new dune (I call it Dune Messiah hehe) stabilises?

[ostera]

@c-cube

please do keep the XDG conventions, it’s nicer than cluttering ~ :-).

In the odd-chance that we don’t use XDG (like we do now), we’ll only ask you to swap ~/.opam for ~/.dune, I promise

@bhoot

will opam the tool go out of use leaving behind only the registry, once the new dune (I call it Dune Messiah hehe) stabilises?

No, opam (the tool) will not go away. There’s still a subset of the community that relies on it and continues to develop it.

But as we get closer to a stable standalone Dune, Dune will become the recommended way of installing and getting started with OCaml, as shown in the OCaml Platform Roadmap.

[octachron]

@ostera

dune will become the blessed way of installing and getting started with OCaml, as shown in the OCaml Platform Roadmap.

Let’s avoid any confusion here: the “OCaml Platform”(aka a Tarides subgroup)'s 'blessed way", this opinion is not necessarily shared by other groups in the community.

For instance, I have no plans to switch to dune for package management.

[avsm]

@octachron

Let’s avoid any confusion here: the “OCaml Platform”(aka a Tarides subgroup)'s 'blessed way", this opinion is not necessarily shared by other groups in the community.

For instance, I have no plans to switch to dune for package management.

Yes indeed, let me quite clear here on my position on this:

• the last time we deprecated a working workflow in the OCaml Platform was opam 1.1 – an opam init on opam 1.2 still works! We try really hard to keep anything already released working.
• the opam client is the blessed way of installing on Tier 1/2 supported OCaml platforms. All of our resources in terms of accepting packages into the opam repository are testing the opam 2.0, 2.1, 2.2 and 2.3alpha matrix today, on numerous platforms.
• this dune preview is an alternative packaging client that is under rapid development, but is nowhere near ready yet for any claims of replacing opam, if ever (see the first point above). This is currently a preview of an unreleased beta, so let’s please watch out for overclaiming @ostera on this aspect of ocaml.org.
• under all of this, the good old opam-repository continues unaffected. Both the opam and dune clients use it, and we work hard to maintain compatibility here. At some point, when the dune client leaves this ‘preview’ mode and actually releases something, it’ll be tested alongside the opam client. But that isn’t happening just yet.

I’ll take a look at the OCaml Platform roadmap next week and ensure that this is clear in what’s on the website.

(edit: the above might have sounded a little critical of the Dune packaging work. It’s not meant to be – I’m really excited to see an alternative client emerge, since that’s how we designed the opam repository. And I talked about the potential of a unified CLI tool back in 2017 at the OCaml Workshop, and it’s only materialising now thanks to all the hard work we see above. But, as with any software engineering effort, there’s a release cycle and iteration that needs to happen in the coming years to catch up with opam, and I’m just trying to balance the new stuff with the reliable existing stuff. Accordingly, the dune preview binaries will be on dune.build instead of ocaml.org for now; see ocaml/infrastructure#156 for why.)

[v-gb]

Hi,

I tried briefly, building seems to just work, but I’m not sure how to get dev tools working so that limits testing, and I have a few questions that don’t seem answered in the existing docs.

@ostera

We have improved support for dev tools. We’re working to streamline this but in the latest binary you can:

• Configure your LSP (in Neovim, Vim, Emacs, etc) to call dune tools exec ocamllsp to get LSP support for your projects out of the box – this may take a little bit the first time it builds the LSP for a compiler version, but its pretty much instant afterwards.
• Call dune fmt to get your project formatted – remember to add an .ocamlformat file if you don’t have one yet. An empty one is enough.
• Call dune ocaml doc to get documentation built

How does one use merlin/ocamlformat in emacs? You talk about LSP but Configuring Your Editor · OCaml Documentation says emacs/vim users don’t use LSP directly. The .emacs code added by opam isn’t clearly adjustable either.

Are we supposed to commit these *.lock directories, for the same reasons as *.opam.locked files? The files in the lock directories look noisy, for files we’d commit.

The libs to install seems to come from the dune project file. Is there a plan to either take dependencies from the dune files directly, or at least sync dune-project with the dune files? To avoid the synchronization caused by the required duplication.

Will dune support declaring/installing OS deps as well, like you can do with opam’s depexts field ?

[Frederic_Loyer]

I guess that if I plan to share a library or a program, I shouldn’t put some tools like utop in the dependency. But it was the only way to make it available through dune and be able to launch dune exec utop.

I am perhaps too hurry. I have seen in the roadmap the intent to have a dune repl (not yet implemented).

Same thing about lsp-server I suppose, but I guess that the OCaml platform should be adapted to support dune.

I have seen that Cygwin is not yet supported (the install program indicates it and abort).

[yawaramin]

This messaging is a bit confusing. The Roadmap has already been published. It quite clearly says,

Following (P5) (Tools are independent, yet unified), Dune becomes the only tool users need to use to develop in OCaml.

Are the main OCaml stakeholders not unified behind this vision? Should this Roadmap not have been published?