OCaml Platform Newsletter: September 2023

Community
1

Welcome to the sixth edition of the OCaml Platform newsletter!

Dive into the latest updates from September and discover how the OCaml Platform is evolving. Just like in previous newsletters, it spotlights the recent developments and enhancements to the OCaml development workflows.

In addition to the updates on the Platform team’s progress highlighted below, don’t hesitate to share your feedback on the upcoming Platform roadmap. We’ve just updated it based on the most recent feedback and are aiming to adopt it in the coming weeks, barring new concerns from the community.

Happy reading!

Releases

Here are all the new versions of Platform tools that were released this month:

For detailed release notes and announcements, explore the OCaml Changelog.

Building Packages

[Dune] Exploring Package Management in Dune

Contributors: @rgrinberg (Tarides), @Leonidas-from-XIV (Tarides), @gridbugs (Tarides), @kit-ty-kate (Tarides), @Alizter

The biggest highlight from September is that the work to expose the compiler and libraries from packages installed by Dune to the rest of the project is now complete! This means that there is now a prototype of Dune package management that can be used to build projects that depend on (simple) opam packages! This is still an early prototype that’s not ready to be tested outside of the core team, but still a significant milestone: :tada:!

In addition to this, work in September focussed on three areas:

  • Increasing coverage of opam features to support more opam packages from the opam-repository. This month, the Dune team added support for new fields, including build-env, setenv, and subst, and they also added support for patching.
  • Designing and implementing a string manipulation DSL for Dune configurations. This will allow users to express the same amount of dynamism found in opam filters in Dune package lockfiles, which is necessary for converting opam build and install commands into Dune expressions.
  • Started working on support for custom opam repositories by making the opam-repository configurable in dune-workspace. The next step is to experiment on how opam repositories are stored and accessed. One idea is that all opam repositories would be stored in one revision storage that would supply all the data. This has the advantage that incremental updates are small, which work like pulling via Git. The repo doesn’t need to be uncompressed, thus less storage and inodes used.

Activities:

[opam] Native Support for Windows in opam 2.2

Contributors: @rjbou (OCamlPro), @kit-ty-kate (Tarides), @dra27 (Tarides), @AltGr (OCamlPro)

In preparation for the upcoming release of opam 2.2~alpha3, the work has focussed on better handling of path rewriting for the setenv and build-env opam fields.

The proposed change will allow users to specify, in the opam file, the path separator and format they want for each environment variable in setenv/build-env. This ensures the environment variables are correctly set and usable on Windows.

The PR is in review and not quite ready to be merged, but this is the last issue scoped for opam 2.2~alpha3.

Activities:

[Dune] Dune Terminal User Interface

Contributors: @Alizter, @rgrinberg (Tarides)

Following the merge of the PR to port Dune TUI to Nottui in August, and the addition of a few features, @Alizter continued the work on building a full-on Terminal User Interface for Dune with two pull requests, namely the addition of a Jobs tab in tui mode, and support for multiline status lines.

Activities:

[Dune] Support on Niche Platforms

Contributors: @Alizter

Dune now builds on both Haiku and Android (using Termux)! This means it is now possible to build and install both OCaml and Dune on these platforms, which should pave the way for more native OCaml development.

For reference, here is a table of Dune’s platform support (with ? indicating that further testing is needed):

Platform Support Watch TUI Cache Sandboxing
Linux Full Yes Yes Yes Yes
MacOS Full Yes Yes Yes Yes
Windows (DKML) Full Yes No* Yes Copy only
Windows (MinGW) Limited Yes Yes Yes Yes
Windows (Cygwin) Limited Yes Yes Yes Yes
Linux (Android) Limited Yes Yes ? ?
FreeBSD Limited Yes Yes ? ?
NetBSD Limited Yes Yes ? ?
OpenBSD Limited ? ? ? ?
Haiku Limited Yes Yes ? ?

If you’re working on one of these platforms, don’t hesitate to open issues on Dune’s bug tracker if you encounter any problem!

Activities:

Generating Documentation

[odoc] Add Search Capabilities to odoc

Contributors: @panglesd (Tarides), @EmileTrotignon (Tarides), @julow (Tarides), @jonludlam (Tarides)

Work continues on adding search capabilities to odoc in order to improve the documentation browsing experience.

In September, the odoc team continued reviewing the different pull requests started in August. Peer-reviews suggested several improvements to to the CLI and the library API.

They also worked on client-side performance improvements by loading the search script only when the user clicks on the search bar, and they made quite a lot of progress on the UI overall.

Activities:

[odoc] Syntax for Images and Assets in odoc

Contributors: @panglesd (Tarides)

The effort to add support for images and assets to odoc and bring images to the OCaml.org package documentation continues!

This month, @panglesd opened a PR with an implementation for asset references.

The exact syntax for medias went through several designs, in particular whether a media is a block, a nestable block, or an inline element. At the end of the month, @panglesd created a PR that builds on the asset references PR in order to add support for medias.

For some time, there has been no official convention on how documentation for opam-installed packages should be built. With the added complexity of having assets, it was a good time to solve this. A documentation PR was opened for this. Warm thank you to @dbuenzli for the thourough review and participating in establishing these conventions!

Activities:

[Dune] Generate Dependencies Documentation with Dune

Contributors: @jonludlam (Tarides)

Currently, Dune only knows how to build the documentation for the packages in your Dune workspace, meaning that you can only read the documentation of your dependencies from the OCaml.org package site. Alternative odoc drivers, like odig, build documentation for all the packages in your switch and have been the recommended solution for users who prefer to read the dependencies’ documentation locally.

In an effort to improve the documentation generation experience with Dune, @jonludlam worked on a new version of Dune rules to generate the documentation. With these rules, Dune will gain the additional ability to build the combination of the two: a coherent set of docs that cover both switch-installed libraries and local libraries.

The PR is in review and is set to be merged in the coming weeks.

Future plans for the new rules include better integration with the rest of the platform, improvements in capabilities to cover the use cases that dune build @doc covers, integration of source rendering, and integration of search (once it lands in odoc!).

Activities:

Editing and Refactoring Code

[Merlin] Support for Project-Wide References in Merlin

Contributors: @voodoos (Tarides), @trefis (Tarides), @Ekdohibs (OCamlPro), @gasche (INRIA)

In August, the Merlin team opened the PR on the compiler that adds the necessary information in the Shapes to implement project-wide references.

The PR received reviews, so the team worked on taking the feedback into account while also continuing work on the rest of the stack (build system rules, the indexer and new locate, and occurrences backends for Merlin).

They also consolidated a release plan and timeline. The plan is to first release an experimental 4.14-based variant of the compiler in order to gather feedback on this eagerly awaited feature before the end of the year. The current aim is to provide official project-wide occurrences support in OCaml 5.2.

Activities:

[Merlin] Improving Merlin’s Performance

Contributed by: @pitag-ha (Tarides), @3Rafal (Tarides), @voodoos (Tarides), @let-def (Tarides)

The Merlin team continues work on improving Merlin performance.

Before diving into specific performance optimisation, they worked on a benchmarking CI to catch performance regressions and measure improvements. While at it, they’ve also worked on a fuzzy testing CI to catch behaviour regressions.

In September, following the Proof of Concept (PoC) of the fuzzy-testing CI (from the work in July), the team continued their work on addressing the limitations of the current CI implementation that would prevent it from being merged in Merlin. Specifically, they focussed on finding a solution to store the fuzzy testing results in a way that wouldn’t bloat the Merlin repository. The current approach is to store the data in a separate Git repository and pull it when running the fuzzy-testing CI. They’ve created a GitHub action workflow that implements this behaviour.

Next, the plan is to complete the work on the Merlin CI before gradually shifting focus to performance optimisations.

Activities:

6 Likes
2

Thanks for the good work. In a couple of posts last year, I had advocated a leaner user experience for newcomers with a few actionable issues:

  • having to create 4 files (dune, dune-workspace, dune-project, .ocamlformat) to build and format simple (< 10 files) projects is a big burden; please make it possible to just work with one file, with very little content, for basic projects
  • upwards exploration is a nightmare for students who forget to add a dune-workspace file: hopefully this could be addressed at the same time as the former item
  • we need warnings to be non-fatal and re-shown at every dune invocation (the dune stanza to have non-fatal warnings exists but no one can remember it), without inhibiting tests

Have you, or do you expect to address these issues?

1 Like
3

Thanks for summarising your suggestions here @grayswandyr!

having to create 4 files (dune, dune-workspace, dune-project, .ocamlformat) to build and format simple (< 10 files) projects is a big burden; please make it possible to just work with one file, with very little content, for basic projects

Part of this is in both the OCamlFormat roadmap (https://github.com/ocaml-ppx/ocamlformat/blob/main/ROADMAP.md) and the OCaml Platform roadmap (https://github.com/tarides/ocaml-platform-roadmap/blob/main/roadmap.md#w8-format-code). The OCamlFormat team did some work on this recently, see Display a warning for new enable/disable-outside-detected-project behavior by gpetiot · Pull Request #2439 · ocaml-ppx/ocamlformat · GitHub.

There have been discussions throughout the years on having a different configuration format for Dune. While there’s nothing planned yet as far as I can tell, I didn’t get the sense that Dune maintainers were against it in principle. I suggest opening an issue on ocaml/dune if you (or anyone) are interested in getting the ball rolling.

upwards exploration is a nightmare for students who forget to add a dune-workspace file: hopefully this could be addressed at the same time as the former item

I’m not certain what you mean by “upwards exploration”. Could you expand on this?

we need warnings to be non-fatal and re-shown at every dune invocation (the dune stanza to have non-fatal warnings exists but no one can remember it), without inhibiting tests

I vaguely remember conversations about this, although I am not able to find the issue on the Dune repository. Perhaps @rgrinberg or @emillon would remember the outcome of the discussions better?

4

Thanks for your quick answer. I can and will open issues but I also wanted to talk about this on the public forum. I’d just like to advocate that these issues look, to me at least, like low-hanging fruits for a far better user experience (I’m not claiming they are easy to implement for dune developers): “for simple projects, just throw in a single basic file and get OCaml rolling (build + utop + test + warnings + format (+ possibly publish a package, thanks to the recent developments described above))!”. Notice I’m not asking to change the format (if you mean switching from sexps), just to have one file. @rgrinberg had suggested doing this in dune-project IIRC.

All our students, one day or the other, forget to add a dune-workspace file and run dune. Then, dune explores the filesystem upwards in the hope of finding a dune-workspace above. Then dune tries to build all OCaml files below the root it found. This results in tens of errors because those files aren’t meant to be built together (they can also rely on older package versions or contain failed experiments).

I get that some people seem to like upwards exploration (which I’d tend to see as a global side effect) and there doesn’t seem to be a plan to change this default behavior. BUT if the idea of a single basic configuration file was implemented, then this would also solve this issue for newcomers.

5

Dune doesn’t require a dune-workspace file, just dune-project is enough: Command-Line Interface — Dune documentation.

2 Likes
6

You do need a root dune-workspace/dune root file to avoid the not-so-user-friendly error as warnings. (Through env stanza).

I think dune is amazing for larger projects. It just needs a few additions/defaults to make your very first ocaml setup as simple as possible.

1 Like