Welcome to the first issue of the OCaml Platform newsletter!
Following in the footsteps of the OCaml.org newsletter and inspired by the Multicore and Compiler newsletters, we’re excited to bring you monthly updates on the progress we’re making in improving the OCaml Developer Experience.
The OCaml Platform is the recommended set of tools to work and be productive with OCaml. The Platform tools fill gaps in the OCaml developer experience and allow developers to perform specific workflows (e.g. building packages, formatting code, generating documentation, releasing packages, etc.).
At the end of the day, all the work we’re doing on the OCaml Platform has one objective: improving the OCaml developer experience, so in this newsletter, we’ll present the progress we’re making on the different projects from the lens of these developer workflows. Based on the results of the OCaml surveys (2020 and 2022), discussions with industrial users, continuous community feedback on Discuss, and other sources of user input, here are the workflows we’re currently working on:
- Building Packages: Our immediate goal for the build workflow is to remove the friction associated to using two different tools for package management and build system. To this end, we plan on integrating opam capabilities directly into Dune, establishing it as the singular tool needed to build OCaml projects. As a by-product of this integration, we aim to improve other workflows such as working on multiple projects, cross-compilation, and improving the overall experience to get started with OCaml.
- Generating Documentation: The state of the OCaml Packages documentation is reported as a major pain point in the OCaml surveys (2020 and 2022). We’re working towards empowering OCaml developers to create high-quality documentation for their packages. Now that the documentation of packages is readily available on OCaml.org, we want to make writing documentation rewarding and straightforward. We’re working towards making Odoc suitable to create manuals by adding new features, improving the navigation, and expanding the odoc markup syntax to support rich content such as tables, images and graphs.
- Editing and Refactoring Code: We aim to enrich the OCaml editor support with more workflows to improve code navigation and automate refactoring. Our main focus currently is on adding support for project-wide references to Merlin. Future work will include implementing a project-wide rename query and queries such as renaming arguments. We are also working towards bringing the editor support for OCaml to the web and third-party platforms such as OCaml Playground, ReplIt, and GitHub Codespaces.
- Formatting Code: Our goal for formatting code is focused on improving accuracy, particularly for the to comments. We also want to strike the right balance between providing a default profile that appeals to most users and not requiring configuration to format your OCaml projects, while still maintaining a fully configurable formatter. Additionally, we plan to enhance the backward compatibility of ocamlformat and better integrate Dune and ocamlformat.
I’ll also take the opportunity to call for new contributors. Platform projects are always looking for new maintainers and contributors, so if you’re interested in the future of OCaml’s developer experience and would like to shape that future with us, please reach out to me or any Platform maintainers. If you’re an industrial user looking for support on the OCaml Platform and would like to fund the maintainers and the developments on the Platform tools, also don’t hesitate to reach out to me.
April has seen a flurry of activity, and we can’t wait to share our progress with you. So let’s get right to it!
In this inaugural issue, we’ll be discussing progress on the following projects:
- Building Packages
- Generating Documentation
- Editing and Refactoring Code
- Formatting Code
Here are the new versions of Platform tools we released in April. Have a look at the OCaml Changelog to read announcements and feature highlights!
We’re currently focused on building prototypes for different parts of the Dune workflow: source fetching, building non-Dune opam packages, and generating a lock file.
- Completed and merged a first version of Source Fetching
- Upstreamed a patch in opam to remove preprocessing of backwards-compatibility code. This helped to reduce the number of dependencies to vendor in Dune when vendoring opam.
- We merged a PR that lays the foundation and provides a skeleton for the features related to building opam packages.
- Following work on 0install-solver, which we’ll use as a solver in Dune 4, we open a PR on
ocaml/opam-repositoryto remove constraints on package versions in conflicts and a patch on opam for the same conflict.
Bringing Tier-1 support for Windows has been a dream for some time and April has seen us get closer than ever before to the first alpha of opam 2.2, which we expect in May. This early alpha is a big step towards the release of opam 2.2 with native Windows support, and is the result of a humongous amount of effort bringing together the work of many people done over the years.
- Configure: Ensure a complementary (32bit on 64bit platforms and 64bit on 32bit platforms) C compiler is installed on Windows ocaml/opam#5522
- Do not silently disable MCCS if a C++ compiler is not present ocaml/opam#5527
- Move the
.ocamlinitscript out of the root directory ocaml/opam#5526
- MCCS on Windows does not respect avoid-version ocaml/opam#5523
- Setup benchmarking using current-bench ocaml/opam#5525
- Update to latest
- Fix the compilation of
camlheaderwhen BINDIR contains escaping characters ocaml/opam#12214
- doc: ? evaluates to true if defined ocaml/opam#5512
- Stop the configure script from downloading and vendoring dependencies ocaml/opam#5511
- Remove preprocessing of backwards-compatibility code ocaml/opam#5508
- Fix linting on
- depexts: reword message ocaml/opam#5499
- Replace usage of CPPO ocaml/opam#5498
- Check for the presence of swhid_core in the configure script ocaml/opam#5497
- Add package stanza on all rules that depend on
Contributors: @emillon (Tarides)
In March, we started restructuring the Dune documentation according to the Diataxis framework. We opened a draft PR to demonstrate the overall target structure. The new structure will improve the usability of the documentation, allowing users to find the information that they are looking for, as well as enable us to better identify gaps that need to be addressed.
In April we’ve continued this work, filling in some of those gaps as well as rewriting documents to better fit in the intended quadrant of the framework.
- Turn “Automatic Formatting” into a how-to ocaml/dune#7479
- Move lexical conventions to reference ocaml/dune#7499
opam.rstinto 3 guides ocaml/dune#7515
- Add some info about writing docs ocaml/dune#7537
- Merge reference info about findlib ocaml/dune#7567
- Add a lexer for opam files ocaml/dune#7574
We’ve merged the PR that brings support for composing Coq theories with Dune!
This was a huge effort lead by Ali Caglayan and Emilio Jesús Gallego Arias that started earlier this year. Starting in Dune 3.8, Coq users will be able to use Dune even if they depend on Coq projects that use other build systems (such as make).
- Merged the PR that adds support for composition with installed theories to the Coq rules
We’re working on a new Terminal User Interface (TUI) mode for Dune. Our goal is to give Dune users an interactive GUI-like experience right from the terminal, making it easier to interact with build targets, observe processes, and more. The work is still very much in progress, but expect gradual improvements of
dune build --display tui in the coming months.
In January, we began working on allowing Dune to execute actions and run inline tests concurrently. This month, we merged the two PRs and the feature will be available in the upcoming Dune 3.8. We’re especially excited about the ability to run inline tests concurrently to speed up test cycles!
- We’ve merged the PR that implements a new concurrent action in Dune, which allows you to execute actions concurrently.
- We also merged the PR that built on this feature to enabled Dune to run inline tests concurrently.
- We’ve patched
ppx_inline_testto leverage the new feature.
Contributors: @gridbugs (Tarides), @Leonidas-from-XIV (Tarides)
In March we added continous benchmarking of Dune builds on a 48 core baremetal system. This is the result of a lot of work that included building a big monorepo for opam packages allowing users to run Dune benchmarks on large code bases.
In April we added support for benchmarking builds in watch mode as well. This allows us to monitor for regressions as we move forward with major initiatives, such as package management.
- Add watch-mode benchmarks of dune monorepo using Dune’s streaming RPC interface
The feature will be available in the upcoming Dune 3.8. You can already read the documentation in Dune’s manual to get a glimpse of how the feature will work. You can also have a look at the opam Melange template built by the Melange team.
- Make Melange work on 4.13-5.1 rather than just 4.14
- In addition to what we did in April, here are some noteworthy PRs that were worked on in previous months:
- Added a new field
melange.runtime_depsto libraries, so that Melange library authors can tell Dune which frontend assets (like CSS or image files, fonts etc) have to be installed with their library
- Speed up rule generation for librariess and executables. This was useful for Melange, but benefits every Dune project.
- Make reactjs-jsx-ppx a standalone ppx
module_systemfield for the melange stanza (e.g.
(module_systems (es6 mjs) (commonjs js) (commonjs cjs))which allows to output multiple module systems / js extensions at the same time to the melange output folder
- Added a new field
We’re working on generating a search index and adding a search bar to
odoc-generated documentation. We’re still exploring the different approaches and we are working with the OCaml.org team to implement a search bar on the OCaml packages documentation.
- We started prototyping the search index and search bar and we’re discussing the design of it. In particular, we’ve used the compiler Shapes in the prototype and we’ll explore using odoc’s path resolver instead as a next step.
We’re working towards making generating documentation in Dune more accessible, especially for newcomers. Currently, the
dune build @doc command generates documentation in the
_build directory, and users simply have to know that they need to
open _build/default/_doc/_html/index.html. To work around this, we’re working on a new
dune ocaml doc command to generate documentation and open it in the browser directly.
- We opened a PR that implements the
dune ocaml doccommand in March. This month, we tested the feature on macOS. We are now working towards completing the Windows tests.
Currently, the only way to create tables with odoc is to inline HTML code in the documentation. Tis is not ideal as the HTML table syntax is not well-suited as documentation markup and the tables can only be rendered by the HTML renderer (so tables are not available in LaTex and manpages). We’re working towards a new special syntax in odoc for creating tables that is easier to use and can be rendered by all renderers.
The syntax support has been merged in odoc-parser. It provides a heavy-syntax, and a ligh-syntax inspired by Markdown:
a | b | c | d
a | b | c | d
The support for the feature in odoc isn’t merged yet, but should be available in the next odoc version!
- No new activity in March, but here are Pull Requests we have been working on until now:
Contributors: @voodoos (Tarides)
Our work on Merlin focuses on the long-standing project to add project-wide occurrences support to Merlin. In April, we continued to work on the compiler patches that allow to generate the index from the compiler, and we updated the Merlin patches to work with the compiler patches, simplifying the Merlin logic that can now rely on the compiler.
The feature requires patches for the OCaml compiler, Dune, and Merlin that are still unreleased, but if you’re courageous, you can give it a try by following the documentation on
- We backported the compiler-side-indexation to 4.14. We performed benchmarks to evaluate the impact on build time and the size of the installed
cmts. We posted the results on the PR, for both the build time and cmts size.
- We also reworked the iterator performing indexation and added most of the missing cases to the indexer. However, some elements remain unindexed due to constraints with the
- We updated the “indexing” external tool following partial indexation implementation in the compiler.
- We also ported new compiler changes to Merlin on the OCaml 5.1 preview, which will allow us to work on project-wide occurrences support.
- Finally, we started refactoring and simplifying the Merlin patches to query the index now that more work is done by the compiler.
Contributed by: @pitag-ha (Tarides)
Following reports that Merlin performance scaled poorly in large code bases, we had been working on Merl-an, a tool to measure the performance of different Merlin requests.
In March, we were able to use it to identify the major performance bottlenecks, the biggest one being the PPX phase. We implemented a caching strategy for PPX and continued to work on it throughout April.
- We worked on fixing the PPX cache and explored mechanisms to toggle the PPX phase cache. We ended up implementing a flag-based approach.
- We added six tests that cover default behavior, cache hits, cache invalidation of three kinds, and behavior in case of PPX dependencies.
Contributors: @nojb (LexiFi)
In February, we released Dune 3.7.0 with support for watch mode on Windows. Building on this work, this month we fixed a couple of issues in Dune and OCaml LSP to allow OCaml LSP to use Dune RPC. This allows VSCode users to leverage Dune RPC and get build statuses and more exhaustive build errors in the editor when Dune is running in watch mode. The fixes are not released yet, but they will be available on the upcoming releases of Dune and OCaml LSP.
- We made a patch in Dune to use the RPC protocol on Windows.
- We fixed a bug in OCaml-LSP to enable the communication with Dune RPC on Windows.
Contributors: @voodoos (Tarides), @3Rafal (Tarides)
We’re at the finish line of our efforts to close the gap between Merlin and OCaml LSP by upstreaming OCaml LSP’s fork of Merlin! This month, we continued on the Merlin PR that adds a hook to OCaml LSP letting it run system commands. We also opened a PR on
ocaml-lsp to use the above patch and remove Merlin’s fork entirely. We’re expecting to release a version of OCaml LSP that uses Merlin as a library very soon.
- We discussed and reviewed changes that let you configure the process spawn for PPXs when using Merlin as a library. This led us to implement ideas concerning the exposed hook for PPX process spawning. Subsequently, we documented the complexities of splitting a PPX command between program and arguments.
- We opened a PR on
ocaml-lspto remove Merlin’s fork and use Merlin as a library.
Back in 2022, @trefis built a prototype of a new OCaml formatter that uses a Conrete Syntax Tree (CST) instead of an Abstract Syntax Tree (AST). This mode retains more information and results in more accurate formatting in a lot of cases, most especially when formatting comments which is a big pain point with OCamlFormat.
Since then, we’ve worked on migrating OCamlFormat’s syntax tree to this CST. We chose not to migrate everything at once to minimize the impact on users as much as possible, making sure that we make formatting changes only when they are bug fixes or clear improvements.
You can track our progress in this PR, which shows a diff of the current syntax tree and the target CST.
- We made progress on the work-in-progress PR to preserve comments attached to labelled args, fixing more regressions.
- We implemented a change to keep literal char value during parsing.
- We worked on preserving the functor concrete syntax in the Parsetree.
- We normalized functions in the parser.
The OCamlFormat team has been working with the Jane Street teams to migrate Jane Street’s code base from ocp-indent to OCamlFormat.
As a result, we’ve made tons of changes to the
janestreet profile in recent months. Perhaps most notably, this work has allowed us to identify issues that were not specific to the
janestreet profile, and consequently we’ve been fixing bugs and implementing formatting improvements across the board.
We’re nearing the end of this project, but April has seen a lot of bug fixes and improvements that we detail below.
- We adjusted the indentation for several language features, including extensions, class-expr function bodies, and module-expr extensions.
- We fixed issues with ocp-indent compatibility and Let vs LetIn detection after ‘struct’.
- We improved the formatting of module arguments and worked on preserving comments attached to labelled args.
- We fixed issues causing changes to the AST due to strings in code blocks and comments.
- We improved the formatting of cinaps comments with strings and worked on parsing and normalizing cinaps comments.
- We made adjustments to the handling of certain language constructs, such as protecting match after
fun _ : _ ->and consistently wrapping ( :: ).
- We extended our test suite with numstat and a single run of ocp-indent.