OCaml Platform Newsletter: April 2023

Welcome to the first issue of the OCaml Platform newsletter!

Following in the footsteps of the OCaml.org newsletter and inspired by the Multicore and Compiler newsletters, we’re excited to bring you monthly updates on the progress we’re making in improving the OCaml Developer Experience.

The OCaml Platform is the recommended set of tools to work and be productive with OCaml. The Platform tools fill gaps in the OCaml developer experience and allow developers to perform specific workflows (e.g. building packages, formatting code, generating documentation, releasing packages, etc.).

At the end of the day, all the work we’re doing on the OCaml Platform has one objective: improving the OCaml developer experience, so in this newsletter, we’ll present the progress we’re making on the different projects from the lens of these developer workflows. Based on the results of the OCaml surveys (2020 and 2022), discussions with industrial users, continuous community feedback on Discuss, and other sources of user input, here are the workflows we’re currently working on:

  • Building Packages: Our immediate goal for the build workflow is to remove the friction associated to using two different tools for package management and build system. To this end, we plan on integrating opam capabilities directly into Dune, establishing it as the singular tool needed to build OCaml projects. As a by-product of this integration, we aim to improve other workflows such as working on multiple projects, cross-compilation, and improving the overall experience to get started with OCaml.
  • Compiling to JavaScript: We’re continuously supporting tools to compile OCaml to JavaScript. Dune integrates well with Js_of_ocaml and we’ve been working on an integration of Dune and Melange, a recent fork of ReScript that aims to bring to integrate closely with the OCaml ecosystem.
  • Generating Documentation: The state of the OCaml Packages documentation is reported as a major pain point in the OCaml surveys (2020 and 2022). We’re working towards empowering OCaml developers to create high-quality documentation for their packages. Now that the documentation of packages is readily available on OCaml.org, we want to make writing documentation rewarding and straightforward. We’re working towards making Odoc suitable to create manuals by adding new features, improving the navigation, and expanding the odoc markup syntax to support rich content such as tables, images and graphs.
  • Editing and Refactoring Code: We aim to enrich the OCaml editor support with more workflows to improve code navigation and automate refactoring. Our main focus currently is on adding support for project-wide references to Merlin. Future work will include implementing a project-wide rename query and queries such as renaming arguments. We are also working towards bringing the editor support for OCaml to the web and third-party platforms such as OCaml Playground, ReplIt, and GitHub Codespaces.
  • Formatting Code: Our goal for formatting code is focused on improving accuracy, particularly for the to comments. We also want to strike the right balance between providing a default profile that appeals to most users and not requiring configuration to format your OCaml projects, while still maintaining a fully configurable formatter. Additionally, we plan to enhance the backward compatibility of ocamlformat and better integrate Dune and ocamlformat.

I’ll also take the opportunity to call for new contributors. Platform projects are always looking for new maintainers and contributors, so if you’re interested in the future of OCaml’s developer experience and would like to shape that future with us, please reach out to me or any Platform maintainers. If you’re an industrial user looking for support on the OCaml Platform and would like to fund the maintainers and the developments on the Platform tools, also don’t hesitate to reach out to me.

April has seen a flurry of activity, and we can’t wait to share our progress with you. So let’s get right to it!

In this inaugural issue, we’ll be discussing progress on the following projects:

Releases

Here are the new versions of Platform tools we released in April. Have a look at the OCaml Changelog to read announcements and feature highlights!

Building Packages

[Dune] Exploring Package Management in Dune

Contributors: @rgrinberg (Tarides), @Leonidas-from-XIV (Tarides), @gridbugs (Tarides), @kit-ty-kate (Tarides)

Earlier this month, we announced that we’ve started exploring package management in Dune. You can read the Request for Comment (RFC) that details our work-in-progress plans for the feature on GitHub.

We’re currently focused on building prototypes for different parts of the Dune workflow: source fetching, building non-Dune opam packages, and generating a lock file.

In April, we merged a first version of Source Fetching. We also started thinking about how Dune could build opam packages and merged a PR laying the foundation for the rules on building them in Dune.

Activities:

[opam] Native Support for Windows in opam 2.2

Contributors: @rjbou (OCamlPro), @kit-ty-kate (Tarides), @dra27 (Tarides), @emillon (Tarides), @Leonidas-from-XIV (Tarides)

Bringing Tier-1 support for Windows has been a dream for some time and April has seen us get closer than ever before to the first alpha of opam 2.2, which we expect in May. This early alpha is a big step towards the release of opam 2.2 with native Windows support, and is the result of a humongous amount of effort bringing together the work of many people done over the years.

Activities:

[Dune] Improving Dune’s Documentation

Contributors: @emillon (Tarides)

In March, we started restructuring the Dune documentation according to the Diataxis framework. We opened a draft PR to demonstrate the overall target structure. The new structure will improve the usability of the documentation, allowing users to find the information that they are looking for, as well as enable us to better identify gaps that need to be addressed.

In April we’ve continued this work, filling in some of those gaps as well as rewriting documents to better fit in the intended quadrant of the framework.

Activities:

[Dune] Composing installed Coq theories

Contributors: @Alizter and @ejgallego (IRIF)

We’ve merged the PR that brings support for composing Coq theories with Dune!

This was a huge effort lead by Ali Caglayan and Emilio Jesús Gallego Arias that started earlier this year. Starting in Dune 3.8, Coq users will be able to use Dune even if they depend on Coq projects that use other build systems (such as make).

Activities:

[Dune] Dune Terminal User Interface

Contributors: @Alizter, @rgrinberg (Tarides)

We’re working on a new Terminal User Interface (TUI) mode for Dune. Our goal is to give Dune users an interactive GUI-like experience right from the terminal, making it easier to interact with build targets, observe processes, and more. The work is still very much in progress, but expect gradual improvements of dune build --display tui in the coming months.

Activities:

[Dune] Running Actions Concurrently

Contributors: @Alizter and @hhugo (Nomadic Labs)

In January, we began working on allowing Dune to execute actions and run inline tests concurrently. This month, we merged the two PRs and the feature will be available in the upcoming Dune 3.8. We’re especially excited about the ability to run inline tests concurrently to speed up test cycles!

Activities:

[Dune] Benchmarking Dune on Large Code Bases

Contributors: @gridbugs (Tarides), @Leonidas-from-XIV (Tarides)

In March we added continous benchmarking of Dune builds on a 48 core baremetal system. This is the result of a lot of work that included building a big monorepo for opam packages allowing users to run Dune benchmarks on large code bases.

In April we added support for benchmarking builds in watch mode as well. This allows us to monitor for regressions as we move forward with major initiatives, such as package management.

Activities:

Compiling to JavaScript

[Dune] Compile to JavaScript with Melange in Dune

Contributors: @anmonteiro, @jchavarri (Ahrefs), @rgrinberg (Tarides)

You may have read that Ahrefs migrated its codebase from ReScript to Melange, a new OCaml-to-JavaScript compiler based on ReScript.

The goal of Melange is to offer an alternative to ReScript that pairs well with the OCaml ecosystem. To that end, Antonio Nuno Monteiro and Javier Chávarri have been working on integrating Melange and Dune, allowing it to easily compile OCaml projects to JavaScript with Melange.

The feature will be available in the upcoming Dune 3.8. You can already read the documentation in Dune’s manual to get a glimpse of how the feature will work. You can also have a look at the opam Melange template built by the Melange team.

Activities

  • Write a manual page in Dune to compile to JavaScript using Melange.
  • Make Melange work on 4.13-5.1 rather than just 4.14
  • In addition to what we did in April, here are some noteworthy PRs that were worked on in previous months:

Generating Documentation

[Odoc] Add Search Capabilities to odoc

Contributors: @panglesd (Tarides), @EmileTrotignon (Tarides)

We’re working on generating a search index and adding a search bar to odoc-generated documentation. We’re still exploring the different approaches and we are working with the OCaml.org team to implement a search bar on the OCaml packages documentation.

Activities:

  • We started prototyping the search index and search bar and we’re discussing the design of it. In particular, we’ve used the compiler Shapes in the prototype and we’ll explore using odoc’s path resolver instead as a next step.

[Dune] User-Friendly Command to Generate Documentation

Contributor: @EmileTrotignon (Tarides), @jonludlam (Tarides)

We’re working towards making generating documentation in Dune more accessible, especially for newcomers. Currently, the dune build @doc command generates documentation in the _build directory, and users simply have to know that they need to open _build/default/_doc/_html/index.html. To work around this, we’re working on a new dune ocaml doc command to generate documentation and open it in the browser directly.

Activities:

  • We opened a PR that implements the dune ocaml doc command in March. This month, we tested the feature on macOS. We are now working towards completing the Windows tests.

[Odoc] Support for Tables in odoc

Contributors: @gpetiot (Tarides), @panglesd (Tarides), @Julow (Tarides), @jonludlam (Tarides)

Currently, the only way to create tables with odoc is to inline HTML code in the documentation. Tis is not ideal as the HTML table syntax is not well-suited as documentation markup and the tables can only be rendered by the HTML renderer (so tables are not available in LaTex and manpages). We’re working towards a new special syntax in odoc for creating tables that is easier to use and can be rendered by all renderers.

The syntax support has been merged in odoc-parser. It provides a heavy-syntax, and a ligh-syntax inspired by Markdown:

{t
  a  | b | c | d
  ---|:--|--:|:-:
  a  | b | c | d
}

The support for the feature in odoc isn’t merged yet, but should be available in the next odoc version!

Activities

  • No new activity in March, but here are Pull Requests we have been working on until now:

Editing and Refactoring Code

[Merlin] Support for Project-Wide References in Merlin

Contributors: @voodoos (Tarides)

Our work on Merlin focuses on the long-standing project to add project-wide occurrences support to Merlin. In April, we continued to work on the compiler patches that allow to generate the index from the compiler, and we updated the Merlin patches to work with the compiler patches, simplifying the Merlin logic that can now rely on the compiler.

The feature requires patches for the OCaml compiler, Dune, and Merlin that are still unreleased, but if you’re courageous, you can give it a try by following the documentation on voodoos/merlin-occurrences-switch.

Activities

  • We backported the compiler-side-indexation to 4.14. We performed benchmarks to evaluate the impact on build time and the size of the installed cmts. We posted the results on the PR, for both the build time and cmts size.
  • We also reworked the iterator performing indexation and added most of the missing cases to the indexer. However, some elements remain unindexed due to constraints with the Typedtree.
  • We updated the “indexing” external tool following partial indexation implementation in the compiler.
  • We also ported new compiler changes to Merlin on the OCaml 5.1 preview, which will allow us to work on project-wide occurrences support.
  • Finally, we started refactoring and simplifying the Merlin patches to query the index now that more work is done by the compiler.

[Merlin] Improving Merlin’s Performance

Contributed by: @pitag-ha (Tarides)

Following reports that Merlin performance scaled poorly in large code bases, we had been working on Merl-an, a tool to measure the performance of different Merlin requests.

In March, we were able to use it to identify the major performance bottlenecks, the biggest one being the PPX phase. We implemented a caching strategy for PPX and continued to work on it throughout April.

Activities:

  • We worked on fixing the PPX cache and explored mechanisms to toggle the PPX phase cache. We ended up implementing a flag-based approach.
  • We added six tests that cover default behavior, cache hits, cache invalidation of three kinds, and behavior in case of PPX dependencies.

[OCaml LSP] Using Dune RPC on Windows

Contributors: @nojb (LexiFi)

In February, we released Dune 3.7.0 with support for watch mode on Windows. Building on this work, this month we fixed a couple of issues in Dune and OCaml LSP to allow OCaml LSP to use Dune RPC. This allows VSCode users to leverage Dune RPC and get build statuses and more exhaustive build errors in the editor when Dune is running in watch mode. The fixes are not released yet, but they will be available on the upcoming releases of Dune and OCaml LSP.

Activities:

  • We made a patch in Dune to use the RPC protocol on Windows.
  • We fixed a bug in OCaml-LSP to enable the communication with Dune RPC on Windows.

[OCaml LSP] Upstreaming OCaml LSP’s Fork of Merlin

Contributors: @voodoos (Tarides), @3Rafal (Tarides)

We’re at the finish line of our efforts to close the gap between Merlin and OCaml LSP by upstreaming OCaml LSP’s fork of Merlin! This month, we continued on the Merlin PR that adds a hook to OCaml LSP letting it run system commands. We also opened a PR on ocaml-lsp to use the above patch and remove Merlin’s fork entirely. We’re expecting to release a version of OCaml LSP that uses Merlin as a library very soon.

Activities:

  • We discussed and reviewed changes that let you configure the process spawn for PPXs when using Merlin as a library. This led us to implement ideas concerning the exposed hook for PPX process spawning. Subsequently, we documented the complexities of splitting a PPX command between program and arguments.
  • We opened a PR on ocaml-lsp to remove Merlin’s fork and use Merlin as a library.

Formatting Code

[OCamlFormat] Migrate OCamlFormat from an AST to a CST

Contributors: @gpetiot (Tarides), @EmileTrotignon (Tarides)

Back in 2022, @trefis built a prototype of a new OCaml formatter that uses a Conrete Syntax Tree (CST) instead of an Abstract Syntax Tree (AST). This mode retains more information and results in more accurate formatting in a lot of cases, most especially when formatting comments which is a big pain point with OCamlFormat.

Since then, we’ve worked on migrating OCamlFormat’s syntax tree to this CST. We chose not to migrate everything at once to minimize the impact on users as much as possible, making sure that we make formatting changes only when they are bug fixes or clear improvements.

You can track our progress in this PR, which shows a diff of the current syntax tree and the target CST.

Activities:

[OCamlFormat] Closing the Gap Between OCamlFormat and ocp-indent

Contributors: @gpetiot (Tarides), @EmileTrotignon (Tarides), @Julow (Tarides)

The OCamlFormat team has been working with the Jane Street teams to migrate Jane Street’s code base from ocp-indent to OCamlFormat.
As a result, we’ve made tons of changes to the janestreet profile in recent months. Perhaps most notably, this work has allowed us to identify issues that were not specific to the janestreet profile, and consequently we’ve been fixing bugs and implementing formatting improvements across the board.

We’re nearing the end of this project, but April has seen a lot of bug fixes and improvements that we detail below.

Activities:

31 Likes

Formatting code

As someone newly learning the language and with declining vision, is there any efforts to make any of the formatters more accessible with tab support (as opposed spaces)? The compiler supports it, but formatters don’t. I find the wide adoption of 2-space indentation very difficult to read, but I’d also prefer not to push my preference for readability on others. Current I’m wrapping ocamlformat and piping through GNU indent/unindent, but it definitely feels like a hack.

3 Likes

Think you can set max-indent to 4 if you want more space indentation. Not tabs but it’s basically the same effect visually.

Haven’t used it myself.

Unfortunately, using hard tabs in your code has precisely the effect of pushing a preference on your collaborators: namely, the preference of using tabs and not spaces, because mixing the two just doesn’t work at all. If you really want to be nice in addition to making the tools work for yourself (and I know this can be a lot to ask), you need to lobby for the support in tools of configurable amounts of space indentation even if they already support tabs.

Wonderful work! Given our recent discussion here, once dune can download stuff, would it make sense to standardize certain ppxs so that things like [@@deriving show] work out of the box?

1 Like

I think you misunderstood the ask. This is not about forcing everyone else to use tabs but about allowing ocamlformat to format using tabs in our private copies of projects. This does not necessarily mean we would be committing the source code with tabs into the repo and force everyone else to use tabs. We could just use e.g. git filters to reformat the code using spaces before committing and pushing. External collaborators need never know that we are privately using tabs.

This seems like a reasonable configuration to add! Do you want to open an issue at GitHub - ocaml-ppx/ocamlformat: Auto-formatter for OCaml code for it?

More generally, an interesting idea in what you’re suggesting is to explore how we can better support having a local configuration that differs from the project’s upstream configuration, without having to push them. As you point out, doing this isn’t easy at the moment. In your case, it would be useful for accessibility, but I imagine people would appreciate having their own formatting configuration that differs from the project’s upstream configuration.

Exciting stuff!

Does this imply renaming a symbol across modules become de facto available?

No, project-wide rename is also on the radar, but the work on project-wide references is for finding references and navigating to them, not refactoring.

There is an open issue. I’m not trying to stir the pot on the tabs vs. spaces debate (like Prettier has been debating), I just want code that’s easier for me to read (not just my code, but others’ too). I brought it up because I saw the general effort going into accessibility in some form or other with the platform initiatives listed here.

Philosophically & in a perfect world, it shouldn’t matter & the code forge just stores the AST rather than plaintext representation where locally/online anyone could use their preferred config.


If using Git, one does have the capability to use hooks to accomplish their preferred settings (within what the formatter supports), but the formatter would need to allow a local config override flag like

$ git config filter.tabspace.smudge 'ocamlformat --override-config $XDG_CONFIG_HOME/.ocamlformat'
$ git config filter.tabspace.clean 'ocamlformat'

ocamlformat can look outside the project for configs, or ignore all configs, but I did not see a way to substitute all settings with a different config file.

Good to know thanks, can’t wait for that one :slight_smile:

2 Likes

You’re right, I did misunderstand. I think having a local formatting option like that would be wonderful.

1 Like

If the forge stores just the AST, how could it present diffs?

This is unfortunately yet another case where a pretty reasonable (IMO) ocamlformat option is waved off on principle.

I’ve been experimenting with topiary as a highly-configurable alternative formatter. It’s recently become opam-installable, so that’s a big barrier removed towards its wider use. Eventually, having a library of “recommended” configs for OCaml usage would make topiary an outstanding option.

2 Likes

This is unfortunately yet another case where a pretty reasonable (IMO) ocamlformat option is waved off on principle.

For a period of time, we intended to trim down the number of configuration options in ocamlformat. We released a version of ocamlformat (it was 0.21.0 I think) where we deprecated tons of issues, and users complained loudly. Since then we rolled back on that plan and while we’re still aiming for a configuration-free default, it’s clear that ocamlformat users want to keep the current level of flexibility.

We should re-open the issue, it was closed when we were aggressively pushing for reducing the number of options.

We have a post we never got to share laying around that clarifies the ocamlformat team’s position on configurations, with a roadmap to reach version 1.0. I’ll try to post it soon.

5 Likes

This is out of scope here, but I still couldn’t resist
Diffing ASTs is an active field of research, see e.g. this or this, or this or many other.
It is difficult, but definitely not impossible. Diffs are just not presented in terms of raw text but in terms of modifications in a tree, and simple way of presenting means pretty-printing both threes and showing the diff at that level (although you can get more clever since you have more fine-grained info)

2 Likes

Thanks for this very useful work!

From an end-user perspective which, I guess, fits the Ocaml Platform goals (in particular for newcomers, as I teach OCaml to postgraduates), it would really be nice if the step of manually creating an empty .ocamlformat file could be get rid of, one way or another.

More generally, I’m linking to some suggestions I had made based on my teaching experience here and there. Hope they’ll be considered and possibly addressed (and if not, it would be great to give a the rationale, to keep track of the answers).

1 Like

I agree! I actually open an issue for this here: Enable formatting projects without an `.ocamlformat` by default · Issue #2092 · ocaml-ppx/ocamlformat · GitHub.

I suggest you write your feedback on the issue as well, it makes it easier for the maintainers to keep track of specific feature requests. From previous conversations, I also think a PR to change the default behaviour of ocamlformat to not require a config file would be welcome (worth asking on the issue though).

2 Likes