4.06 release work: a partial progress report

Ecosystem
1

As many of you know, the maintainers of the OCaml compiler distribution are hard at work preparing the next release of the language, 4.06.0 – the dream is to have a release by the end of October. The release effort, however, is not limited to the compiler distribution : there is a lot of work going behind the scenes on updating the library ecosystem to changes in the new version. I’m creating this topic to give a (partial) view of what’s happening, so that people have a better idea of what’s the work involved – and maybe they want to help!

(For a list of changes in 4.06, see the Changes file of the release branch.)

Compiler distribution

On the “public” side of the compiler distrubtion front, we have done two beta releases of the distribution and @octachron released a beta version of the manual:

On the development side (also public) of the compiler distribution, three activities are happening thanks to maintainers and external contributors alike:

  • We are busy reviewing and discussing the pull requests that we are considering for inclusion in 4.06 (see the 4.06.0 milestone on github); there are 10 PRs still open in this category. Note that being marked for 4.06 does not mean “we know we want it in the release”, but “we have to decide what to do with it before the release”. Since the feature freeze mid-September, we are mostly working on bugfixes – fixes for regressions in 4.06 or to features that are appearing in 4.06 are high-priority.

  • We are also reviewing the bug reports on Mantis that are marked for consideration before the 4.06 release (see the Roadmap). There are currently 8 bugs open. Some of these bugs are reported as 4.06-regression (and thus high priority), but many of those bugs have been around for longer. The current mantis process is to encourage ourselves to review and re-triage, before each release, as many of the open bugs as possible, to motivate maintainers/contributors to fix those that seem easy (release work is also a time where some people are more motivated to work on the OCaml codebase, so fixing old bugs can be a useful way to channel this energy).

In theory, we could do a release as soon as these two categories have been emptied (by merging/fixing or deciding to postpone), except for what comes next.

The opam repository

There is a lot of work going on in the opam repository since the beta started, aimed at making sure that the OCaml open-source ecosystem is as ready for 4.06 as possible. This works tends to be even more distributed than compiler-distribution contributions (which are already fairly diverse), and I’m personally very happy with the effort that many people have been pouring into it.

A general goal is that when they switch to a new release, end-users that do opam install foo should not see a build failure; the package should either build correctly or be marked as non-installable. This means that we need to:

  1. fix the packaging metadata of packages that fail to build to indicate that they are incompatible with the release
  2. get new version released (or sometimes patch an existing version) that are compatible with the new OCaml release

Our goal for the OCaml language is to have strong backward-compatibility from one language release to another, so in theory the work required to update end-users code should be small. However:

  • Some packages at the base of the ecosystem, in particular almost all of the ppx universe, are tightly coupled to the compiler distribution, so they almost always need an update (and them being broken prevents a lot of packages from building correctly : they are deep down the dependency chain of almost everyone). I think that sexplib is not yet 4.06.0-ready for example (or that change in the last couple days), and this has a huge impact on the availability of other packages.
  • this release is a bit special is that the -safe-string option which was introduced in 4.03 but optional so far is now the default. This means that unless the build scripts are changed to explicitly use the -unsafe-string option, all packages that mutate strings (rather than bytes) have to be fixed. Thankfully, most of the time the fixes are fairly easy to make (use Bytes instead of String for mutating operations, and Bytes.{of,to}_string to convert in and out), so updating codebases is not a lot of work.

Some package authors are checking their own package and fixing it, and that is very nice. Some contributors are more active than that: they actively help updating the dependencies of their packages (or just packages they care about). Sending an opam-repository PR to fix metadata, or sending an upstream patch to fix a package are super-helpful, but a simpler thing that already helps is to try to compile packages, and to report the breakage to maintainers of the dependencies that are causing the build failure. (Maintainers can work in a demand-driven way, they fix stuff when people complain.)

For example, I have personally interacted with @jpdeplaix who pinged a couple packages I follow, and fixed a bunch of opam metadata; his work will have a very nice impact on the quality of the final release. Many other people are contributing (and it’s hard to have a global view of the work involved without subscribing to the development repository of all OCaml packages), see for example the opam-repository contributors this month.

I have been monitoring the buildability of the whole opam-repository ecosystem using a personal instance of @lefessan’s opam-builder tool (the temporary output is there; I dislike the new interface but I don’t have the time to improve it right now), and Damien Doligez uses his own opamcheck tool to systematically build opam packages in a different way (we talked, and we may try to join forces after the release). I hoped to obtain a good/clear enough output that I could encourage people to look at it to find packages to fix, but I haven’t had the time for this release. In any case I can tell you that the global health of the 4.06 packages has been improving sharply this month, thanks to a couple regression fixes on the compiler-distribution side and a lot of work to fix/update the packages from everyone.

Thanks!

14 Likes
2

P.S.: I should have mentioned it explicitly above, but if you have questions or comments about this particular release (the compiler distribution or some packages), or the release process in general, here would be a perfect place to send them :slight_smile:

1 Like