Deprecating ocaml-migrate-parsetree in favor of Ppxlib, also as a Platform tool

Hello everyone :wave:

We’re planning to deprecate ocaml-migrate-parsetree (OMP) finally. OMP forms part of the OCaml Platform, so we can’t and wouldn’t just do that without the approval of the community. So first a bit of context:

What is OMP anyways?

OMP used to be an extremely important project in the meta-programming-related chunk of OCaml’s ecosystem. Since the introduction of extension nodes and attributes in OCaml 4.2 in 2014, meta-programming in OCaml is most commonly done on OCaml’s parsetree. The caveat: The parsetree is extended and modified between OCaml minor versions, meaning that handling the OCaml parsetree directly is unstable. OMP came to the rescue by introducing a concept we call versioned parsetrees (or versioned ASTs) together with migrations between those fixed versions of the parsetree, allowing the workflow

That workflow allows one PPX to guarantee compatibility with multiple compiler versions (here, PPX = PreProcessor eXtension stands for the meta-programming instance). That was extremely important at the time to introduce a more version-flexible ecosystem and to reduce the maintenance burden of PPXs!

OMP also provided the concept of a driver, which is one single binary driving the applicatoin of all PPXs. Without that, using n PPXs would mean copying the workflow pictured above n times. That’s an incredible performance overhead. The OMP driver took care of applying all PPXs, so fewer parsetree migrations and less communication between different processes were necessary (before: one process per PPX).

Why was OMP not enough?

However, the OMP driver was still duplicating parsetree migrations, and it did one whole parsetree pass per PPX. And because of the latter, the outcome of the preprocessor phase used to depend on the order of PPXs, which didn’t follow any semantically relevant or otherwise reasoned pattern. The reason why it would have been extremely complicated to improve the OMP driver further was that each PPX chose its own versioned parsetree version. There was no agreement between different PPXs.

Ppxlib

So the solution was to have the PPXs agree on one versioned parsetree version. That’s what Ppxlib does. It also does a lot more by providing an extensive API to write PPXs easily and others, but the main point here is that it consolidates a consistent PPX ecosystem wrt parsetree versions. The way it does so is by exposing one fixed versioned parsetree version that all PPXs are defined against.

Thanks to that, the Ppxlib driver can get rid of unnecessary parsetree migrations, and it can merge a whole bunch of PPXs into one AST pass, improving performance and guaranteeing a clear order of PPX transformations. Also important: Having all PPXs on the same versioned parsetree version, makes it easy to have them all on the one of the latest OCaml version. For details I’m not going into, that’s the only (reasonable) way to have the PPXs support the latest compiler syntax features, so that’s a huge advantage of Ppxlib as well.

Ppxlib still isn’t perfect and comes with its own little set of problems, but it does solve the mentioned big problems of performance overhead, unclear composition semantics, parsetree-version-fragmentation among different PPXs, and lack of latest compiler syntax support. So by now, the whole OCaml opam ecosystem has shifted from OMP to Ppxlib (more below).

The latest update on the state of Ppxlib, also including info about OMP and the ecosystem’s transition from one to the other is this one from 2021. All Ppxlib maintainers have very very little time for communication and similar. If a new update would be strongly appreciated, let us know. For here, let’s focus on OMP:

What does it mean to deprecate OMP?

Given that by now, OMP’s maintenance is held at the bare bare minimum anyways, deprecating it would only have one clear impact:

Real-life

We’d stop adding new compiler support, so the last supported compiler would be OCaml 5.0. Concretely: We won’t add the new versioned parsetrees and migrations for the new compilers anymore.

Formalities

We’d also mark the OMP opam packages as deprecated, and we’d move OMP to the Deprecate section in the set of OCaml Platform tools (now it’s in the Sustained section).

Is it ok to deprecate OMP?

You tell us :slight_smile: Here’s some info:

Opam packages

All packages on opam have moved to Ppxlib now. There’s only a hand-full of packages that formally still depend on OMP, but all of that seems to be formal left-overs in the opam files, a left-over test dependency or similar. I’ve opened issues on those packages, mentioning the potential upcoming deprecation of OMP, and nobody has complained.

Distribution packages

There are still a few distributions that keep on packaging OMP on their distribution, e.g. there’s OMP on Debian, OMP on Fedora and OMP on Arch. However, the reason for that seems merely historical (with unquestioned inertia): There are no reverse dependencies on Debian or Fedora. On Arch, there are a few reverse dependencies, none of them still being relevant in the present. I’ll reach out to the three respective package managers. If anyone has any info or relevant context about this, please share it!

Users/community

And this is where we’re asking for feedback. Does it sound good to everyone to deprecate OMP? We do have the strong impression that pretty much everyone has moved to Ppxlib, but if you think we’re missing something important, please let us know (or, just use this opportunity to say something nice about OMP :slight_smile: ).

Original author and current maintainer

Obviously, the current maintainer/“sustainer” (me) thinks it’s good to deprecate OMP finally. And the original author, @let-def, is behind it as well.

Btw, @let-def, thanks a lot for this crucial piece of OCaml software! It contains a lot of really good ideas, concepts, and implementation details! Even once deprecated, it will persist - partly quite literally inside Ppxlib in the form of the versioned parsetrees, the migrations, and parts of the driver.

17 Likes

Hello again,

As a last heads-up: If nobody objects, we’ll deprecate OMP next week.

3 Likes

OMP is now deprecated :white_check_mark: Transitioning the whole OCaml ecosystem from OMP to Ppxlib has been a long journey, which is now completed.

The deprecation means that OMP will be incompatible with any new compiler version. The first incompatible compiler version is OCaml 5.1.

You can see the deprecation reflected in OMP’s opam packages, in notes in OMP’s README and repo description, and in its life cycle in the OCaml Platform. There’s also an OCaml Changelog entry about the deprecation.

Thanks a lot to everyone involved in OCaml’s transition from OMP to Ppxlib, for example by porting their PPX!

4 Likes