Hi everyone,
I hope you’re all having a nice summer (or a nice whichever season you’re in, of course)! We’ve set up a new wiki page on the ppxlib repository containing a status overview of the current ppxlib transition, which aims at keeping the PPX ecosystem always up-to-date. We’ll keep that wiki page up-to-date, as well.
@jeremiedimino and @NathanReb have already explained our three-part plan for this transition in different posts here on discuss. Nothing has changed in that plan, but it has been a while since we last posted about the overall transition and even longer since we last posted about the Astlib transition in detail. So if you want, you can refresh your memory about that transition and get updated about its current state (in more detail than the new wiki page) by reading this post.
Which Issues ppxlib was Facing
With ocaml-migrate-parsetree (OMP), the PPX ecosystem became cross-compiler-compatible. With ppxlib, the latest compiler features were supported more easily and broadly within the PPX ecosystem, while ppxlib also brought along other improvements such as the one in performance and the clear composition semantics when using several PPXs. With that, both OMP and ppxlib have taken away several maintenance burdens from the PPX maintainers and have created a more homogeneous and up-to-date PPX ecosystem. However, we were facing the following issues:
- To keep the PPX ecosystem cross-compiler compatible
a)ppxlibwas handling parts of the unstablecompiler-libsAPI to abstracting them away;
b) theOMP/ppxlibmaintainers needed to keep the AST migration information up-to-date by coordination with the compiler devs. - To guarantee new feature support,
ppxlibneeded to bump theppxlibAST to the newest version. - Bumping the AST implies a breaking change. That was an issue for a homogeneous and up-to-date PPX ecosystem.
- Not all PPXs migrated from
OMPtoppxlib. That was also an issue for a homogeneous and up-to-date PPX ecosystem.
Some time ago, there was the very ambitious plan of tackling Issues 1, 2, and 3 all at once by writing a stable AST abstraction and upstreaming it to the compiler. That plan has been put on ice for now. Instead we’re currently on track with a more down-to-earth plan, outlined below.
Tackling the Issues in Three Parts
The plan we’re currently following contains three simultaneous parts. It approaches three of the four issues I’ve pointed out above. However, it leaves the need to bump the AST (Issue 2) untouched.
Part One: Astlib as an Interface between ppxlib and the Compiler
The first part works towards continuous cross-compiler compatibility (Issue 1 above) while making the situation of still having PPXs based on OMP (Issue 4 above) even more of a problem. It consists of implementing an interface module called Astlib between ppxlib and the compiler, then upstreaming it to the compiler. As long as Astlib is stable and up-to-date, the rest of ppxlib won’t be affected by any compiler changes—neither by new AST versions nor by compiler library changes.
The first step of this part of the plan was moving the OMP driver and other OMP features from OMP to ppxlib. That was done in August 2020, and it introduced OMP2. Since the PPX driver has to be unique, this was the start of having the PPX ecosystem split into the two incompatible worlds of OMP1 PPXs on one hand and ppxlib PPXs on the other hand.
By now, we have written Astlib as an internal ppxlib library and have reduced ppxlib's compiler library usage as much as possible to keep Astlib minimal. As you can see, it contains a minimal compiler library sub-API in addition to the former OMP modules of our supported ASTs and the migration information between them. We will upstream Astlib to the compiler asking for it to be kept stable and up-to-date, while also keeping our local copy for old compiler support.
Part Two: Sending Patch PRs when Bumping the AST
So, thanks to Part One of the plan, ppxlib will always be compatible with the development compiler OCaml trunk and the newest compiler version. However, to also support the newest compiler features, we need to bump the internal ppxlib AST to the newest version. That modifies some of the AST nodes and so it breaks any PPX that rewrites one of those nodes (Issue 3 above). Usually just a handful of PPXs are affected, but we still want them to be up-to-date.
Our current plan doesn’t provide a solution for that problem, but it does make handling the problem more efficient and, once again, it takes away the burden from the PPX maintainers. Since the AST bump to 4.10, whenever we bump the AST, we send patch PRs to the PPXs we break. Not much has changed since February, when @NathanReb last explained our workflow of sending patch PRs in detail. To some it up: we create a workspace with all ppxlib reverse dependencies on opam fulfilling a certain standard, which we call the ppx-universe. We then fix the PPXs that break all at once and open the PRs.
Lately, the ppx-universe has also proven very useful to make well-founded decisions regarding our API by having an easy look at our reverse dependencies. You can find a ppx-universe snapshot, currently from March, on our repo.
In our experience, once the ppx-universe is created and “builds up to the expected breakages,” writing a couple of patches takes very little time, so we plan to make the tooling that creates and interacts with the workspace more sophisticated.
Part Three: Porting PPXs to Put an End to the “Split-World Situation”
As explained above, Part One split the PPXs into the two incompatible worlds of OMP1 PPXs on one hand and ppxlib PPXs on the other hand. That made the fact that some PPXs were still based on OMP (Issue 4 above) even more of a problem. For some PPX maintainers, the reason to avoid porting their PPXs to ppxlib was that ppxlib depended on base and stdio, so we decided to tackle this situation by three means:
- Dropping the
baseand thestdiodependencies, which was done in August last year. Now, all dependencies are the very basicocaml,dune,ocaml-compiler-libs,stdlib-shims,sexplib0andppx_derivers. - Porting and reviewing some of the most important PPXs ourselves. So far we’ve ported
js_of_ocaml,bisect_ppx, andtyxmlwith the help of the respective maintainers, and we’ve also reviewed several ports. - Spreading the word about the need to port PPXs and asking for help.
About a year ago, we made a non-exhaustive list of PPXs that needed to be ported. Since then, this community has proven to be awesome and there has been an amazing porting effort by a lot of people. So by now, all packages on that list have been ported with the exception of one(*). So hopefully the “split world” situation can soon be considered past. 
By the way, thanks to all involved in porting PPXs to ppxlib! It has been a great joint effort so far. 
And if anyone still has or comes across a project somewhere that needs porting and wants to port it, that’s awesome!
You can find the full list of opam packages that are still stuck in the OMP1 world by filtering for them in opam’s health check pipeline. However, notice that that’s a generated list, so it also contains libraries that intrinsically form part of the OMP1 ecosystem (such as ppx_tools_versioned), PPXs that have already been ported but haven’t relesed their port on opam yet (such as graphql_ppx), deprecated PPXs that aren’t marked as deprecated yet (such as mirage-dns), and several PPXs that only transitively depend on OMP1.
(*) ppx_import has a PR for a port to ppxlib, but it’s not quite ready to be merged just yet.
ppx is now bridging this nicely with as OMP is being phased out of the dependency graphs of popular ppxes. I’m also really glad to see your use of bulk opam package builds to help drive this process, which makes all the effort that goes into the opam metadata maintenance and the health-check service feel much more worthwhile!