I hope you’re all having a nice summer (or a nice whichever season you’re in, of course)! We’ve set up a new wiki page on the ppxlib repository containing a status overview of the current
ppxlib transition, which aims at keeping the PPX ecosystem always up-to-date. We’ll keep that wiki page up-to-date, as well.
@jeremiedimino and @NathanReb have already explained our three-part plan for this transition in different posts here on discuss. Nothing has changed in that plan, but it has been a while since we last posted about the overall transition and even longer since we last posted about the
Astlib transition in detail. So if you want, you can refresh your memory about that transition and get updated about its current state (in more detail than the new wiki page) by reading this post.
OMP), the PPX ecosystem became cross-compiler-compatible. With
ppxlib, the latest compiler features were supported more easily and broadly within the PPX ecosystem, while
ppxlib also brought along other improvements such as the one in performance and the clear composition semantics when using several PPXs. With that, both
ppxlib have taken away several maintenance burdens from the PPX maintainers and have created a more homogeneous and up-to-date PPX ecosystem. However, we were facing the following issues:
- To keep the PPX ecosystem cross-compiler compatible
ppxlibwas handling parts of the unstable
compiler-libsAPI to abstracting them away;
ppxlibmaintainers needed to keep the AST migration information up-to-date by coordination with the compiler devs.
- To guarantee new feature support,
ppxlibneeded to bump the
ppxlibAST to the newest version.
- Bumping the AST implies a breaking change. That was an issue for a homogeneous and up-to-date PPX ecosystem.
- Not all PPXs migrated from
ppxlib. That was also an issue for a homogeneous and up-to-date PPX ecosystem.
Some time ago, there was the very ambitious plan of tackling Issues 1, 2, and 3 all at once by writing a stable AST abstraction and upstreaming it to the compiler. That plan has been put on ice for now. Instead we’re currently on track with a more down-to-earth plan, outlined below.
The plan we’re currently following contains three simultaneous parts. It approaches three of the four issues I’ve pointed out above. However, it leaves the need to bump the AST (Issue 2) untouched.
The first part works towards continuous cross-compiler compatibility (Issue 1 above) while making the situation of still having PPXs based on
OMP (Issue 4 above) even more of a problem. It consists of implementing an interface module called
ppxlib and the compiler, then upstreaming it to the compiler. As long as
Astlib is stable and up-to-date, the rest of
ppxlib won’t be affected by any compiler changes—neither by new AST versions nor by compiler library changes.
The first step of this part of the plan was moving the
OMP driver and other
OMP features from
ppxlib. That was done in August 2020, and it introduced
OMP2. Since the PPX driver has to be unique, this was the start of having the PPX ecosystem split into the two incompatible worlds of
OMP1 PPXs on one hand and
ppxlib PPXs on the other hand.
By now, we have written
Astlib as an internal
ppxlib library and have reduced
ppxlib's compiler library usage as much as possible to keep
Astlib minimal. As you can see, it contains a minimal compiler library sub-API in addition to the former
OMP modules of our supported ASTs and the migration information between them. We will upstream
Astlib to the compiler asking for it to be kept stable and up-to-date, while also keeping our local copy for old compiler support.
So, thanks to Part One of the plan,
ppxlib will always be compatible with the development compiler OCaml trunk and the newest compiler version. However, to also support the newest compiler features, we need to bump the internal
ppxlib AST to the newest version. That modifies some of the AST nodes and so it breaks any PPX that rewrites one of those nodes (Issue 3 above). Usually just a handful of PPXs are affected, but we still want them to be up-to-date.
Our current plan doesn’t provide a solution for that problem, but it does make handling the problem more efficient and, once again, it takes away the burden from the PPX maintainers. Since the AST bump to
4.10, whenever we bump the AST, we send patch PRs to the PPXs we break. Not much has changed since February, when @NathanReb last explained our workflow of sending patch PRs in detail. To some it up: we create a workspace with all
ppxlib reverse dependencies on opam fulfilling a certain standard, which we call the ppx-universe. We then fix the PPXs that break all at once and open the PRs.
Lately, the ppx-universe has also proven very useful to make well-founded decisions regarding our API by having an easy look at our reverse dependencies. You can find a ppx-universe snapshot, currently from March, on our repo.
In our experience, once the ppx-universe is created and “builds up to the expected breakages,” writing a couple of patches takes very little time, so we plan to make the tooling that creates and interacts with the workspace more sophisticated.
As explained above, Part One split the PPXs into the two incompatible worlds of
OMP1 PPXs on one hand and
ppxlib PPXs on the other hand. That made the fact that some PPXs were still based on
OMP (Issue 4 above) even more of a problem. For some PPX maintainers, the reason to avoid porting their PPXs to
ppxlib was that
ppxlib depended on
stdio, so we decided to tackle this situation by three means:
- Dropping the
stdiodependencies, which was done in August last year. Now, all dependencies are the very basic
- Porting and reviewing some of the most important PPXs ourselves. So far we’ve ported
tyxmlwith the help of the respective maintainers, and we’ve also reviewed several ports.
- Spreading the word about the need to port PPXs and asking for help.
About a year ago, we made a non-exhaustive list of PPXs that needed to be ported. Since then, this community has proven to be awesome and there has been an amazing porting effort by a lot of people. So by now, all packages on that list have been ported with the exception of one(*). So hopefully the “split world” situation can soon be considered past.
By the way, thanks to all involved in porting PPXs to
ppxlib! It has been a great joint effort so far. And if anyone still has or comes across a project somewhere that needs porting and wants to port it, that’s awesome!
You can find the full list of opam packages that are still stuck in the
OMP1 world by filtering for them in opam’s health check pipeline. However, notice that that’s a generated list, so it also contains libraries that intrinsically form part of the
OMP1 ecosystem (such as
ppx_tools_versioned), PPXs that have already been ported but haven’t relesed their port on opam yet (such as
graphql_ppx), deprecated PPXs that aren’t marked as deprecated yet (such as
mirage-dns), and several PPXs that only transitively depend on
ppx_import has a PR for a port to
ppxlib, but it’s not quite ready to be merged just yet.