Ppx: omp 2.0.0 and next steps

Community
, ,
1

quick summary:

  • ocaml-migrate-parsetree 2.0.0 release
  • you should add a upper bound in your dev repos
  • ppxlib compatible version coming soon
  • ppxlib is now the official ppx library supported by the OCaml platform

Hi everyone,

As previously announced, we are releasing the version 2.0.0 of ocaml-migrate-parsetree. At the moment nothing is compatible with the new version and we will soon release a version of ppxlib that is compatible with it. If your project depends on ocaml-migrate-parsetree, you should add a upper bound to your development repository.

If you plan to use ocaml-migrate-parsetree 2.0.0 directly, please note however that this is a transitory package. The technology implemented by ocaml-migrate-parsetree will live on and hopefully find a new home in the compiler repository proper. However, ocaml-migrate-parsetree as a standalone project will eventually stop being maintained.

I am also taking the opportunity to announce that ppxlib is the first ppx library officially supported by the OCaml platform, and the one we recommend all ppx authors to use. It is the library that we plan to maintain for the long term.

Other libraries such as ppx_tools or ppx_tools_versioned may continue to be maintained by open source contributors, however they will not be maintained by the OCaml platform and will not receive updates from the platform when new compilers are released. Only ppxlib will receive updates from the platform.

If you would like to port your project to use ppxlib and are experiencing difficulties or have any question, please get in touch by replying to this post or opening a ticket on github.com/ocaml-ppx/ppxlib.

The overall plan described in this post is the result of various discussions and/or collaborative effort between the following people: @avsm, @ceastlund, @Drup, @gasche, @jeremiedimino, @kit-ty-kate, @let-def, @NathanReb and @pitag.

Next steps

As soon as the new version of ppxlib is released, we will work towards our next milestone. As a reminder, our current goal is to setup a ppx ecosystem that is continously compatible with the trunk of OCaml. To achieve that goal, we plan to add a stable API called “Astlib” on top of the compiler libraries. To keep things sustainable on the compiler side and increase flexibility, Astlib will be minimal and will be targeted at ppxlib only rather than be a general API aimed at ppx authors.

The difficulty of this API is that it must expose a stable interface to the OCaml AST, which is composed of a large collection of data types. To make it work, we plan to use the technology developed in ocaml-migrate-parsetree; i.e. whole AST migration functions.

While we eventually want Astlib to live in the compiler repository, we will initially develop it inside the ppxlib repository. Once it is ready, we will submit it for inclusion in the compiler. Although, we will keep a copy inside ppxlib for older versions of the compiler.

We also plan to setup a smooth workflow for compiler developers to update Astlib when they change the development AST.

Once this is all done, we will be in a situation where the ppx ecosystem is compatible with the trunk of OCaml at all time. And as a result, new releases of the compiler will no longer break ppx packages as long as they limit themselves to the ppxlib API.

Future

While this work will make the ecosystem compatible with the trunk of OCaml at all times, it will essentially move the backward compatibility issue from the compiler to ppxlib.[1] This will already give us a lot more flexibility as for instance a single version of ppxlib can be compatible with a wide range of OCaml versions. However, we recognise that it is not usual to ask a community to rely on an unstable API.

We made this choice as a trade-off between sustainability and complexity. Indeed, we want to maintain Astlib and Ppxlib over the long term and the best way to make things sustainable is to use simple and clear designs. While we do have solutions in our sleeves that would provide a fully stable ppx API, these are much more complicated to maintain and work with.

To mitigate this, we are setting up a Dune based workflow to upgrade all ppx rewriters at once. So once the system is rolling and if your ppx rewriters are up to date and using Dune, you should expect to receive pull requests as we update ppxlib. This last part will take some time to be fully rolling, so please bear with us :slight_smile:

In any case, about a year after this new world is fully setup, we will review the situation and decide whether it is sustainable or whether we need to go all the way and mint a fully stable ppx API.

Timeline

  • today: ocaml-migrate-parsetree 2.0.0 is being released

  • next week: a ppxlib compatible version is released

  • December 2020: astlib is ready inside the ppxlib repository

  • next OCaml release after that: astlib lives in the compiler

  • September 2021: we review the situation and decide what to do next

[1]: At any given time the API of ppxlib refer to a single version of the OCaml AST. In order to allow OCaml users to enjoy both ppx rewriters and new language features, the version of the AST selected by ppxlib needs to be bumped after each release of the compiler, which is a breaking change that has the potential to break several ppx packages. As a result, ppx packages will still need to be regularly updated in order to stay compatible with the latest version of ppxlib.

17 Likes
2

pardon the stupid question, but given:

As a result, ppx packages will still need to be regularly updated in order to stay compatible with the latest version of ppxlib.

what does the 2.0 version change compared to the current situation?

1 Like
3

ppx packages currently target many versions of OCaml ASTs. With ppxlib 2.0+, they will target a single version (chosen by ppxlib, and bumped regularly). This in turn means that the nightmarish complexity of Ocaml_migrate-parsetree migration between multiple OCaml AST versions can be safely retired (with much respect to the omp contributors who kept it going for far longer than I had initially believed possible).

2 Likes
4

In addition to what @avsm said, I’ll say that releasing omp 2 is the first concrete step we are taking towards simplifying the ppx stack. In practice, we could have left things as they were and waited for astlib to be ready. However, we announced our plan to re-organise the ppx world more than a year ago and I wanted to avoid the “Dune Nuken Forever” effect. With this plan, we are moving the world forward incrementally one milestone at a time.

In any case, if you switch to ppxlib now, from your point view things shouldn’t change until September 2021 when we will review the situation.

3 Likes
5

Curious to hear what you think about the recently announced pa_ppx which relies on camlp5’s AST. Afaik this AST is already akin to what Astlib would eventually be?

1 Like
6

An orthogonal effort. We (that is, the ppx maintainers) are not going to take a hard dependency on camlp5 after spending years migrating away from camlp4, and then years maintaining camlp4 out-of-tree while migrating users and libraries. Feel free to read the archives of wg-camlp4 for the justifications for that decision.

Astlib (as eventually present in the OCaml compiler) would depend only on the OCaml compiler libraries. Others are, of course, extremely free to maintain any such code generation systems for their own needs. All such contributions are welcome in opam-repository, but we make no guarantee from an OCaml release perspective that we’ll consider the maintenance burden of these other tools.

2 Likes
7

The thread is from 2013 :slightly_smiling_face:. No one is suggesting to go back to custom syntax extensions, I think it’s more about the way to preprocess away all the extension points. It is true that camlp5 is a big dependency, but then so is OMP and the likes.

Now, seriously, I’ll stop wasting y’all’s time and I’ll wait for Astlib.

1 Like
8

What happens with just ppx project that was announced in https://discuss.ocaml.org/t/the-future-of-ppx/?

I noticed it wasn’t updated for a long time while the activity in ppxlib is steady. Is that old ppx effort abandoned?

1 Like
9

What is the ocaml platform at this point?

1 Like
10

@c-cube no worries, it’s good to clarify these things. We were expecting questions :slight_smile:

We have been working on ppx and the ppx stack for years now and since last year we have been putting a lot of effort towards simplifying it. I haven’t looked in detail at pa_ppx, but as Anil my first impression is that I’m not particularly excited about reviving the camlp* world. If pa_ppx was becoming widely adopted and ppx users massively moved to it, then after some time we would of course acknowledge the new reality and review our plans. But for now, we are continuing as planned. At the end of the day this is all open-source and everyone is free to make their own choice, so all I can say is best of luck to pa_ppx and let see how it goes.

This is what I was referring to when I said “While we do have solutions in our sleeves that would provide a fully stable ppx API”. We eventually decided that this solution was too complex technically and decided to go for a technically simpler one. So we went back to work on ppxlib. But again, we will review the situation after a year. I personally expect that it will work well and we can continue with it. Future will tell.

What is the ocaml platform at this point?

@avsm can probably describe it better than me, but I would say it is a group of people whose responsibility is to maintain a coherent set of tools and libraries to serve as basis to develop OCaml applications.

3 Likes
11

While discussing pa_ppx, one point that, from a user’s point of view, is a real enabler for ppxlib is the driver and compilation model. I am not sure if the design of pa_ppx allows (eventually) getting to the point where all the rewriters can be combined into one pass. My personal experience is that without that, no matter what the rest of the interface looks like, it is just not feasible to use. Compilation times suffer a lot, and they slow down significantly for every additional rewriter. So I would say that until pa_ppx gets to a point where that is possible, it does not make sense for work on the build system to invest serious time in to special support for it, and ppx(lib) ought to stay the default for users and rewriter authors. At the point when compilation time isn’t O(number of rewriters), then I think reconsidering the situation would make sense.

3 Likes
12

That’s not a problem.

pa_ppx, like camlp5 generally, has always been able to deal with multiple drivers being combined into a single (bytecode or native) executable. There has always been a mkcamlp5 (mkcamlp5.opt) script that can build such executables, and I’ve been using executables built this way as part of some of the “trip tests” for camlp5 compatibility for several months now.

But on the other hand, being able to write PPX rewriters with such ease means that I write more of them. So for instance, in the case of “type extensions” (e.g. type M.t += A = N.E – add N.E to the type M.t under the name A) a deriver would be unable to infer the type of the payload of N.E and hence couldn’t derive code for it. So instead, I write

type B.u += C of int [@rebind_to A.A ;] \
     [@@deriving show, sexp, yojson, eq]

and there are N+1 passes: N(=4 here) derivers, followed by a “cleanup” pass that changes that constructor declaration to C = A.A before handing it to the Ocaml compiler.

I have code (generated via a PPX rewriter, ha!) to do hash-consing of the AST but haven’t gotten around to using it to speed up the rewrite passes.

3 Likes
13

6 posts were split to a new topic: Adopting macros as an alternative to ppx