Ocaml-migrate-parsetree 2.0.0

Community
,
1

Dear all,

We are preparing a 2.0.0 release of ocaml-migrate-parsetree. This new major release will simplify the project to its core purpose, which is to provide migration functions between the various versions of the parsetree types. Everything else, such as the driver or the per-version copy of various modules from the compiler distribution will go away. This will have an impact for projects that are currently using these functionalities and this post describes what we think such projects should do and how we plan to help.

The motivation for this change is to simplify ocaml-migrate-parsetree and make it easier to maintain. The core functionality it provides is valuable and worth maintaining, however right now the project is way too big and difficult to maintain. Going through compiler upgrades has been unpleasant and difficult. And because of this, bugs of the segfault kind have accidentally been introduced and detected much later.

Once we have dropped everything that doesn’t fit in the purpose of the project, what will remain will be pretty simple, coherent and easy to maintain.

What is the replacement?

At the moment, ocaml-migrate-parsetree offers one way of writing ppx rewriters. For instance, one might write the following to create a ppx rewriter with ocaml-migrate-parsetree:

open Migrate_parsetree
open Ast_404

let structure = ...
let signature = ...

let () =
  Driver.register ~name:"ppx_cstruct" Versions.ocaml_404
    (fun _config _cookies -> {default_mapper with structure; signature})

Following the simplification, the above code will break for at least two reasons:

  • the Migrate_parsetree.Driver will be gone
  • the default_mapper and mapper type coming from
    Ast_404.Ast_mapper will be gone

As a result, ocaml-migrate-parsetree itself will no longer be enough to write ppx rewriters.

We recommend that ppx rewriters using the ocaml-migrate-parsetree API switch to ppxlib. The purpose of ppxlib is to provide a good API for ppx rewriters with in particular a simple and good composition model, so it seems like a natural choice. Since ppxlib will take a more central place in the ecosystem, we are currently addressing the more controversial aspects of this library. For instance, we are dropping the dependency on Base.

Organising the transition

The following ticket is the meta issue tracking the preparation of the 2.0.0 release:

To make this transition smoother, we are planning to help with the porting effort. We have prepared a list of ppx rewriters to port to ppxlib. This list is not exhaustive and in particular excludes ppx rewriters that have no reverse dependencies in opam in order to keep this work manageable for us. In any case, help porting projects to ppxlib would be very much appreciated.

If you would like to help porting a project, please mention on the corresponding ticket from the list that you are looking at porting this project so that we know someone is working on it. If you would like to port a project that is not on this list, you should feel free to open a new ticket and mention that you are porting this project. In this case, please make sure to add the label port-to-ppxlib to the issue.

Once we have successfully converted a few projects, we will proceed with the release of ocaml-migrate-parsetree 2.0.0 and a compatible ppxlib release. At this point, most existing reverse dependencies of ocaml-migrate-parsetree will be marked as incompatible with ocaml-migrate-parsetree 2.0.0 and we will continue porting existing projects to ppxlib to gradually make the world compatible with ocaml-migrate-parsetree 2.0.0.

Consequences for other ppx libraries

ppx_deriving

The master of ppx_deriving is now using ppxlib, so there shouldn’t be much to do to make ppx_deriving compatible with ocaml-migrate-parsetree 2.0.0.

ppx_tools_versioned

ppx_tools_versioned will no longer have a reason to exist in this new state of the world and so will stop being updated.

7 Likes
2

PS: I’m away from tomorrow until the end of the week, so I won’t be able to answer questions until next week.

3

Hi @jeremiedimino,

Thanks for the heads-up! I have a couple of questions:

  1. what is the story with dune? Right now, dune “just works” with ppx rewriters based on OMP, what will happen after the announced evolutions?
  2. Will the planned updates to ppxlib (getting rid of base, etc.) be done in time for the release of OMP 2.0.0?
  3. Would it be possible to make a minimal version of ppxlib/OMP for those that are willing to write a ppx rewriter against compiler-libs directly and do not need/want the rest of the library, but still want their rewriters to be compatible with dune?
  4. Could you say a few words on how the evolution of ppxlib is related to ppx?

Thanks!

Best wishes,
Nicolás

2 Likes
4

There is also ppx_deriving_protobuf #28 for porting to ppxlib.

Would also be interesting to make Multicore transition easier: ocaml-migrate-parsetree #89.

1 Like
5

Hi @nojb,

what is the story with dune? Right now, dune “just works” with ppx rewriters based on OMP, what will happen after the announced evolutions?

OMP won’t provide a driver anymore. To write a ppx rewriter that is compatible with Dune, you will have to use ppxlib. You could also write your own driver, thought I wouldn’t recommend doing that; we are trying to reduce the number of APIs rather that increase it :slight_smile:

Will the planned updates to ppxlib (getting rid of base, etc.) be done in time for the release of OMP 2.0.0?

Yes. @ceastlund started working on dropping the Base dependency and since it will be the same group of people doing the ppxlib and omp changes, we will make sure that there is a new version of ppxlib available on time. We plan to do things in this order:

  • prepare OMP2 and an updated ppxlib
  • add upper bounds to all OMP reverse dependencies in the opam repository
  • release OMP2
  • release an updated ppxlib shortly after OMP2 is merged in the opam repository

Would it be possible to make a minimal version of ppxlib/OMP for those that are willing to write a ppx rewriter against compiler-libs directly and do not need/want the rest of the library, but still want their rewriters to be compatible with dune?

Technically yes. That said, I wouldn’t recommend going down this route. Once most of the dependencies of ppxlib have been dropped and OMP has been simplified, the resulting OMP+ppxlib combo will be pretty lean. If there are aspects of ppxlib that would prevent people from switching to it, we are happy to consider those.

Once we have finished our work on simplifying the ppx ecosystem, I also expect that the bare ppx API in the compiler will disappear, so I would definitely not recommend to use it.

Could you say a few words on how the evolution of ppxlib is related to ppx 1?

Sure. The ppx project is part of this work. Our plan was to move to a world like this:

  • the compiler offers a minimal stable Astlib library, using a dynamic AST
  • the ppx project is written only against Astlib and not the rest of compiler-libs
  • ppx has a stable API
  • ppx rewriters are written against the ppx API

In this world, existing ppx rewriters are compatible with trunk at all times. Moreover, when a new compiler is released:

  • existing released ppx rewriters are immediately compatible with the new compiler
  • one can use existing ppx rewriters and the new features of the language

in other words, you can in theory release a ppx rewriters once and never touch it again. The trade-off is that it uses a technology that is quite complex and users don’t have direct access to the parsetree types.

However, we don’t expect everyone to switch to this API at once. So we will need a transition period where the existing APIs are implemented as shims on top of the new ppx API. The more ppx APIs we have to consider and the more difficult this work will be for us. So one can see this work of simplifying OMP and ppxlib as preparatory work.

Now, all of the above was our original plan. We are now considering an alternative plan that would lead to a technically simpler system that is easier to reason about. The new plan would lead to the following world:

  • the compiler offers a minimal stable Astlib library, using the OMP technology
  • ppxlib is written only against Astlib and not the rest of compiler-libs
  • ppx rewriters are written against ppxlib

In this world, existing ppx rewriters are compatible with trunk at all times. Moreover, when a new compiler is released:

  • existing released ppx rewriters are immediately compatible with the new compiler
  • one cannot use existing ppx rewriters and the new features of the language until:
    • ppxlib has been updated to use the new parsetree types
    • the subset of ppx rewriters that are broken by this change have been updated as well

This new plan requires updating potentially many packages after each new compiler release. We know that from a technical point of view this is relatively easy. In fact, we do that all the time at Jane Street: when we bump the version of the AST, one single person sit downs and updates all the ppx rewriters at once, which doesn’t take very long. In the open-source world, this was historically impossible, though now that most OCaml projects are using dune this is becoming more realistic; we can create one big dune workspace with all the ppx rewriters and update them all at once. That said, there is a social component to this approach since the various packages are owned by various groups of people and even if one person does all the work, we still need various groups of people to take action to merge PRs and make new releases. So we plan to try this idea with one or two compiler release. In fact, @NathanReb is experimenting with this idea as we speak for the 4.10 upgrade of ppxlib. If this all works out well, we’ll continue with this new plan. Otherwise, we’ll go back to the original plan.

2 Likes
6

Indeed, either of the plan I described in my above message would help. The key point is to introduce a stable Astlib API in the compiler.

1 Like
7

We had a discussion this morning about ppxlib and versioning with @gasche and @NathanReb. In case anyone is interested, here is a summary of the discussion.

2 Likes