Adopting macros as an alternative to ppx

Ecosystem
1

After years of chasing issues with ppx, I think the core team needs to take a serious look at the alternative, which is integrating a macro language into OCaml itself. Honestly, it seems like less work at this point, and I fear there’s a sunk-cost fallacy at play here that keeps providing inadequate solutions to the ecosystem over very long time periods.

Here is an example of a user who attempted to solve the problem by himself. I’m sure the core team can pull this off, especially now that astlib basically moves much of the burden over to the compiler.

8 Likes
2

Then it would make sense to pull off something closer to BER MetaOCaml, at least conceptually.

2 Likes
3

I’m not sure on why that isn’t a valid solution, user space macros can be easily implemented with ppx, we just need to get something popular. Same ideas as on ppx_metaquot but instead emitting patched code, the only problem that I can see is getting cross module macros, which would be cool to get data from the AST inside cmi on user space to be queried by another build

1 Like
4

The fundamental problem of ppx is that it targets the AST, which is not a stable interface. This is the problem that keeps haunting the ppx ecosystem in different variations. A built-in macro system would take away that problem and be far easier to maintain. At this point, ppx is going to inject layers of backwards-compatibility into the compiler, making it the responsibility of the core dev team to maintain it. It seems wise to consider the alternative of creating a proper macro language rather than dragging backwards-compatibility baggage into the compiler proper. We could still keep ppx, but if it’s possible to do most of what ppx does in a macro language (and despite the potential of ppx to make global changes, experience shows that local changes are really the main application), then ppx would be relegated to specific niche uses that would be less bothered by delayed compatibility with changes to the AST.

A user-level macro system utilizing ppx is neat and interesting, but other than making macros more accessible to users, it doesn’t solve the main problem.

A good reference point is the Rust macro system, which allows matching on different syntactic elements, constructing some code, then matching on some other sub-elements, etc.

4 Likes
5

[I’m not a fan of the current *implementation methodology* of the PPX subsystem, but] I doubt it’s possible to give a full-power macro-system for a language, without inevitably being forced to make some hard choices. The PPX/OMP/ppxlib designers have chosen among them, and while I might disagree with their choices, they -are- on the menu of choices that lead to a powerful macro sysetm. Here are some of the choices:

(1) [I don’t know Rust’s macro system well, but last I looked, it wasn’t very powerful] You allow only a restricted macro system, so you don’t have to implement deep support for pattern-matching/construction of ASTs

(2) You allow very powerful macros (basically able to take apart and rebuild arbitrary code fragments) and then you’ve got some more choices to make:

(a) via something OMP, you allow for version-to-version compatibility as the AST type itself changes [con: yeah, gotta maintain OMP]

(b) you commit to fixing all existing PPX rewriters soon after each major Ocaml release [con: ouch, lotta work, also wow, talk about chaos]

(c ) you eschew binary AST communication between the macro-system and the compiler; and since concrete syntax changes very, very slowly, this allows you to change the AST type in the compiler without blowing up the macro system [con: slower, maybe also more fragile]
ETA: by this I mean, use source-code as the comms format between PPX and compiler. This is obvs slower, somewhat fragile, and also prevents any decent attempt at mapping errors back to source-code locations.

(d) you use a very “loose” AST type in the macro-system, so that minor changes of the compiler’s AST type don’t induce changes in this loose AST type [con: errors at runtime that might have been caught earlier]

Of course, #2a is the current course-and-speed. #2d has been proposed, and while it seems attractive, I think it’ll be more trouble than it’s worth.

I don’t see how shifting the work onto the compiler team helps: they still need to maintain compatibility with code written against older versions of the compiler, which might be expecting a slightly different AST type, which leads to these same choices.

Now, I -do- think that it’s possible to make OMP much cheaper to build, maintain, and use, using PPX rewriters. I’m in the process of prototyping that right now. But that’s neither here nor there: the truth is, something like OMP is … necessary.

Or at least, so it seems to me.

1 Like
6

Another option would be to restrict AST (therefore macro subsystem compatibility layers) to “editions”, like Rust did.

7

We could also keep iterating on the “powerful macros” side with OMP and the likes, while having a nice, simple, stable system of macros like rust. (And deriving, which is also a nice restricted subset.) A solution that gets you to 80% for 20% of the complexity :stuck_out_tongue:

5 Likes
8

There was announcement of some multi-staged programming effort similar to (and inspired by) MetaOCaml years ago. Maybe it was this one: https://oliviernicole.github.io/about_macros.html

Does anyone know what happened (or didn’t happen) with it? It seemed really promising. I like MetaOCaml, but in practice I’m going to stick with the mainline supported compiler, so I’ve been hopeful for something making into the language as standard.

1 Like
9

Hi, I wrote that blog post during the internship when I was implementing that prototype. I’m afraid I cannot offer any clues as to whether it will be considered for integration someday, as I have not heard anything on that topic.

I wanted to stress that this kind of macros in the style of MetaOCaml can do interesting stuff for code optimization (e.g. we were able to port the strymonas library from MetaOCaml to our system pretty easily), but not as much for simplifying syntax or generating boilerplate code, like ppx are useful for. The reason is that these macros are not analytical, i.e. they can build OCaml expressions but not take them apart. You might be able to do some fancy things using the type system, but I doubt it could be a replacement for e.g. ppx_deriving in the current design.

8 Likes
10

My opinion is that there should be rust style macro in OCaml, that is not very powerful macros, but that have a very low barrier to entry. The ppx system should still exist for more complicated stuff.

2 Likes
11

Given the recent report on ppxlib, I thought it would be reasonable to bump/resurface this topic.

Since ppx’s objectives really are no longer in play, and the operationalized process is “option 2(b)” per @Chet_Murthy’s telling (but with patches coming out of some centralized process):

…it seems to me that there is a clear opening (as well as potential userbase) for something quite different from ppx that offers a simpler surface area, easier implementation, and more stable interface. Whether that implies a more rust-esque system or a source-to-source transliteration, there’s a lot of conceptual ground that is left uncovered by the current “blessed” spectrum of cppo/ppx.

5 Likes