Best Practices for Attributes in PPX

Learning
1

I’m using Ppxlib to write a PPX transformer. The documentation indicates that, in order to preserve forward compatibility, I should be using Ast_builder.Default in order to construct expressions. I need to transform an expression and copy its attributes: something like turning

(1+2)[@foo]

into

(1[@foo])+(2[@foo])

However, Ast_builder.Default doesn’t appear to have any functions which construct expressions with non-empty pexp_attributes fields. Am I missing something? Am I relegated to writing a record update expression in this case?

2

I’m not a 100% sure, but using metaquot might solve your issue without loss of forward compatibility

3

When I was working on pa_ppx_parsetree I came across this problem, too. Here’s an explanation of what’s going wrong. Erm, I mean, here’s my story for what’s going wrong (grin and I’m sticking to it!)

If you look at the OCaml grammar, you’ll find that there’s no good support for an expression without attributes. Everywhere, it’s an expression with attributes. So if you want to write a metaquotation for an expression with attributes, what would you write? Suppose you have an expression 1 [@foo]. Now you want to replace that 1 with a metavariable, so you wrote $e$ [@foo] (or whatever way that you write metavariables in metaquot). But e is now a metavariable, of type … expression. So it has attributes itself. So what does this mean? Does it mean remove the attributes of e and replace them with [@foo] ? Or does it mean append? prepend ? sort ? It’s all a little vague, isn’t it?

I just set this aside back when it came up, but now that you’ve dredged it back up, and in such an obviously relevant situation, I’m going to dig into it and see if there’s a way to represent a solution.

But (at least, I suspect) you’re not crazy to think that there’s a problem.

1 Like
4

OK, that was easier than I thought it’d be.

# let e1 = {%expression| (1+2)[@foo] |} ;;
# let a2 = {%attribute| [@bar] |} ;;

# let move1 e = match e with
{%expression| ($e1$ + $e2$) $algattrs:l$ |} ->
  {%expression| ($noattrs:e1$ $algattrs:l$) + ($noattrs:e2$ $algattrs:l$) |}
;;

# let pexp e = Fmt.(pf stdout "%a" Pprintast.expression e) ;;
# pexp (move1 e1) ;;
((1)[@foo ]) + ((2)[@foo ])
# e1 |> (fun e -> {%expression| $noattrs:e$ |}) |> pexp ;;
1 + 2- : unit = ()
#  e1 |> (fun e -> {%expression| $noattrs:e$ $algattrs:[a2]$ |}) |> pexp ;;

So an expression antiquotation (that is, an antiquotation in a position that is an expression and not a pattern) in a position that is predicted to be an expression, can be marked as “noattrs”, and that will cause the attrs to be stripped off. And this can be combined with adding attributes as in the last expression above.

This is another example of why and how one needs to be able to step outside the standard OCaml grammar in a lightweight way, to get metaquotations that are actually useful (where “actually useful” is defined as “makes the need to work directly with AST constructors almost nonexistent”).

1 Like
5

You’ve certainly sold me on the approach you’re using. Part of the reason that I’m asking these PPX questions is that, in the process of writing my own fairly complicated transformer, I’m trying to modularize in such a way as to offer the core facilities of the transformer as a library. I’d like that library to be forward compatible, so I’m trying to learn best practices.

With forward compatibility in mind: what kind of stability and support is there with pa_ppx_parsetree? I’m sold on the idea of it, but I’m quite attuned to IDE support and I know that supporting a superlanguage’s syntax is a heavy burden in that regard.

That said, my question was more about the API for ppxlib.metaquot: I can’t find any functions which allow me to construct an expression with attributes. That is, one can manually write something like

{ pexp_desc = Pexp_ident(Pexp_constant(Pconst_char('a')));
  pexp_loc = loc;
  pexp_loc_stack = [];
  pexp_attributes = [
    { attr_name = Loc.make ~loc "b";
      attr_payload = PStr([]);
      attr_loc = loc;
    }
  ];
}

which is fragile: if the expression type gets modified at all, we’re in trouble. Of course, one might write the much more legible

[%expr 'a'[@b]]

but, as we’ve discussed, ppxlib.metaquot has its limitations and this is one of them. So instead, I was opting for the documentation-suggested Ast_builder.Default module which contains a series of constructor functions for expressions. For instance, one can write

Ast_builder.Default.pexp_ident ~loc (Loc.make ~loc "x")

to represent the variable x or even just

echar 'a'

to represent the character constant 'a' as an expression. This is all very useful, but I can’t find anything in the ppxlib.metaquot documentation or library about how to create an expression with attributes attached (or how to pattern match on an expression’s attributes) without resorting to directly interacting with the record structures. For now, I’ve written

let pexp_with_attrs (attributes : attribute list) (expr : expression)
  : expression =
  { expr with pexp_attributes = attributes }
;;

so I can just

pexp_with_attrs [b_attr] @@ echar 'a'

but again: that seemed clunky, so I thought I’d ask if I was doing this properly. I’d hate to burn a bunch of time writing terribly-styled code if I’d just missed a small detail.

Thanks again for the help and perspective!

6

It’s based on Camlp5 (even though it can be used to build PPX rewriters that work with the standard infrastructure, via GitHub - chetmurthy/ppx_jsobject_conv at experimental-parsetree-quotation-hacking ) so … I think it’s fair to say that I’m the only person using it and/or working on/with it.

So, it’s fair to say: if you want something that’s got a community behind it, you should not use this thing. You might say that I’m John the Baptist howling in the desert, only … I’m not howling and I’m not in the desert: I’m sitting in a comfy chair listening to The Scorpions and making observations that I’m sure will be ignored.

As I did with ppx_jsobject_conv, I’m happy to demonstrate the correctness of my thesis by worked examples. But this is a far cry from an accepted, widely used package.

1 Like
7

i forgot to answer: there are two answers to this question: the first is that the syntax is completely compatible with normal OCaml. The syntax is plain old PPX extensions. The second answer is that the PPX rewriter infrastructure to expand these extensions is all based on Camlp5. That is incompatible with the normal PPX infrastructure, and also incompatible with the incremental parsing used in IDEs like Merlin. It should be possible to modify this package so that it can be used from the normal PPX infrastructure, but since I’m the only one using it and it seems pretty clear that nobody else is interested in this approach, it’s difficult for me to gin up the energy to do the work.

8

Since Preserve quoted attributes on antiquoted payloads by ncik-roberts · Pull Request #441 · ocaml-ppx/ppxlib · GitHub, in ppxlib.metaquot quoted attributes are merged with antiquoted one.

To quote (:smiley:) the PR:

let e = [%expr (() [@attr1])] in
[%expr [%e e] [@attr2]]

(* Old output: *)
() [@attr1]

(* New output: *)
() [@attr1] [@attr2]

(I think it was released in ppxlib 0.31.0)

1 Like
9

The fact that Ast_builder or Ast_pattern does not allow to build/match anything with an attribute is worth opening an issue though.

10

Done. Thanks for the encouragement to do so. :slight_smile:

11

This all makes sense. For what it’s worth, I agree with the perspective you’re taking. I’d love to see languages like OCaml have well-defined mechanisms for extending the grammar as you’ve done here and the developer toolchain have facilities for extending support to them (either by encoding and source mapping or by corresponding extension mechanisms). But I too am not sufficiently motivated to make the technique work.

So, by strained analogy, I suppose that makes me a doofy little cactus nearby in the desert. I’m relatively inert and not likely to be of much help in spreading the message, but I’m planted in the same desert. :smiley:

Thanks for the chat! It was enlightening and has provided me with enough perspective to be comfortable with how my PPX transformer looks. I was just worried I was making a hash of it and wouldn’t find out until I got an embarrassing PR. :wink: