# PPX interface spec (with ocamlc/ocamlopt)? **URL:** \ **Category:** Ecosystem\ **Tags:** ppx\ **Created:** [May 5, 2023, 4:43am UTC](https://discuss.ocaml.org/t/ppx-interface-spec-with-ocamlc-ocamlopt/12100 "2023-05-05T04:43:42Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Chet\_Murthy](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/chet_murthy/32/1501_2.png) [@Chet\_Murthy](https://discuss.ocaml.org/u/Chet_Murthy)\ **Post date:** [May 5, 2023, 4:43am UTC](https://discuss.ocaml.org/t/ppx-interface-spec-with-ocamlc-ocamlopt/12100/1 "2023-05-05T04:43:42Z")
I searched a bit, and couldn’t find a documented specification for how ocamlc/ocamlopt/dune (and perhaps ocamlfind) invoke PPX rewriters. Is this documented someplace? Failing that, I’m going to reverse-engineer it from the code, but it sure would be nice if there were some specification. Anybody know of one? P.S. what I mean is, the precise way in which the compiler will invoke these PPX rewriter executables, with what arguments, and what those arguments mean. It would be nice to know the same for “staged\_pps” rewriters. ---
**Author:** ![sim642](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/sim642/32/3597_2.png) [@sim642](https://discuss.ocaml.org/u/sim642)\ **Post date:** [May 5, 2023, 6:15am UTC](https://discuss.ocaml.org/t/ppx-interface-spec-with-ocamlc-ocamlopt/12100/2 "2023-05-05T06:15:31Z")
[OCaml - Native-code compilation (ocamlopt)](https://v2.ocaml.org/manual/native.html) says: > -ppx command > After parsing, pipe the abstract syntax tree through the preprocessor command. The module Ast\_mapper, described in chapter [29](https://v2.ocaml.org/manual/parsing.html#c%3Aparsinglib): [Ast\_mapper](https://v2.ocaml.org/api/compilerlibref/Ast_mapper.html) , implements the external interface of a preprocessor. The linker chapter starts with: > The interface of a -ppx rewriter > > A -ppx rewriter is a program that accepts a serialized abstract syntax tree and outputs another, possibly modified, abstract syntax tree. This module encapsulates the interface between the compiler and the -ppx rewriters, handling such details as the serialization format, forwarding of command-line flags, and storing state. Not sure if this is enough detail or not. * * * Regarding dune’s `staged_pps`, I think each one of those might be a separate `-ppx` argument or something. Normally with ppxlib (and possibly even ppx\_deriving), all ppx-s are compiled into a single executable that calls all the ppx rewriters or derivers. ---
**Author:** ![Chet\_Murthy](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/chet_murthy/32/1501_2.png) [@Chet\_Murthy](https://discuss.ocaml.org/u/Chet_Murthy)\ **Post date:** [May 5, 2023, 7:06am UTC](https://discuss.ocaml.org/t/ppx-interface-spec-with-ocamlc-ocamlopt/12100/3 "2023-05-05T07:06:11Z")
To be frank, it’s not a spec. But that’s OK: I’m reverse-engineering it. It appears that the interface of a PPX rewriter executable is EXECUTABLE file1 file2 the serialized AST is found in file1 (either as an interf or impl – I haven’t figured out how you indicate the difference between the two – maybe it reads the magic number) and the result of PPX rewriting is written to file2, again as a serialized AST. ---
**Author:** ![nojb](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/nojb/32/519_2.png) [@nojb](https://discuss.ocaml.org/u/nojb)\ **Post date:** [May 5, 2023, 8:09am UTC](https://discuss.ocaml.org/t/ppx-interface-spec-with-ocamlc-ocamlopt/12100/4 "2023-05-05T08:09:20Z")
> [@Chet\_Murthy](#): > > the precise way in which the compiler will invoke these PPX rewriter executables, with what arguments, and what those arguments mean. While you can use `-ppx` (and `-pp`) to invoke a preprocessor, it is not necessary (and has the downside of going through the shell). In fact the compiler drivers `ocamlc` and `ocamlopt` will happily accept a file containing a marshalled AST instead of source code as argument. What Dune does is to build a single `ppx.exe` executable that bundles all PPX preprocessors in a single pass. This tool takes as input a source code file `foo.ml` and outputs a marshalled AST `foo.pp.ml`. After producing `foo.pp.ml`, Dune invokes the compiler with _this_ file as argument. Thus, the compiler never does any preprocessing on its own, all the preprocessing logic is managed by Dune and encoded in the `ppx.exe` helper executable. Cheers, Nicolas ---
**Author:** ![nojb](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/nojb/32/519_2.png) [@nojb](https://discuss.ocaml.org/u/nojb)\ **Post date:** [May 5, 2023, 8:11am UTC](https://discuss.ocaml.org/t/ppx-interface-spec-with-ocamlc-ocamlopt/12100/5 "2023-05-05T08:11:46Z")
> [@Chet\_Murthy](#): > > I haven’t figured out how you indicate the difference between the two – maybe it reads the magic number You can use `-intf` and `-impl` to tell the compiler if the file you are passing corresponds to an interface or an implementation. Cheers, Nicolas ---
**Author:** ![Chet\_Murthy](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/chet_murthy/32/1501_2.png) [@Chet\_Murthy](https://discuss.ocaml.org/u/Chet_Murthy)\ **Post date:** [May 5, 2023, 8:24am UTC](https://discuss.ocaml.org/t/ppx-interface-spec-with-ocamlc-ocamlopt/12100/6 "2023-05-05T08:24:17Z")
Ah, I meant the PPX rewriter executable. I tried with each of a .ml and a .mli file, and the invocation of the executable is the same. I conclude that it’s using the magic number to decide what it’s getting. ---
**Author:** ![nojb](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/nojb/32/519_2.png) [@nojb](https://discuss.ocaml.org/u/nojb)\ **Post date:** [May 5, 2023, 8:43am UTC](https://discuss.ocaml.org/t/ppx-interface-spec-with-ocamlc-ocamlopt/12100/7 "2023-05-05T08:43:14Z")
> [@Chet\_Murthy](#): > > I tried with each of a .ml and a .mli file, and the invocation of the executable is the same. I conclude that it’s using the magic number to decide what it’s getting. Not sure I follow; source code does not contain magic numbers. The PPX driver `ppx.exe` accepts the same `-intf` and `-impl` flags as the compiler to choose between interface and implementation. Absent that, it simply looks at the extension of the file (`.ml` or `.mli`). The PPX driver logic is provided by `ppxlib`: [https://github.com/ocaml-ppx/ppxlib/blob/main/src/driver.ml](https://github.com/ocaml-ppx/ppxlib/blob/main/src/driver.ml). Cheers, Nicolas ---
**Author:** ![Chet\_Murthy](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/chet_murthy/32/1501_2.png) [@Chet\_Murthy](https://discuss.ocaml.org/u/Chet_Murthy)\ **Post date:** [May 5, 2023, 8:48am UTC](https://discuss.ocaml.org/t/ppx-interface-spec-with-ocamlc-ocamlopt/12100/8 "2023-05-05T08:48:20Z")
Here’s the trace of commands of ocamlc on ML and MLI files. The script “PPX” is: ```auto #!/bin/bash -x SRC=$1 DST=$2 /home/chet/Hack/Opam-2.1.2/GENERIC/5.0.0/lib/ppx_here/./ppx.exe --as-ppx $SRC $DST cp $SRC saved.src cp $DST saved.dst ``` So it seems that the PPX driver is _not_ passed `-impl` and `-intf` args. ocamlc on an ML file: ```auto ocamlfind ocamlc -verbose -package fmt,camlp5 -ppx ./PPX -c foo.ml Effective set of compiler predicates: pkg_fmt,pkg_compiler-libs,pkg_compiler-libs.common,pkg_camlp-streams,pkg_camlp5,autolink,byte + ocamlc.opt -verbose -ppx ./PPX -c -I /home/chet/Hack/Opam-2.1.2/GENERIC/5.0.0/lib/fmt -I /home/chet/Hack/Opam-2.1.2/GENERIC/5.0.0/lib/ocaml/compiler-libs -I /home/chet/Hack/Opam-2.1.2/GENERIC/5.0.0/lib/camlp-streams -I /home/chet/Hack/Opam-2.1.2/GENERIC/5.0.0/lib/camlp5 foo.ml + ./PPX '/tmp/camlppx309261' '/tmp/camlppxbd5172' ``` ocamlc with an mli file: ```auto + ocamlfind ocamlc -verbose -package fmt,camlp5 -ppx ./PPX -c foo.mli Effective set of compiler predicates: pkg_fmt,pkg_compiler-libs,pkg_compiler-libs.common,pkg_camlp-streams,pkg_camlp5,autolink,byte + ocamlc.opt -verbose -ppx ./PPX -c -I /home/chet/Hack/Opam-2.1.2/GENERIC/5.0.0/lib/fmt -I /home/chet/Hack/Opam-2.1.2/GENERIC/5.0.0/lib/ocaml/compiler-libs -I /home/chet/Hack/Opam-2.1.2/GENERIC/5.0.0/lib/camlp-streams -I /home/chet/Hack/Opam-2.1.2/GENERIC/5.0.0/lib/camlp5 foo.mli + ./PPX '/tmp/camlppxce9c6c' '/tmp/camlppxd1307e' ``` P.S. PPX rewriters are not passed source, right? They’re passed serialized ASTs, and return the same. ---
**Author:** ![nojb](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/nojb/32/519_2.png) [@nojb](https://discuss.ocaml.org/u/nojb)\ **Post date:** [May 5, 2023, 8:56am UTC](https://discuss.ocaml.org/t/ppx-interface-spec-with-ocamlc-ocamlopt/12100/9 "2023-05-05T08:56:00Z")
> [@Chet\_Murthy](#): > > P.S. PPX rewriters are not passed source, right? They’re passed serialized ASTs, and return the same. I was explaining what Dune does (which does not use `-ppx`). Dune passes source code as input to the PPX rewriter. On the other hand, the mechanism used if you pass `-ppx` to the compiler indeed uses marshalled ASTs for input and the magic number to know whether it is an interface or an implementation. Cheers, Nicolas ---
**Author:** ![jjb](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/jjb/32/2713_2.png) [@jjb](https://discuss.ocaml.org/u/jjb)\ **Post date:** [May 5, 2023, 9:17am UTC](https://discuss.ocaml.org/t/ppx-interface-spec-with-ocamlc-ocamlopt/12100/10 "2023-05-05T09:17:17Z")
Just to emphasize: there are significant build speed advantages to the approach based on building a separate ppx driver. First, if there are multiple rewriters, they get combined into a single pass over the parsetree in the ppxlib driver approach. That has mattered a lot in my experience. Second, when the preprocessed parsetree is written to a file, it can be tracked as an intermediate result by the build system. Depending on what you are doing, reusing that work can be meaningful. ---
**Author:** ![Chet\_Murthy](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/chet_murthy/32/1501_2.png) [@Chet\_Murthy](https://discuss.ocaml.org/u/Chet_Murthy)\ **Post date:** [May 5, 2023, 6:04pm UTC](https://discuss.ocaml.org/t/ppx-interface-spec-with-ocamlc-ocamlopt/12100/11 "2023-05-05T18:04:32Z")
My current little project is to add “ocaml.ppx.context” support to the invocation path to Camlp5. I agree that combining PPX rewriters into a single executable (and when possible, the smallest number of passes) is a good thing. Camlp5 already does the former (and always has); the latter is a matter of topological sort and some clever merging. I’d like to reuse the result of preprocessing, but that requires being able to tease apart findlib packages into the part used by the preprocessor, and the part used by the compiler (ocamlc/opt). That’s a little too complicated to do in a Makefile.