[ANN] ppx_deriving.6.0.2 and ppx_deriving_yojson.3.8.0

[NathanReb]

I am happy to announce the release of ppx_deriving.6.0.2 and ppx_deriving_yojson.3.8.0, the first release of those packages in years!

The main feature here is the port of ppx_deriving’s standard derivers ([@@deriving show, make, ord, eq, ...]) and ppx_deriving_yojson to ppxlib’s Deriving api.
There are no changes to how you’d use those derivers but many benefits:

• Better performances and better integration with other ppx-es as the code is now generated as part of ppxlib’s driver main AST rewriting phase rather than in a separate, dedicated phase.
• They can now be used with [@@deriving_inline]
• None of them will break the location invariant required by merlin anymore, fixing a long lasting bug and providing a much better user experience.

You can find the full release notes for ppx_deriving here and for ppx_deriving_yojson here.

I’d like to thank @sim642 for all their work on the ppxlib ports and their patience, and all our other contributors.

I’d also like to thank the OCaml Software Foundation who has been funding my work on those releases.

[c-cube]

Thank you for working on this! It’s great to hear that these (imho very important) ppxes are modernized.

[yawaramin]

Thank you! ppx_deriving_yojson is low key one of the best parts of the OCaml ecosystem.

[zoj613]

Could you elaborate on why you feel this way? How is it different from Jane Street’s ppx_yojson_conv? which one of the two do you recommend?

[c-cube]

I’m sure ppx_yojson_conv is high quality, but personally I look at its
dependency list and see base, ppx_js_style, and ppx_yojson_conv_lib,
which tells me it’s opinionated for people in the JaneStreet
ecosystem.

ppx_deriving_yojson has been here for a really long time and works just
fine, this ppxlib change brings it to current best practices, so it’s
great!

[zoj613]

Thanks for the response. I like that the return type of the ppx_deriving_yojson conversion functions is Result, unlike the exception (or undefined behavior?) throwing jane street one which makes it a little harder to work with. I noticed that with the Jane Street package in order to use Yojson.Safe.t as a conversion type you have to use a hack like so: Deriving converters for `Yojson.Safe.t` · Issue #7 · janestreet/ppx_yojson_conv · GitHub . Does ppx_deriving_yojson have the same issue?

[c-cube]

It looks like using Yojson.Safe.t is directly supported, see the table
in GitHub - ocaml-ppx/ppx_deriving_yojson: A Yojson codec generator for OCaml.

[Chet_Murthy]

Something seems to have broken in ppx_deriving (and also ppx_deriving_yojson) between 5.2.1 and 6.0.2. It seems like using ppx_deriving.show from dune gives different results than using it from ocamlfind. Attached is a trivial example (which I extracted from the “show” test in the ppx_deriving sources).

The problem appears to be that the with_path option (which used to be by default true) is not recognized (at least, when being build via ocamlfind). I also built a little dune project and verified that the below test succeeds, and that the with_path option is recognized and works the way one expects.

Has something changed about the way that these plugins must be invoked ? Is ocamlfind no longer a supported path to invoke them?

P.S. I got the same problems with ppx_deriving_yojson, but I’m guessing that the root cause is the same as for ppx_deriving.show.

open OUnit2

let printer = fun x -> x

type v = Foo | Bar of int * string | Baz of string [@@deriving show]
let test_variant ctxt =
  assert_equal ~printer "Foo.Foo"                (show_v Foo);
  assert_equal ~printer "(Foo.Bar (1, \"foo\"))" (show_v (Bar (1, "foo")));
  assert_equal ~printer "(Foo.Baz \"foo\")"      (show_v (Baz "foo"))

let suite = "Test deriving(show)" >::: [
    "test_variant"         >:: test_variant
  ]

let _ = run_test_tt_main suite
;;
print_string (show_v Foo) ;;

If I compile this with ocamlfind, I get:

+ ocamlfind ocamlc -g -custom -package ounit2,ppx_deriving.show -linkpkg -o foobad foo.ml
+ echo '================ BAD ================'
================ BAD ================
+ ./foobad
F
==============================================================================
Error: Test deriving(show):0:test_variant.

File "/home/chet/Hack/Camlp5/tmp/ppx_deriving/_build/oUnit-Test deriving(show)-moskva#01.log", line 2, characters 1-1:
Error: Test deriving(show):0:test_variant (in the log).

File "foo.ml", line 7, characters 1-1:
Error: Test deriving(show):0:test_variant (in the code).

expected: Foo.Foo but got: Foo
------------------------------------------------------------------------------
Ran: 1 tests in: 0.10 seconds.
FAILED: Cases: 1 Tried: 1 Errors: 0 Failures: 1 Skip:  0 Todo: 0 Timeouts: 0.

ETA: I hope its clear that everything works correctly when built with dune. If I take the test_deriving_show.ml from the ppx_deriving sources and build that using ocamlfind, I get similar errors to that above; I just used a smaller example b/c it’s … smaller.

[sim642]

I don’t know why it is different with ocamlfind, but it probably has to do with this code: ppx_deriving/src_plugins/show/ppx_deriving_show.ml at 32f7c31ecfcbca6d53f3655a00e8852f4751123e · ocaml-ppx/ppx_deriving · GitHub (ppx_deriving_yojson has a copy of the same logic).
This is responsible for constructing the path and the old ppx_deriving API code to do so was ported to use ppxlib functions instead. I don’t know whether the problem is in this code itself or some of the ppxlib functions it calls. As the comment says, there are other issues with getting path information from ppxlib as well.

[Chet_Murthy]

NO, this is all wrong! I was testing against the wrong version of ppx_deriving. OOPS!

Interesting. I had noticed that the PPX-expanded version of the file that dune creates, had this “context” at its start:

[@@@ocaml.ppx.context
  {
    tool_name = "ppx_driver";
    include_dirs = [];
    hidden_include_dirs = [];
    load_path = ([], []);
    open_modules = [];
    for_package = None;
    debug = false;
    use_threads = false;
    use_vmthreads = false;
    recursive_types = false;
    principal = false;
    transparent_modules = false;
    unboxed_types = false;
    unsafe_string = false;
    cookies = []
  }]

but since it contained nothing special, I just ignored it. But now, based on your pointer, I put this bit of text into the source file, and ran the same ocamlfind command from my previous post (that previously gave an error) and now it no longer gives an error.

So I’m concluding that the way that dune invokes PPX rewriters, is by first prepending this “context”, and then invoking the rewriters, in whatever order is specified. But ocamlfind doesn’t prepend this context, hence the differing behaviour.

I don’t know where to open a bug, or even whether anybody who maintains ppxlib cares.

I don’t use ppxlib or the standard PPX infrastructure: I develop and maintain a comprehensive PPX rewriter infrastructure based on Camlp5 (pa_ppx and a bunch of other packages), that is completely independent of ppxlib. The only reason I even found this discrepancy, is that I keep pa_ppx compatible with the standard PPX rewriters, by running the standard unit-tests, copied-over from the standard PPX rewriter sources. So I build and run those unit-tests, but since pa_ppx is entirely Makefile-based, obviously I bumped into this “interesting” bug.

One last thing: I see in that comment

(* Cannot use main_module_name from code_path because that contains .cppo suffix (via line directives), so it's actually not the module name. *)

that somehow if a file test.cppo.ml is preprocessed to produce test.ml, then the module-name will be derived from test.ml and not from test.cppo.ml. But this seems wrong: the convention almost everywhere I’ve ever worked with languages that support #line directives, is that the original source filename is what should be used whenever something needs the file-name (hence, the module-name).

[sim642]

@Chet_Murthy

One last thing: I see in that comment

(* Cannot use main_module_name from code_path because that contains .cppo suffix (via line directives), so it's actually not the module name. *)

that somehow if a file test.cppo.ml is preprocessed to produce test.ml, then the module-name will be derived from test.ml and not from test.cppo.ml. But this seems wrong: the convention almost everywhere I’ve ever worked with languages that support #line directives, is that the original source filename is what should be used whenever something needs the file-name (hence, the module-name).

What the comment refers to is that if your file was named foo.cppo.ml and preprocessed to foo.ml while having #line directives, then the buggy ppxlib logic would give main module name Foo.cppo, but that’s not a valid OCaml module name.
So constructor Foo with path would be printed as Foo.cppo.Foo. If I remember correctly, the old ppx_deriving based show plugin did not include .cppo in the path like that, so the workaround was to preserve that old behavior.

[Chet_Murthy]

I’m sorry, I was testing against the old (working as expected) version of ppx_deriving. OOPS! Sorry sorry. Please disregard my previous comment. But regarding what you wrote above, actually no, the way ppx_deriving.5.2.1 worked, is that if source file is named “foo.bar.buzz.ml”, then the “file path” in the generated code is "Foo.bar.buzz". Which is not a valid module-name, but … there it is.

ETA: here is an example:

1. file “foo.bar.buzz.ml”

open OUnit2

let printer = fun x -> x

type v = Foo | Bar of int * string | Baz of string [@@deriving show { with_path = true }]
let test_variant ctxt =
  assert_equal ~printer "Foo.Foo"                (show_v Foo);
  assert_equal ~printer "(Foo.Bar (1, \"foo\"))" (show_v (Bar (1, "foo")));
  assert_equal ~printer "(Foo.Baz \"foo\")"      (show_v (Baz "foo"))

let suite = "Test deriving(show)" >::: [
    "test_variant"         >:: test_variant
  ]

let _ = run_test_tt_main suite
;;
print_string (show_v Foo) ;;

2. command to preprocess and build:

ocamlfind ocamlc -g -custom -package ounit2,ppx_deriving.show -linkpkg -o foobad foo.bar.buzz.ml

3. output per “-dsource”

open OUnit2
let printer x = x
type v =
  | Foo 
  | Bar of int * string 
  | Baz of string [@@deriving show { with_path = true }]
let rec pp_v :
  Ppx_deriving_runtime.Format.formatter -> v -> Ppx_deriving_runtime.unit =
  ((let open! ((Ppx_deriving_runtime)[@ocaml.warning "-A"]) in
      fun fmt ->
        function
        | Foo ->
            Ppx_deriving_runtime.Format.pp_print_string fmt
              "Foo.bar.buzz.Foo"
        | Bar (a0, a1) ->
            (Ppx_deriving_runtime.Format.fprintf fmt
               "(@[<2>Foo.bar.buzz.Bar (@,";
             ((Ppx_deriving_runtime.Format.fprintf fmt "%d") a0;
              Ppx_deriving_runtime.Format.fprintf fmt ",@ ";
              (Ppx_deriving_runtime.Format.fprintf fmt "%S") a1);
             Ppx_deriving_runtime.Format.fprintf fmt "@,))@]")
        | Baz a0 ->
            (Ppx_deriving_runtime.Format.fprintf fmt
               "(@[<2>Foo.bar.buzz.Baz@ ";
             (Ppx_deriving_runtime.Format.fprintf fmt "%S") a0;
             Ppx_deriving_runtime.Format.fprintf fmt "@])"))
  [@ocaml.warning "-A"])
and show_v : v -> Ppx_deriving_runtime.string =
  fun x -> Ppx_deriving_runtime.Format.asprintf "%a" pp_v x[@@ocaml.warning
                                                             "-32"]
include struct let _ = fun (_ : v) -> () end[@@ocaml.doc "@inline"][@@merlin.hide
                                                                    ]
let test_variant ctxt =
  assert_equal ~printer "Foo.Foo" (show_v Foo);
  assert_equal ~printer "(Foo.Bar (1, \"foo\"))" (show_v (Bar (1, "foo")));
  assert_equal ~printer "(Foo.Baz \"foo\")" (show_v (Baz "foo"))
let suite = "Test deriving(show)" >::: ["test_variant" >:: test_variant]
let _ = run_test_tt_main suite

Notice in pp_v that the “show” of constructor Foo is Foo.bar.buzz.Foo.

ETA: oops, it’s too late here, I’m not thinking straight. Your comment and mine are compatible.

1. if a file “foo.bar.buzz.cppo.ml” is preprocessed to produce a source file “foo.bar.buzz.ml”
2. then ppx_deriving.show (v5.2.1) will produce code that uses a file_path “Foo.bar.buzz”. So it does indeed (as you say) ignore the #line directive and go straight to the source-file-name. But if that source-file-name isn’t a valid module-name … too bad! Ha! And this remains the case even if we use “-o yaddayadda.cmo” to output to a different CMO file. Of course, code that wants to use that module has to use the module name Yaddayadda.

[Chet_Murthy]

@sim642

What the comment refers to is that if your file was named foo.cppo.ml and preprocessed to foo.ml while having #line directives, then the buggy ppxlib logic would give main module name Foo.cppo, but that’s not a valid OCaml module name.

If I understand the reasoning here, it is that the module-name should be generated based upon the final and actual filename of the source file before PPX-rewriting starts. B/c if that final filename is a messy name “foo.bar.buzz.ml”, what happens? I see that dune doesn’t allow such filenames, so maybe that’s an answer.

But it seems like the correct module-name might be the module-name that is generated by whatever the final filename is for the “.cmo” file. So if there is a “-o yaddayadda.cmo”, then that is the file-name that is used. And if that isn’t present, then sure, it’s the final-and-actual filename of the source.

From the standpoint of someone who doesn’t use dune for builds, this seems very, very dune-centric. Which, sure, I guess whatever.

[sim642]

Indeed, this doesn’t really guarantee a valid module name in such general cases. Ignoring #line for show paths and using the actual input filename is what ppx_deriving did in the past and the new behavior should match that still. Whether deliberate or not, this heuristic seems to have been good enough with common build (system) practices.

I don’t know if ppx-s could even know/use the final and actual module name used for the object file. For example, I think dune decouples ppx preprocessing from the actual compilation and produces an intermediate .pp.ml file.

So a perfect system probably isn’t possible, but some conventions about names of input, preprocessed, ppx-ed and object files need to be assumed.

[NathanReb]

I’m a bit late to the party here, sorry about that.

@Chet_Murthy did you manage to get it working the way you’d like? I remember having issues with pa_ppx when releasing those packages. From what I gathered trying to debug it, ppxlib wasn’t able to determine the filename properly and therefore was returning an empty Code_path.t. It seems like ppxlib isn’t playing nice with ocamlfind in your setup. If the issue persists could you please report it upstream to either or both of those packages?

From what I can tell ppx_deriving and ppx_deriving_yojson behave as expected here and the bug comes from higher up.

[Chet_Murthy]

Yes, that’s pretty much what I found, but from the other end (trying to get pa_ppx ready for release). Again, I should make clear that the only reason I use ppxlib-derived packages is in unit-tests to compare their results with those from pa_ppx-based equivalents. So this behaviour, while annoying, isn’t an actual problem for me: I just adjust the unit-tests and keep going.

That said, I’ll report it upstream. I guess I should report to ppxlib ? Would that be the right place ?

[NathanReb]

Yes ppxlib would be the right place!

[Chet_Murthy]

One thing I wonder about, is whether building with ocamlfind is intended to be fully-supported. PPX rewriters are pretty important for development, and if the only supported build system is dune, that’s … not great. ocamlfind is a stand-in for all the other non-dune build-systems. So bazel probably would have the same problems (since it would have to use ocamlfind to deal with findlib packages).

I wonder what the actual thought is about this in the broader OCaml community, and the community esp. of ppxlib developers.

[yawaramin]

This sounds like it would be good to discuss in a new thread.

[orbitz]

I noticed a change in behaviour around strict in the latest ppx_deriving_yojson. I’m not sure if it is a “fix” or a “bug”.

The test case is:

type 'a value = 'a [@@deriving yojson]

type t_ = {foo: string} [@@deriving yojson {strict = false}]
and t = t_ value [@@deriving yojson]

And then the following works:

of_yojson (`Assoc [("foo", `String "bar")]);;
- : t Ppx_deriving_yojson_runtime.error_or = Ok {foo = "bar"}

And the following fails:

of_yojson (`Assoc [("foo", `String "bar"); ("baz", `String "baz")]);;
- : t Ppx_deriving_yojson_runtime.error_or = Error "t_"

This worked on previous versions of ppx_deriving_yojson. The fix appears to be:

type t_ = {foo: string} [@@deriving yojson {strict = false}]
and t = t_ value [@@deriving yojson {strict=false}]

I was a bit surprised at this. My actual use case involves a hand written of_yojson for the value type, and I was a surprised to see the strict context seems to be passed around in the of_yojson function that gets passed into it.