ATD - static types for json APIs, v1.13.0 | new documentation home

We’ve made a lot of internal changes to the ATD suite of tools, including atdgen. Atdgen is a full-featured program that takes type definitions and derives OCaml parsers and serializers for json. For example,

type hello = {
  greeting : string;
  ?fruit : fruit option;
}

type fruit = [ Peach | Cherry ]

results in two OCaml functions of the following types:

val hello_of_string : string -> hello
val string_of_hello : hello -> string

The purpose of this announcement is primarily to bring the attention of the community to the existence of this tool, which most people should be using instead of straight yojson (the parser and AST-handling library on which atdgen relies).

This is a stable release available from opam, and the changes are mostly internal. Different tools relying on atd have been consolidated into one repo, efforts are ongoing to make the codebase more accessible, and new, exciting features are being developed!

$ opam install atdgen

Check out the documentation’s new home!

Martin, on behalf of the ATD team.

13 Likes

The new documentation looks great :slight_smile:

In my scant use of JSON for OCaml stuff so far, I’ve primarily relied on ppx_deriving_yojson. Are there use cases for which atdgen is inherently better suited, or is it mainly just a matter of methodological preferences (e.g., of command line tools and separate file formats over inline annotations)?

2 Likes

Atdgen is geared toward interoperability between applications.

The main features that distinguish it from its camlp4-based predecessor json-static and from ppx_deriving_yojson are:

  • The specification of the ATD language is not tied to the OCaml language.
  • Generated code is plain text that’s easy for the user to inspect.
  • Direct parsing/serializing without going through an AST.

Historically, atdgen was created out of a desire to not depend on the “new camlp4” released around 2007, which suffered many undocumented changes as well as unsatisfying build-time performance. Atdgen only needed a small fraction of the features of camlp4 anyway. Atdgen was developed at that time with a focus of mapping type definitions to various languages.

4 Likes

Thanks for the very helpful explanation!

Would this explanation be a good addition somewhere near the top of the atd docs? I guess part of it is there but maybe not all?

1 Like

Absolutely. We still need to migrate some of the old docs to the new location (task #109). After that we’ll need to write a good intro for the whole site (task #118).

1 Like

Also, it is possible to get more info about:

For a variety of good reasons JSON’s null value may not be used to indicate that a field is undefined.

In the cases I use atdgen for, this is mostly just confusing (integrating with pre-existing API’s, and have to be careful in reviews that no one uses option and always uses nullable), so I’m curious what the reasons for this are.

This quote is a little disturbing, even in context (it’s from the atdgen tutorial). What happens is that object fields whose value is null are simply ignored.

The tutorial’s example is:

type vector_v3 = {
  ~x: int;
  ~y: int;
  ?z: int option;
}

The relevant section of the generated OCaml code is this:

    match i with
      | 0 ->
        if not (Yojson.Safe.read_null_if_possible p lb) then (
          field_x := (
            (
              Atdgen_runtime.Oj_run.read_int
            ) p lb
          );
        )
      | 1 ->
        if not (Yojson.Safe.read_null_if_possible p lb) then (
          field_y := (
            (
              Atdgen_runtime.Oj_run.read_int
            ) p lb
          );
        )
      | 2 ->
        if not (Yojson.Safe.read_null_if_possible p lb) then (
          field_z := (
            Some (
              (
                Atdgen_runtime.Oj_run.read_int
              ) p lb
            )
          );
        )
      | _ -> (
          Yojson.Safe.skip_json p lb
        )

We can see that for both the optional fields with a default (x and y) and for optional fields without a default (z), the first thing we do is read and ignore any null value that may be there.

The documentation definitely needs some fixing. Thanks for reporting this.

Atdgen’s reference manual is correct:

Null JSON fields by default are treated as if the field was missing. They can be made meaningful with the keep_nulls flag.

The tutorial is and has always been wrong on that point.

1 Like

It’s more interesting than this: based on the commit history, early versions of atdgen did indeed treat null fields differently than missing fields. This was written to the tutorial on April 25, 2011. The next day though, the behavior was changed in atdgen but the tutorial didn’t get corrected. It’s only in 2017 that the <json keep_nulls> annotation was added, allowing fields with null values to not be discarded.

2 Likes