Ppx_deriving_yojson : nested `Assoc

EliasGit2017 · May 7, 2023, 3:11pm

Hi everyone,
I am currently trying to serialize/deserialize Json values using ppx_deriving_yojson.
my json looks like this :

type t = {
    id: string [@key "Id"];
    names: string list [@key "Names"];
    image: string [@key "Image"];
    image_id: string [@key "ImageID"];
    command: string [@key "Command"];
    created: int64 [@key "Created"];
    ports: Port.t list [@key "Ports"];
    size_rw: int64 option [@default None] [@key "SizeRw"];
    size_root_fs: int64 option [@default None] [@key "SizeRootFs"];
    labels: (string * string) option [@default None] [@key "Labels"];
    state: string [@key "State"];
    status: string [@key "Status"];
    host_config: Container_summary_host_config.t option [@default None] (* [@key "HostConfig"] *);
    network_settings: Container_summary_network_settings.t option [@default None] (* [@key "NetworkSettings"] *);
    mounts: Mount_point.t option list [@default []] (* [@key "Mounts"] *);
} [@@deriving yojson { strict = false }, show ];;

The problem here is with the labels key which is a json object too (an Assoc of Yojson.Safe).
How can I serialize / unserialize the whole type t knowing that labels is a simple key-value pairs where the key is user defined and can take different values ?

Frederic_Loyer · May 7, 2023, 3:38pm

I have tried to compile and run your examples (minus some field where I have no detail of the actual type, like host_config).

And it works well. The following sentence:

let () = print_string @@ Yojson.Safe.to_string @@ to_yojson a

prints:

{"Id":"aa","Names":["ff"],"Image":"","ImageID":"ee","Command":"ee","Created":8,"Labels":["a","b"],"State":"","Status":""

Then the labels are correctly serialized. What do you expect more?

EliasGit2017 · May 7, 2023, 4:04pm

To give more details : I can post the json which I am trying to unserialize
Here is the output from utop where i am testing it

Openapi.Container_summary.of_yojson (Yojson.Safe.from_string {|{
    "Id": "ad1ad195fe314991b364e124573eb9240093a2718586932789086f9c960cce88",
    "Names": [
      "/dazzling_goldberg"
    ],
    "Image": "alpine",
    "ImageID": "sha256:b2aa39c304c27b96c1fef0c06bee651ac9241d49c4fe34381cab8453f9a89c7d",
    "Command": "/bin/sh",
    "Created": 1683414855,
    "Ports": [],
    "Labels": {
      "this_is_a_label": "good_otter",
      "another_label" : "nice_cat"
    },
    "State": "exited",
    "Status": "Exited (0) 15 hours ago",
    "HostConfig": {
      "NetworkMode": "default"
    },
    "NetworkSettings": {
      "Networks": {
        "bridge": {
          "IPAMConfig": null,
          "Links": null,
          "Aliases": null,
          "NetworkID": "9b8d4180e3a28d08cd055ef460dfdb369e5f669795068508b2f7483d513fd100",
          "EndpointID": "",
          "Gateway": "",
          "IPAddress": "",
          "IPPrefixLen": 0,
          "IPv6Gateway": "",
          "GlobalIPv6Address": "",
          "GlobalIPv6PrefixLen": 0,
          "MacAddress": "",
          "DriverOpts": null
        }
      }
    },
    "Mounts": [
      {
        "Type": "bind",
        "Source": "/home",
        "Destination": "/home/elias/Desktop/camelwhale/camelwhale",
        "Mode": "",
        "RW": true,
        "Propagation": "rprivate"
      }
    ]
  }|});;
: Openapi.Container_summary.t Ppx_deriving_yojson_runtime.error_or =
Result.Error "Container_summary.t.labels"

As you can see, the labels field is causing some issues
PS : The json is the result of a simple docker container list. I am trying to build an SDK for the Docker Engine deamon.

geoffder · May 7, 2023, 4:12pm

In your posted code, labels has type (string * string) option (not a list).

Frederic_Loyer · May 7, 2023, 4:24pm

Note also that

("a","b")

(which should be writed [ "a", "b" ] in JSON)

Is not a synonymous to

{ a="b" }

Then you can also have some issue when match the last one with a (string*string) option. And like goeffer said, an option is not a list. You can’t twist the OCaml typing system to give labels two pairs.

Leonidas · May 8, 2023, 12:39pm

Also, ppx_deriving_yojson doesn’t really handle maps where the keys are variable too. So you’d need to specify the type as Yojson.Safe.t which will give you the object as-is, and from there you can process it.

Alternatively declare it as custom type labels and implement labels_of_yojson which will parse an Yojson.Safe.t into your labels type, e.g. some kind of Map.t with string keys.

Chet_Murthy · May 8, 2023, 8:43pm

You can also implement a type of “maps with string keys,” viz.

type 'a _stringmap = (string * 'a) list[@@deriving yojson { strict = false }, show ]
type 'a stringmap = (string * 'a) list[@@deriving show ]

let string_to_yojson j = Pa_ppx_runtime.Runtime.Yojson.string_to_yojson j

let stringmap_of_yojson sub1 (j : Yojson.Safe.t) = match j with
    `Assoc l -> _stringmap_of_yojson sub1 (`List (List.map (fun (s,v) -> `List [string_to_yojson s; v]) l))

[NOTE that the functions to/from string above are from `pa_ppx`, so probably will need to be recoded for ppx_deriving_yojson`]

Then you can modify your type to

type t = {
    id: string [@key "Id"];
    names: string list [@key "Names"];
    image: string [@key "Image"];
    image_id: string [@key "ImageID"];
    command: string [@key "Command"];
    created: int64 [@key "Created"];
    ports: Port.t list [@key "Ports"];
    size_rw: int64 option [@default None] [@key "SizeRw"];
    size_root_fs: int64 option [@default None] [@key "SizeRootFs"];
    labels: string stringmap [@default []] [@key "Labels"];
    state: string [@key "State"];
    status: string [@key "Status"];
    host_config: Container_summary_host_config.t option [@default None] (* [@key "HostConfig"] *);
    network_settings: Container_summary_network_settings.t option [@default None] (* [@key "NetworkSettings"] *);
    mounts: Mount_point.t option list [@default []] (* [@key "Mounts"] *);
  } [@@deriving of_yojson { strict = false }, show ];;

and your JSON input works unmodified.

Notice the type of labels is now string stringmap.

P.S. I only implemented the direction “yojson → t”, not “t → yojson”. The second direction should be straightforward.

EliasGit2017 · May 8, 2023, 10:09pm

Thank you, this looks promising, I was going for a something like (as proposed by @Leonidas ):

module StringMap = Map.Make(struct type t = string let compare = compare end);;
let yojson_of_stringmap m = StringMap.bindings m |> [%to_yojson: (string * string) list];;

And was working on the %of_yojson part.
But your solution seems to fit better. I am trying to implement it but I get a

 dune utop .                                                                                                      ─╯
File "_none_", line 1:                  
Error: Module `Pa_ppx_runtime is unavailable (required by `Openapi__Container_summary')

even though I did add pa_ppx to my dune file.
Any idea why this is happening ?

Chet_Murthy · May 9, 2023, 12:47am

Ah.

Don’t use pa_ppx. Continue to use the ppx_deriving (or whatever it was) that you were using before. pa_ppx is part of a suite of PPX derivers based on Camlp5, and they are incompatible (in the sense of don’t interoperate) with the standard ppxlib-based infrastructure. They provide the same function, but … well, they’re not standard.
When I get to a stable location, I’ll write the “of_yojson” code. It’s not hard.

EliasGit2017 · May 9, 2023, 12:49am

Thank you, it really helps me a lot as I am trying to build a Docker Engine SDK for Ocaml and I might encounter this problem again

Chet_Murthy · May 9, 2023, 1:43am

OK, here’s a new set of map stuff:


type ('a,'b) _map = ('a * 'b) list[@@deriving yojson { strict = false }]
type 'b stringmap = (string * 'b) list[@@deriving show ]

let string_to_yojson j : Yojson.Safe.t = Pa_ppx_runtime.Runtime.Yojson.string_to_yojson j
let string_of_yojson msg j = Pa_ppx_runtime.Runtime.Yojson.string_of_yojson msg j

type maprow = string * Yojson.Safe.t[@@deriving yojson { strict = false } ]

let stringmap_of_yojson sub1 (j : Yojson.Safe.t) = match j with
    `Assoc l -> _map_of_yojson (string_of_yojson "string_of_yojson") sub1 (`List (List.map maprow_to_yojson l))

let stringmap_to_yojson sub1 (l : 'a stringmap) : Yojson.Safe.t =
  let l = List.map (fun (k,v) -> (k, sub1 v)) l in
  `Assoc l

BTW, you have other type errors, e.g. in Container_summary_network_settings. The “networks” field is not a list of Network.t objects. There’s too little data for me to tell what you want to do, but perhaps you want a map from string to Network.t (where “bridge” is a key).

EliasGit2017 · May 9, 2023, 2:00am

Thank you for your help,
I do need to go deeper and debug all the nested objects for many json outputs (sent by the docker deamon). “bridge” will be a key containing

"NetworkSettings": {
    "Networks": 
{
    "bridge": 
    {
        "NetworkID": "7ea29fc1412292a2d7bba362f9253545fecdfa8ce9a6e37dd10ba8bee7129812",
        "EndpointID": "88eaed7b37b38c2a3f0c4bc796494fdf51b270c2d22656412a2ca5d559a64d7a",
        "Gateway": "172.17.0.1",
        "IPAddress": "172.17.0.8",
        "IPPrefixLen": 16,
        "IPv6Gateway": "",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "MacAddress": "02:42:ac:11:00:08"
    }
}

To my knowledge, this is the generic output containing all the description of a docker network.
I will look into it as soon as I manage to get the labels working. Your solution fits well, I just need to fix the pa_ppx_runtime issue by using something compatible with ocaml 5.0 . The goal is to have a complete Docker Engine SDK for Ocaml and give users the ability to manage the deamon locally (swarm mode and remote access is another “story”).

Chet_Murthy · May 9, 2023, 2:11am

There should be equivalent runtime entrypoints available for ppx_deriving_yojson: when I wrote my pa_ppx version, I mimicked the former as much as possible – that is to say, the only difference is how the two PPX rewriters are coded, not so much the code they generate.

P.S. A sugggestion: it is very much worth using -dsource to inspect the output of the PPX rewriter ppx_deriving_yojson to see what the code looks like. For instance, to see what code it generates for the type string.

Topic		Replies	Views
Deserialize a numeric JSON field into a variant using ppx_deriving_yojson Learning ppx_deriving_yojson	7	1012	September 7, 2022
Using Map and ppx yojson Learning ppx , ppx_deriving , ppx_deriving_yojson	12	881	July 5, 2023
Serializing a record to JSON with ppx_yojson_conv? Learning	8	1517	October 29, 2021
[ANN] first release of ppx_deriving_jsonschema Community announce	1	221	September 26, 2024
[ANN] ppx_yojson: a ppx extension for Yojson literals and patterns Community ppx , json , yojson	2	1418	October 4, 2018

Ppx_deriving_yojson : nested `Assoc

Related Topics