How do we use `bin_prot` with mutually recursive modules?

Ecosystem
1

Consider the following snippet

open Bin_prot.Std

module rec A: sig
  type t = C of string
          | D of B.t [@@deriving bin_io]
end = struct
  type t = C of string
         | D of B.t [@@deriving bin_io]
end and B: sig
    type t = E of string
           | F of B.t [@@deriving bin_io]
end = struct
    type t = E of string
           | F of B.t [@@deriving bin_io]
end

Attempting to build it fails with an error

       Error: Cannot safely evaluate the definition of the following cycle
       of recursively-defined modules: B -> B. There are no safe modules
       in this cycle (see manual section 10.2).

Right. Digging a bit deeper, I can reproduce the error just with [@@deriving bin_shape] or [@@deriving bin_read] or [@@deriving bin_write].

I understand why bin_shapecauses this error (there’s obviously a conflict in evaluation order), not so sure why bin_read or bin_write do.

So, what should I do? I suppose I could hand-roll the (de)serializers for my modules (in my actual code, I have 3 mutually recursive modules and a dozen types), but this doesn’t seem very optimal.

2

You can use dune show pp bla/foo.ml to see what bin_read and bin_write generate. Here, it’s two generated functions and a pair so they can referred to as one value, and that’s sadly enough to break the safe module recursion criteria.

One option is to not expose these pairs:

module type Min_binable = sig
  type t
  val bin_size_t : t Bin_prot.Size.sizer
  val bin_write_t : t Bin_prot.Write.writer
  val bin_read_t : t Bin_prot.Read.reader
  val __bin_read_t__ : (int -> t) Bin_prot.Read.reader
end

module rec A: sig
  type t = C of string
         | D of B.t
  include Min_binable with type t := t
end = struct
  type t = C of string
         | D of B.t [@@deriving bin_read, bin_write]
end and B: sig
    type t = E of string
           | F of A.t
  include Min_binable with type t := t
end = struct
    type t = E of string
           | F of A.t [@@deriving bin_read, bin_write]
end

You won’t have bin_shapes this way, which is usually fine. The main issue would be if you need to pass your types into apis that require type t [@@deriving bin_io]. If need be, you could work around that by creating fresh uninformative shapes. Not sure what’s the function for that, but you could start with whatever type t [@@deriving bin_io] does.

Otherwise, you could define the types as a single mutually recursive group, then reexport them in their own modules. That requires writing each type twice, but your code sample already does that, so no difference.

type a = C of string
       | D of b
and b = E of string
      | F of a
[@@deriving bin_io]

module A = struct
  type t = a = C of string | D of b [@@deriving bin_io]
end
module B = ...

(or you could have only type t = a [@@deriving bin_io] to avoid repeating constructors. You wouldn’t be able to refer to constructors as A.C, but type directed disambiguation would still work: let x : A.t = C "")

Finally, in some cases (I suspect not here), it’s possible to define the mutually recursive group first with as few functions as possible, and then define modules one by one, like sexp_of here and here. It may only work when the serialization code is not recursive, even though the type definitions are recursive (and in fact, the values are cyclic).

1 Like
3 
type a = C of string
       | D of B.t
and b = E of string
      | F of A.t
[@@deriving bin_io]

module A = struct
  type t = a = C of string | D of b [@@deriving bin_io]
end
module B = ...

How would that work? a refers to A, so it cannot be defined before A, right?

4

That’s just a typo, I corrected the example. a shouldn’t refer to A.t, only to a.

1 Like
5

Makes sense, thanks.

I have indeed tried that, but just to make things fun, one of my mutually recursive modules actually looks like

and D: sig … end = struct
  include Set.Make(struct
    type t = C.t
    let compare = …
  end)
end

As far as I understand, this cannot be expressed at all without mutually recursive modules, right?

And of course, I’ve just realized that I actually need a working implementation of bin_shape to ensure type-safe deserialization of serialized data.

Do you think I’m doomed?

6

The bin_io code is type safe no matter what. bin_shape only makes it possible to detect some misunderstandings like one program sending a value of type { dont_check_x : bool } while another one reads { check_x : bool }, which would “succeed” since field names are not serialized. But I have never seen a bug like this (I have never seen anyone use bin_shape in this way either).

I think my suggestion with Min_binable would still work if you have to use recursive modules? (or forking ppx_bin_prot to add a [@@deriving bin_read_fun] that generates the read/write functions without the reader/writer records, but that doesn’t seem more powerful).

I think otherwise, you’d have to do something annoying like defining the types first, then the bin io. Something like:

type a = A of b and b = B of a (* although with recursive modules here *)
type a2 = a = A of b2 and b2 = b = B of a2 [@@deriving bin_io]

I think it might be necessary to define a placeholder bin_io between the two groups of type definitions, it’s unclear without trying, in which case it’d look more like:

type a = A of b and b = B of a (* although with recursive modules here *)

let bin_write_b_ref = ref (fun _ -> assert false)
let bin_write_b ... = !bin_write_b_ref  ...
(* etc for bin_size_b and bin_read_b *)

type a2 = a = A of b and b2 = b = B of a2 [@@deriving bin_io]

let () = bin_write_b_ref := bin_write_b2 (* etc *)
1 Like
7

I’d argue that this is a form of type-unsafety :slight_smile: , since you can easily use this to encode a variant of Obj.magic.

I’m coming to the conclusion that I’m diving a bit too deep into yak shaving. I’ll try and find another way to ensure this type-safety.

Thanks for all your answers!