(Non-protocol-specific) binary serialization library?

Learning
1

I’ve been working on several libraries that need to send and receive packets in specific binary formats (PGX, TDS). Right now they both have extremely-low-level code to convert ints to bytes with the right endianness, but it seems excessively low level to put in every library I write that deals with a binary protocol.

I know there are a bunch of libraries for parsing text, or for working with defined protocols like msgpack, but is there anything for arbitrary binary encoding? I’m thinking something like struct in Python, although I’m not married to that particular implementation.

(As a side note: Another reason I’d like something like this is that all of my current implementations handle one byte at a time, and it would be nice if we could do this more efficiently).

2

My current implementation of this in TDS involves adding a fold_bytes function to every type, like:

let fold_bytes ~init ~f t order =
  order
  |> List.fold_left ~init ~f:(fun acc shift_bytes ->
    shift_right t (shift_bytes * 8)
    |> Uint8.of_uint32
    |> Uint8.fold_bytes ~f ~init:acc)

let fold_bytes_be ~init ~f t =
  [ 3 ; 2 ; 1 ; 0 ]
  |> fold_bytes ~init ~f t

let fold_bytes_le ~init ~f t =
  [ 0 ; 1 ; 2 ; 3 ]
  |> fold_bytes ~init ~f t

Although this makes things readable at the expense of most-likely being even slower than the usual implementations. Having something somwhat readable and also not horribly slow would be nice.

3

I used to use bitstring for such things. But it’s a long time since i last had a look at it. I found it easy to use.

2 Likes
4

Ooo nice. That looks like exactly what I was looking for:

https://bitstring.software/examples/

1 Like
5

You can also check out angstrom which provides a nice parsing API and cstruct which includes a ppx somewhat similar to bitstring.

1 Like
6

The usual companion to the angstrom parser combinator library is faraday binary serialization library. These libraries are both fairly mature and they can work in combination or separately. Alas, we don’t have anything quite as comprehensive as serde in the Rust language world.

3 Likes