# (Non-protocol-specific) binary serialization library?

**URL:** <https://discuss.ocaml.org/t/non-protocol-specific-binary-serialization-library/2768>\
**Category:** Learning\
**Created:** [October 24, 2018, 6:37pm UTC](https://discuss.ocaml.org/t/non-protocol-specific-binary-serialization-library/2768 "2018-10-24T18:37:32Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![BrendanLong](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/brendanlong/32/522_2.png) [@BrendanLong](https://discuss.ocaml.org/u/BrendanLong)\
**Post date:** [October 24, 2018, 6:37pm UTC](https://discuss.ocaml.org/t/non-protocol-specific-binary-serialization-library/2768/1 "2018-10-24T18:37:33Z")

</div>

I’ve been working on several libraries that need to send and receive packets in specific binary formats ([PGX](https://github.com/arenadotio/pgx), [TDS](https://github.com/brendanlong/ocaml-tds)). Right now they both have extremely-low-level code to convert ints to bytes with the right endianness, but it seems excessively low level to put in every library I write that deals with a binary protocol.

I know there are a bunch of libraries for parsing text, or for working with defined protocols like msgpack, but is there anything for arbitrary binary encoding? I’m thinking something like [struct](https://docs.python.org/3/library/struct.html) in Python, although I’m not married to that particular implementation.

(As a side note: Another reason I’d like something like this is that all of my current implementations handle one byte at a time, and it would be nice if we could do this more efficiently).

---

<div class="post-metadata">

**Author:** ![BrendanLong](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/brendanlong/32/522_2.png) [@BrendanLong](https://discuss.ocaml.org/u/BrendanLong)\
**Post date:** [October 24, 2018, 6:39pm UTC](https://discuss.ocaml.org/t/non-protocol-specific-binary-serialization-library/2768/2 "2018-10-24T18:39:34Z")

</div>

My current implementation of this in TDS involves adding a `fold_bytes` function to every type, like:

> <https://github.com/brendanlong/ocaml-tds/blob/master/src/uint32.ml>

```auto
let fold_bytes ~init ~f t order =
  order
  |> List.fold_left ~init ~f:(fun acc shift_bytes ->
    shift_right t (shift_bytes * 8)
    |> Uint8.of_uint32
    |> Uint8.fold_bytes ~f ~init:acc)

let fold_bytes_be ~init ~f t =
  [3 ; 2 ; 1 ; 0]
  |> fold_bytes ~init ~f t

let fold_bytes_le ~init ~f t =
  [0 ; 1 ; 2 ; 3]
  |> fold_bytes ~init ~f t

```

Although this makes things readable at the expense of most-likely being even slower than the usual implementations. Having something somwhat readable and also not horribly slow would be nice.

---

<div class="post-metadata">

**Author:** ![ttamttam](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/ttamttam/32/132_2.png) [@ttamttam](https://discuss.ocaml.org/u/ttamttam)\
**Post date:** [October 24, 2018, 6:53pm UTC](https://discuss.ocaml.org/t/non-protocol-specific-binary-serialization-library/2768/3 "2018-10-24T18:53:52Z")

</div>

I used to use bitstring for such things. But it’s a long time since i last had a look at it. I found it easy to use.

---

<div class="post-metadata">

**Author:** ![BrendanLong](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/brendanlong/32/522_2.png) [@BrendanLong](https://discuss.ocaml.org/u/BrendanLong)\
**Post date:** [October 24, 2018, 7:05pm UTC](https://discuss.ocaml.org/t/non-protocol-specific-binary-serialization-library/2768/4 "2018-10-24T19:05:51Z")

</div>

Ooo nice. That looks like exactly what I was looking for:

[https://bitstring.software/examples/](https://bitstring.software/examples/)

---

<div class="post-metadata">

**Author:** ![hcarty](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/hcarty/32/1824_2.png) [@hcarty](https://discuss.ocaml.org/u/hcarty)\
**Post date:** [October 24, 2018, 9:29pm UTC](https://discuss.ocaml.org/t/non-protocol-specific-binary-serialization-library/2768/5 "2018-10-24T21:29:07Z")

</div>

You can also check out [angstrom](https://github.com/inhabitedtype/angstrom) which provides a nice parsing API and [cstruct](https://github.com/mirage/ocaml-cstruct) which includes a ppx somewhat similar to bitstring.

---

<div class="post-metadata">

**Author:** ![jhw](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/jhw/32/1639_2.png) [@jhw](https://discuss.ocaml.org/u/jhw)\
**Post date:** [October 25, 2018, 6:28am UTC](https://discuss.ocaml.org/t/non-protocol-specific-binary-serialization-library/2768/6 "2018-10-25T06:28:09Z")

</div>

The usual companion to the [angstrom](https://opam.ocaml.org/packages/angstrom/) parser combinator library is [faraday](https://opam.ocaml.org/packages/faraday/) binary serialization library. These libraries are both fairly mature and they can work in combination or separately. Alas, we don’t have anything quite as comprehensive as [serde](https://serde.rs) in the Rust language world.