# Libraries for parsing binary formats?

**URL:** <https://discuss.ocaml.org/t/libraries-for-parsing-binary-formats/4004>\
**Category:** Ecosystem\
**Created:** [June 30, 2019, 9:20am UTC](https://discuss.ocaml.org/t/libraries-for-parsing-binary-formats/4004 "2019-06-30T09:20:48Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![lindig](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/lindig/32/532_2.png) [@lindig](https://discuss.ocaml.org/u/lindig)\
**Post date:** [June 30, 2019, 9:20am UTC](https://discuss.ocaml.org/t/libraries-for-parsing-binary-formats/4004/1 "2019-06-30T09:20:48Z")

</div>

What is a good way to parse binary file formats with OCaml? I’ve seen both [bitstring](https://bitstring.software) and [cstruct](https://github.com/mirage/ocaml-cstruct) but assume there are more. Typical tasks involve decoding bit fields, reading a length field in order to know how much to read next, reading 8, 16, 32, and 64-bit values. In addition to libraries I would be interested in projects that use them.

---

<div class="post-metadata">

**Author:** ![XVilka](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/xvilka/32/5334_2.png) [@XVilka](https://discuss.ocaml.org/u/XVilka)\
**Post date:** [June 30, 2019, 12:59pm UTC](https://discuss.ocaml.org/t/libraries-for-parsing-binary-formats/4004/2 "2019-06-30T12:59:26Z")

</div>

I use cstruct, with some patches (for now) for supporting bitfields: [https://github.com/mirage/ocaml-cstruct/pull/215](https://github.com/mirage/ocaml-cstruct/pull/215)

I should note it is still WIP and very brittle.

---

<div class="post-metadata">

**Author:** ![nojb](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/nojb/32/519_2.png) [@nojb](https://discuss.ocaml.org/u/nojb)\
**Post date:** [June 30, 2019, 6:34pm UTC](https://discuss.ocaml.org/t/libraries-for-parsing-binary-formats/4004/3 "2019-06-30T18:34:31Z")

</div>

You can also use something like [https://github.com/OCamlPro/ocplib-endian](https://github.com/OCamlPro/ocplib-endian) if you want to keep it simple.

---

<div class="post-metadata">

**Author:** ![Chet\_Murthy](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/chet_murthy/32/1501_2.png) [@Chet\_Murthy](https://discuss.ocaml.org/u/Chet_Murthy)\
**Post date:** [June 30, 2019, 8:02pm UTC](https://discuss.ocaml.org/t/libraries-for-parsing-binary-formats/4004/4 "2019-06-30T20:02:06Z")

</div>

[trigger warning: camlp5 ahead] This is going to be about de/marshaling binary **wire formats** and not about binary data in-memory. I no longer have access to the code, so this comment is at best a “trip report”. Perhaps not so useful.

I’ve had to do this a few times: decoding ethernet/IP/TCP/CORBA packets and payloads (needed programmatic dissectors that could do statistical analyses of response-times, and hadn’t found the tools which capture/dissect TCP and dump into files for further analysis – to diagnose bugs in multi-tier server-side applications). Another time, I wanted to decode/re-encode Java classfiles (to perform aspect-oriented injection of code into existing Java classfiles). There have been others. I find that the stream-based parsers-printers of camlp5 are a -great- tool.

Almost all binary wire-formats that I’ve dealt with turn out to be recursive-descent, and so writing de/marshallers for them with stream (LL(1)) parsers/printers is a doddle.

(1) of course, stuff like “read a length-count”, demarshal N of some type" is trivial

(2) eventually you might want to (re-)marshal. One of the nice things about camlp5 is that pretty-printers look superficially like parsers, which makes it much easier to keep the marshaller in sync with the parser.

(3) Eventually one thing jumps out, that doesn’t fit into stream-parsers: “skip forward to an N-byte alignment boundary”. So I re-implemented -only- those parts of the Stream module that were used by generated parsers, with a`Stream.t` type implemented using a buffer, viz.

```auto
module Stream = struct

  type 'a t = {mutable buf : 'a array;
	       mutable buf_pos : int;
	       mutable buf_bound : int;
	       mutable abs_ofs : int;
	       eof : 'a ;
	       filbuf : 'a t -> int}

```

and then one can write functions that return the absolute position in the “stream”, etc.

(4) of course, switching to this “flat” representation of the parsing-buffer also improves efficiency dramatically.

This replacement of `Stream.t` with something custom pretty much corresponds to “write a hacked lexer with interesting behaviour” that we do all the time when it comes to writing parsers, of course.

Unfortunately, I wrote all this code for IBM back in 2001, and they had pretty unenlightened policies about open-source. All lost in the sands of time.

---

<div class="post-metadata">

**Author:** ![perry](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/perry/32/518_2.png) [@perry](https://discuss.ocaml.org/u/perry)\
**Post date:** [July 1, 2019, 12:44am UTC](https://discuss.ocaml.org/t/libraries-for-parsing-binary-formats/4004/5 "2019-07-01T00:44:43Z")

</div>

Is it possible to liberate the interesting parser tools from camlp5 so that they’re more standalone?

---

<div class="post-metadata">

**Author:** ![Chet\_Murthy](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/chet_murthy/32/1501_2.png) [@Chet\_Murthy](https://discuss.ocaml.org/u/Chet_Murthy)\
**Post date:** [July 1, 2019, 12:48am UTC](https://discuss.ocaml.org/t/libraries-for-parsing-binary-formats/4004/6 "2019-07-01T00:48:30Z")

</div>

not really. Camlp5 is a system for building PreProcessors and PrettyPrinters (used to be called camlp4 for obvs. reasons), and before it was written, DDR wrote stream-parsers/printers as a preprocessor for caml-light. He wrote camlp4 (used to be called chamau) as a generalization of the idea, and the stream-grammar-extension was (IIRC) the first implemented use of chamau. That was all 25 years ago. At this point, if you want stream-{parsers,printers} you need to use camlp{4,5} to get 'em.

But I routinely use camlp{4,5} on parts of my code for a project, and PPX extensions in other parts. They’re incompatible at the source-code level, but there’s no reason you can’t mix 'em as binaries.

---

<div class="post-metadata">

**Author:** ![XVilka](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/xvilka/32/5334_2.png) [@XVilka](https://discuss.ocaml.org/u/XVilka)\
**Post date:** [July 1, 2019, 2:16am UTC](https://discuss.ocaml.org/t/libraries-for-parsing-binary-formats/4004/7 "2019-07-01T02:16:11Z")

</div>

There is also [Angstrom](https://github.com/inhabitedtype/angstrom), but I don’t know if it is possible to make it work on non-byte granularity, as in a sequence of bits.

---

<div class="post-metadata">

**Author:** ![mseri](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/mseri/32/91_2.png) [@mseri](https://discuss.ocaml.org/u/mseri)\
**Post date:** [July 1, 2019, 6:50am UTC](https://discuss.ocaml.org/t/libraries-for-parsing-binary-formats/4004/8 "2019-07-01T06:50:52Z")

</div>

I remember reading about using bitstream not (too) long ago. I have just found the link: [https://andreas.github.io/2014/08/22/implementing-the-binary-memcached-protocol-with-ocaml-and-bitstring](https://andreas.github.io/2014/08/22/implementing-the-binary-memcached-protocol-with-ocaml-and-bitstring)

There is also [parsifal](https://github.com/picty/parsifal). It seems maintained but I have no experience with it (and I don’t know if there is a usable library part or it only provides the binaries). I only remember vaguely the papers: [http://spw14.langsec.org/papers/pasifal-report.pdf](http://spw14.langsec.org/papers/pasifal-report.pdf) and [https://www.ieee-security.org/TC/SPW2014/papers/5103a191.PDF](https://www.ieee-security.org/TC/SPW2014/papers/5103a191.PDF)

---

<div class="post-metadata">

**Author:** ![lindig](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/lindig/32/532_2.png) [@lindig](https://discuss.ocaml.org/u/lindig)\
**Post date:** [July 1, 2019, 8:45am UTC](https://discuss.ocaml.org/t/libraries-for-parsing-binary-formats/4004/9 "2019-07-01T08:45:24Z")

</div>

Thanks for all the recommendation. So far, I like [bitstring](https://bitstring.software/) and I wonder why it is not more widely used. I believe a bitstring-based parser works best in a monadic style that chains individual parsers together. These combinators are not part of bitstring, but they are easy to implement.

---

<div class="post-metadata">

**Author:** ![jhw](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/jhw/32/1639_2.png) [@jhw](https://discuss.ocaml.org/u/jhw)\
**Post date:** [July 2, 2019, 5:04pm UTC](https://discuss.ocaml.org/t/libraries-for-parsing-binary-formats/4004/10 "2019-07-02T17:04:17Z")

</div>

> [@lindig](#):
>
> Typical tasks involve decoding bit fields, reading a length field in order to know how much to read next, reading 8, 16, 32, and 64-bit values.

This is handled in the [`Cf_decoder`](https://bitbucket.org/jhw/orsetto/src/default/src/cf/cf_decode.mli) module in my [Orsetto](https://opam.ocaml.org/packages/orsetto/) project. Its CBOR decoder uses it.