# \[ANN\] First release of data-encoding, JSON and binary serialisation **URL:** \ **Category:** Community\ **Created:** [September 27, 2019, 7:46am UTC](https://discuss.ocaml.org/t/ann-first-release-of-data-encoding-json-and-binary-serialisation/4444 "2019-09-27T07:46:49Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![raphael-proust](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/raphael-proust/32/415_2.png) [@raphael-proust](https://discuss.ocaml.org/u/raphael-proust)\ **Post date:** [September 27, 2019, 7:46am UTC](https://discuss.ocaml.org/t/ann-first-release-of-data-encoding-json-and-binary-serialisation/4444/1 "2019-09-27T07:46:49Z")
On behalf of Nomadic Labs, I’m pleased to announce the first public release of `data-encoding`: a library to encode and decode values to JSON or binary format. `data-encoding` provides fine grained control over the representation of the data, documentation generation, and detailed encoding/decoding errors. In the [Tezos](https://gitlab.com/tezos/tezos) project, we use `data-encoding` for binary serialisation and deserialisation of data transported via the P2P layer and for JSON serialisation and deserialisation of configuration values stored on disk. The library is available through opam (`opam install data-encoding`), hosted on Gitlab ([https://gitlab.com/nomadic-labs/data-encoding](https://gitlab.com/nomadic-labs/data-encoding)), and distributed under MIT license. This release was only possible following an effort to refactor our internal tools and libraries. Most of the credit for this effort goes to Pietro Abate and Pierre Boutillier. Additional thanks to Gabriel Scherer who discovered multiple bugs and contributed the original crowbar tests. Planned future improvements of the library include: - splitting the library into smaller components (to minimise dependencies when using only a part of the library), and - providing multiple endianness (currently the library only provides big-endian binary encodings). ---
**Author:** ![UnixJunkie](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/unixjunkie/32/638_2.png) [@UnixJunkie](https://discuss.ocaml.org/u/UnixJunkie)\ **Post date:** [October 8, 2019, 7:12am UTC](https://discuss.ocaml.org/t/ann-first-release-of-data-encoding-json-and-binary-serialisation/4444/2 "2019-10-08T07:12:52Z")
Is this faster and/or safer than stdlib’s Marshal? ---
**Author:** ![raphael-proust](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/raphael-proust/32/415_2.png) [@raphael-proust](https://discuss.ocaml.org/u/raphael-proust)\ **Post date:** [October 11, 2019, 5:45am UTC](https://discuss.ocaml.org/t/ann-first-release-of-data-encoding-json-and-binary-serialisation/4444/3 "2019-10-11T05:45:52Z")
## The short answer I’m not aware of benchmarking comparing it to stdlib’s marshal. If I had to guess I’d say it’s probably slower in most cases. ## The longer answer The library is intended to give fine control over the representation of data. To this end, you first assemble a `t encoding` for the type `t` you want to serialise. The performance of the serialisation depends heavily on how you use the provided combinators to assemble that `encoding`. E.g., you can represent all integers in the native machine format (which is probably faster) or, provided your code has the right invariants, you can represent some integers in a fraction of the space (say `int8`, or `int16`). Because the library gives you a lot of control, you can choose where you want to be on the space-vs-time tradeoff. You can include a lot of dynamic check right in the encoding, or you can leave that responsibility to the caller. * * * All in all, it’s a pretty different library. You could use combinators to define an `encoding` that’s compatible with a serialising library written in a different language. You could use a standard compression algorithm as part of an encoding. You could manually align the encoded data on an arbitrary boundary so that it is split cleanly by network packets so that partial decoding works better. In short: you have more control over both the representation of the data and the process of serialisation/deserialisation, so you can decide to go fast (although I’m not sure you can go as fast as Marshal) or go compact or something else. ---
**Author:** ![cemerick](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/cemerick/32/1383_2.png) [@cemerick](https://discuss.ocaml.org/u/cemerick)\ **Post date:** [October 15, 2019, 8:29pm UTC](https://discuss.ocaml.org/t/ann-first-release-of-data-encoding-json-and-binary-serialisation/4444/4 "2019-10-15T20:29:50Z")
@raphael-proust only talked about performance; I haven’t dug very far into `data-encoding`'s implementation, but I assume that it doesn’t carry the risk of segfaults or similar failures, which can happen when using `Marshal` to deserialize data with a layout that does not match the desired type. ---
**Author:** ![raphael-proust](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/raphael-proust/32/415_2.png) [@raphael-proust](https://discuss.ocaml.org/u/raphael-proust)\ **Post date:** [October 16, 2019, 12:04am UTC](https://discuss.ocaml.org/t/ann-first-release-of-data-encoding-json-and-binary-serialisation/4444/5 "2019-10-16T00:04:43Z")
That’s correct. When trying to deserialise, the failure modes of deserialisation are more informative: ```auto type read_error = | Not_enough_data | Extra_bytes | No_case_matched | Unexpected_tag of int | Invalid_size of int | Invalid_int of {min : int; v : int; max : int} | Invalid_float of {min : float; v : float; max : float} | Trailing_zero | Size_limit_exceeded | List_too_long | Array_too_long exception Read_error of read_error val pp_read_error : Format.formatter -> read_error -> unit ``` Note also that the library includes some safe versions of deserialisation functions that return `option` values. ```auto (** [of_bytes enc buf] is equivalent to [read enc buf 0 (length buf)]. The function returns [None] if the buffer is not fully read. *) val of_bytes : 'a Encoding.t -> Bytes.t -> 'a option (** [of_bytes_exn enc buf] is equivalent to [of_bytes], except @raise [Read_error] instead of returning [None] in case of error. *) val of_bytes_exn : 'a Encoding.t -> Bytes.t -> 'a ``` Note however that it is possible for users of the library to create encodings that can fail with a different exception. Specifically, because users have control over the deserialisation process, they are free to raise exceptions when appropriate to their own deserialisation process. ---
**Author:** ![hannes](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/hannes/32/2302_2.png) [@hannes](https://discuss.ocaml.org/u/hannes)\ **Post date:** [October 16, 2019, 8:52am UTC](https://discuss.ocaml.org/t/ann-first-release-of-data-encoding-json-and-binary-serialisation/4444/6 "2019-10-16T08:52:55Z")
> [@raphael-proust](#): > > some safe versions of deserialisation functions that return `option` I wonder why not have `val of_bytes : 'a Encoding.t -> Bytes.t -> ('a, read_error) result` since you already spent the time to construct these nice errors!? (and there’s `Result.to_opton` in the stdlib, but the other way around – once you’ve `None` you can’t recover the `read_error`.) ---
**Author:** ![raphael-proust](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/raphael-proust/32/415_2.png) [@raphael-proust](https://discuss.ocaml.org/u/raphael-proust)\ **Post date:** [October 16, 2019, 9:53pm UTC](https://discuss.ocaml.org/t/ann-first-release-of-data-encoding-json-and-binary-serialisation/4444/7 "2019-10-16T21:53:14Z")
This is due to a series of historical design decisions. Mostly it has to do with the interplay between the error monad and the data-encoding library in the original project: Error management in the Tezos project uses an error monad (not yet released, but possibly some time in the future). And it turns out that having encodings for errors was more useful than having errors for encodings: so the error monad depends on the data-encoding library but not vice versa. And so the data-encoding used very bare-bones error-management. * * * I’ll add this feature to the data-encoding library. And I’ll include it in the `v0.2` release. ---
**Author:** ![raphael-proust](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/raphael-proust/32/415_2.png) [@raphael-proust](https://discuss.ocaml.org/u/raphael-proust)\ **Post date:** [February 14, 2020, 2:53pm UTC](https://discuss.ocaml.org/t/ann-first-release-of-data-encoding-json-and-binary-serialisation/4444/8 "2020-02-14T14:53:32Z")
The newly released version (0.2) addresses this. All the binary reading/writing primitives use `result` by default and have `_opt` and `_exn` variants. The JSON primitives are not yet changed because they rely on an external library that has more idiosyncratic error management. (This will eventually be fixed in a future version.) ---
**Author:** ![BikalGurung](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/bikalgurung/32/5105_2.png) [@BikalGurung](https://discuss.ocaml.org/u/BikalGurung)\ **Post date:** [May 20, 2020, 9:16am UTC](https://discuss.ocaml.org/t/ann-first-release-of-data-encoding-json-and-binary-serialisation/4444/9 "2020-05-20T09:16:52Z")
Is this lib okay to use with `js_of_ocaml`? ---
**Author:** ![raphael-proust](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/raphael-proust/32/415_2.png) [@raphael-proust](https://discuss.ocaml.org/u/raphael-proust)\ **Post date:** [May 20, 2020, 9:39am UTC](https://discuss.ocaml.org/t/ann-first-release-of-data-encoding-json-and-binary-serialisation/4444/10 "2020-05-20T09:39:59Z")
If I remember correctly, there was some issue with the `zarith` dependency (which is used for arbitrary precision integers), but this may have been fixed now. I’ll try to check and let you know. ---
**Author:** ![raphael-proust](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/raphael-proust/32/415_2.png) [@raphael-proust](https://discuss.ocaml.org/u/raphael-proust)\ **Post date:** [May 20, 2020, 9:52am UTC](https://discuss.ocaml.org/t/ann-first-release-of-data-encoding-json-and-binary-serialisation/4444/11 "2020-05-20T09:52:41Z")
It should be possible to get it to work using the `zarith_stubs_js` package for support. But I don’t know if it would work out-of-the-box. I’ll open an issue on the tracker and I’ll try to get js-support packaged for the next release. If you try it and manage to get it to work (and even if you don’t), let me know! ---
**Author:** ![BikalGurung](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/bikalgurung/32/5105_2.png) [@BikalGurung](https://discuss.ocaml.org/u/BikalGurung)\ **Post date:** [January 3, 2021, 6:45pm UTC](https://discuss.ocaml.org/t/ann-first-release-of-data-encoding-json-and-binary-serialisation/4444/12 "2021-01-03T18:45:24Z")
While discussing data serialization/deserialization with Marshal in [Marshal determinism and stability](https://discuss.ocaml.org/t/marshal-determinism-and-stability/7041/), I came to know that Marshal does not guarantee that serialized data between different ocaml versions work. In particular I am looking at `Binary` encoding/decoding of this package. Does `data-encoding` support serializing/deserializing among diffrent ocaml versions? ---
**Author:** ![raphael-proust](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/raphael-proust/32/415_2.png) [@raphael-proust](https://discuss.ocaml.org/u/raphael-proust)\ **Post date:** [January 4, 2021, 8:58am UTC](https://discuss.ocaml.org/t/ann-first-release-of-data-encoding-json-and-binary-serialisation/4444/13 "2021-01-04T08:58:56Z")
The serialisation format of data is meant to depend exclusively on your definition of the encoding (the `'a encoding` value that you produce) and not on the version of the language that is used. In other words: yes, you can serialise/deserialise among different ocaml versions. This is a very important property of `data-encoding`'s use in Tezos because it is vital that newer version of the binaries are able to decode data produced by older versions of the binaries. We have some regression tests specifically designed to catch changes in the binary representation of existing data. Note however, that the Tezos project only bumps its OCaml dependency from time to time. As a result, the aforementioned regression tests have not been run on the most recent releases of OCaml. Also note that the data-encoding library is only available for a given range of OCaml versions (currently `>= 4.08`). It should be possible to add support for some older versions (if it just means adding a stdlib forward compatibility shim) but not all (some dependencies of `data-encoding` such as `zarith` might have hard limits). ---
**Author:** ![raphael-proust](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/raphael-proust/32/415_2.png) [@raphael-proust](https://discuss.ocaml.org/u/raphael-proust)\ **Post date:** [February 4, 2021, 2:07pm UTC](https://discuss.ocaml.org/t/ann-first-release-of-data-encoding-json-and-binary-serialisation/4444/14 "2021-02-04T14:07:22Z")
> [@UnixJunkie](#): > > Is this faster and/or safer than stdlib’s Marshal? You might be interested in version 0.3 of the library, and specifically about the performance improvements that shipped in it: [[ANN] data-encoding.0.3: performances and streaming - #2 by raphael-proust](https://discuss.ocaml.org/t/ann-data-encoding-0-3-performances-and-streaming/7205/2)