Yo!
I am happy to announce the first public release of Bytream, a library for streaming bytes and crunching them.
This library contains angostic I/O runtime mechanisms for organization byte streams processing for write and read in an efficient manner, e.g. for buildings codecs and protocol implementations.
The idea and motivation for creating this project arose from the need to effectively parse all kinds of binary formats and protocols. Existing tandem Angstrom and Faraday is great, but it is more expressive than I need. And Bytesrw and Iostream talk about how to receive and send data, not how to process or record it.
That’s why Bytream solves this problem. It borrows ideas from slices and lifetimes from ByteSw and unbuffering from Angstrom, allowing you to write regular OCaml code like with the Stdlib channels, which work in streaming mode at any I/O runtime.
First release details
Starting from the first version, two abstractions are already available to you, one for input (Bytream.In.t), and the other for outputting data (Bytream.Out.t). Both use Bigarray under the hood to represent an array of bytes. The motivation for this choice is to avoid duplication and fix runtime when transferring this data to external functions.
As already mentioned, inheriting ideas from Bytesrw, Bytream uses the mechanism of chunks to feed the stream. An example illustrates the basic concept of chunking:
(* Queue as a byte chunk source. *)
let queue =
let queue = Queue.create () in
Queue.add "he" queue;
(* ... *)
Queue.add "d!" queue;
queue
in
(* Reader function that returns chunks of text from the source. *)
let reader () =
match Queue.take_opt queue with
| None ->
(** For close incoming byte stream, the reader
should raise an End_of_file exception. *)
raise End_of_file
| Some chunk -> Bstr.of_string chunk
in
let in_stream = Bytream.In.make reader in
Bytream.In.input_string in_stream 7
(* - : string = "hello w" *)
and alternative for outgoing byte stream.
(* Queue as a byte chunk sink. *)
let queue = Queue.create () in
(* Writer function that outputs chunks of text to the sink. *)
let writer (~buffer, ~length:len, ..) =
Queue.add Bstr.(sub_string ~off:0 ~len buffer) queue
in
let out_stream = Bytream.Out.make writer in
(* ... *)
In real cases, we will of course use channels, files, sockets, and other things to communicate with the outside world. And do it streaming.
module Bson = struct
(* ... *)
let from_channel ic =
let in_stream = Bytream.In.of_channel ic in
Codec.input_document in_stream
let into_channel oc doc =
let out_stream = Bytream.Out.of_channel oc in
Codec.output_document out_stream doc
let to_string doc =
Bytream.Out.with_into_string @@ fun out_stream ->
Codec.output_document out_stream
Working with Lwt through Lwt_direct is currently available in an experimental capacity.
#require "bytream.lwt";;
let something_into_channel oc =
let%lwt out_stream = Bytream_lwt.Out.of_channel oc in
(* The usual Bytream code. *)
With other direct style I/O runtimes (like Eio, Miou, etc), it will work much better.
Or yet example.
match request with
| `Post "/archives/", body_stream ->
(* The reader has its own internal buffer mechanism that allows it to bufferize
the contents of the body stream and decode them without copying chunks. *)
let reader = Archive_reader.in_stream_of body_stream in
let archive_meta =
Bytream.In.make Archive_reader.(to_handler reader)
|> Archive_reader.input_archive_without_contents
in
let blob = Archive_reader.blob reader in
process_archive ~meta:archive_meta ~blob ()
(* ... *)
Plans for the next release
- Improve documentations
- Add unit tests
- Add combinators?
- Your suggestions…
Installation and play
Already available at OPAM!
You can install the bytream library using the OPAM package manager or any other method you prefer.
$ opam install bytream
You can also get the latest version of the upstream (developer) branch.
$ opam pin bytream.dev https://github.com/dx3mod/bytream.git
If you are using Dune, please add the bytream library to your dependencies.
Showcases
You can explore ecosystem libraries that use Bytream to better understand its applicability.
- Rpmfile is the library for reading and writing RPM packages has been ported from Angstrom since version 1.0.0 (currently, we are in the process of development);
- Intel_hex planned;
- and many more…
Afterword
I will be very happy to hear your opinion about this library. Maybe you could provide some feedback. Also, comments from senior colleagues about implementation details and the API would be appreciated. The source code is open under the MIT license, so it’s always ready for changes.
Enjoy it! 
