How to write, read file using bin_prot?

Learning
1

I’m confused about how to use bin_prot to write and read serialized data to/from a file.

bin_prot has many functions with “write” and “read” in their names, and some return functions with type writer, etc. (see Bin_prot.Write and Bin_prot.Read). However, it looks as if reading and writing is always to/from a Bin_prot.Common.buf buffer (which is a Bigarray.Array under the hood).

Nothing in bin_prot seems to deal with file I/O. (I haven’t explored every function in every bin_prot module, but I’ve checked every module that looks like it might be relevant.) This seems sensible; file I/O can be done elsewhere.

So my guess has been that the intended way to use bin_prot is to write or read from a special bin_prot buffer to perform serialization conversions. Then one writes the buffer to a file using non-bin_prot functions.

However, the type of bin_prot's buffers are specific to bin_prot. For example, the fuction Core.Out_channel.output_buffer has type out_channel -> Base__Buffer.t -> unit = <fun>, but Base_Buffer.t is not compatible with Bin_prot.Common.buf. I also have not figured out how to write bin_prot buffers using I/O functions in Pervasives.

What’s a good way to use bin_prot to serialize to/from a file? Any help, pointers (“Go read X”), etc. will be appreciated. Thanks.

(Nothing in the bin_prot README addresses these questions afaics. I also haven’t yet found answers in either version of RWO.)

2

Yeah, I don’t see anything in the README about how to actually do I/O with bin_prot.

Bin_prot.Common.buf is the same type as Core_kernel.Bigstring.t, so you can use, e.g.:

to get file data into/out of a Bigstring.t, after which you can deal with the buffer using bin_prot.

There is also the Unpack_buffer module, which is briefly mentioned in a comment in Bin_prot.Utils.

1 Like
3

That worked! Thanks @bcc32. Core.Bigstring (along with Core.Unix for file open/close operations) provide what I need.

I used Core.Bigstring.write and Core.Bigstring.read, and will try memory mapped files later, too.

(For others reading this later, I’ll note that Core.Unpack_buffer mentioned in Bin_prot.Utils seems to have moved from Core to Core_kernel, and doesn’t directly work with regular disk-style files afaics.)