I’m looking for advice how best to implement a small tool that copies data from one file descriptor to another using Lwt on Unix to interleave read and write operations. This needs to work for file descriptors where the size of the data is not known in advance. I would like to optimise throughput by carrying out read operations while writing data - so just repeatedly reading a small block and then writing it is not what I am after.
- I could implement a ring buffer using non-blocking IO that is shared between a reader and writer thread. A possible advantage is that memory requirements are explicit but requires handling offsets.
- I could use a bounded
Lwt_streamto pass buffers of 64k (or so) between a reader and a writer thread. This is more abstract but one might have to be careful about creating lots of short-lived buffers.
- Your idea here.