Hey all, I’m new to the ocaml ecosystem and I’m trying to get my bearings on libraries and conventions around common operations.
I have some code that downloads a lot of large files. Currently I’m doing this with Cohttp and Async using
Cohttp_async.Response.to_pipe. Coming from a systems programming world, this is a really straight forward API.
Using it in combination with
Async.Pipe.iter, does does exactly what one would assume. The writer blocks intermittently, keeping memory use low while the reader picks data off the fifo. Using it in combination with output channels lets you write it out to a file, or open up the standard input of another program (super handy). And when I say blocking the writer, I mean the literal thread handling that work. This is all “hidden” by the async libraries.
I haven’t figured out how to approach this in Lwt yet; enqueue data into a fifo, blocking the writer so memory usage stays low, and then have the ability do dequeue that data into an arbitrary file descriptor.
If anyone has input, I’d greatly appreciate it. In the meantime, I’ll keep looking through the docs.