Interface for `map` over nested result type (like DB result sets)

Something I’ve wanted to add to PGX and Mssql for a while is a way to execute a query and deal with each row as they come in, instead of building up a list and then working with that.

For example, it’s common to run a query and then immediately rewrite the result:

Mssql.execute "SELECT a, b, c FROM example"
>>| List.map ~f:t_of_row

This holds the entire DB result set in memory + the list we’re generating, so it uses ~twice as much peak memory as this would (if it existed):

Mssql.execute_map "SELECT a, b, c FROM example" ~f:t_of_row

The list interface also forces you to wait for all of the results to come in, and something like this would be much lower-latency:

Mssql.execute_iter "SELECT a, b, c FROM example ~f:print_row

Unfortunately in databases, some queries return multiple result sets:

Mssql.execute_multi_result db "SELECT a; SELECT b"
>>| fun (results: Row.t list list) -> ...

Which is all background to my question: Is there some map/fold-like interface that makes sense for a function like this, which returns a nested list? Are there examples of libraries that do this?

I can come up with some messy ways to do this if I only care about an iter version, like:

val execute_multi_result_iter : result_set_over_f:(unit -> unit) -> row_f:(Row.t -> unit) -> db -> unit Deferred.t

But that seems… not ergonomic, and I can’t really figure out how to design a fold-like function at all.

Am I missing something obvious?

My iter library deals with
('a -> unit) -> unit functions as first class objects, including
Iter.flat_map that does this kind of flattening.

(an old presentation on the library back when it was called “sequence”:
https://simon.cedeela.fr/assets/talks/sequence.pdf)

I think one thing that might make this more complicated is that switching to something like this would require that the algorithm become pull-base instead of push-based… which is arguably a good idea, but I’m a little scared of how complicated it would be since it would be interacting with a stateful protocol in a separate thread. It would be easier if I could maintain the push-based nature of iter/map/fold :\

Iter just wraps iter-like functions, so it’d remain push-based.

Ahh I see. I think for my purposes it’s complicated because the Iter.t is only valid within a particular context, but I can make a function val execute_multi_result_f : t -> string -> f:(Row.t Iter.t Iter.t -> 'a) -> 'a Deferred.t, and that’s not-bad from an ergonomics perspective.

Thanks for the help! I ended up using Iter like this: https://github.com/arenadotio/ocaml-mssql/pull/27