How to make http calls in batches (with a delay in between)

Learning
1

Hello Everyone,
I am trying to learn the Cohttp library. I have an array of URLs (size 800+) and the API supports making 1 call at a time (they don’t have a bulk endpoint). But if I try to make 800 calls in a list.map then the API kicks me out (Server closed connection unexpectedly). So I am trying to create batches of 50 and then wait for 1 second for each batch. This way I will not be hammering the server. I wrote this code

  1. Create batches of 50 from the big list of 800
let split size list = 
  let create_result result str =
    let hd = result |> List.hd in 
    let tl = result |> List.tl in 
    if hd |> List.length <= size - 1 then
      (str :: hd) :: tl
    else
      [str] :: result in
  list |> List.fold_left create_result [[]]

Now that I have a list of lists I need to make http calls. For this I have written this code

let get_metadata_for_trails () : string list Lwt.t =
  trailsList // list of tuples with 800 entries
  |> Util.split 50 
  |> List.map (fun batch -> Lwt.bind (Lwt_unix.sleep 1.0) (fun _ -> batch |> List.map (make_http_call) |> Lwt.all)) 
  |> Lwt.all 
  |> Lwt.map(List.flatten)

But my code is not working when I run it… it just waits for a very long time then finally I see some results but I get a cryptic error message

“Fatal error: exception SSL read() error: error:00000000:lib(0)::reason(0)”

I have no idea how to troubleshoot this.I googled but didn’t find any meaningful solution.

My full code is here GitHub - abhsrivastava/weather-metadata: Hobby project to download weather data from weather.gov API

It certainly has something to do with the size of my list because if I comment out my list of tuples from size of 800 to only 3, then everything works fine. but the list of 800 always fails with the above error message (so it seems the server kicks me out of crashes).

2

Without going into specifics, it sounds like you need a rate limiter for your HTTP calls. Personally I’ve used this with some success: Module Lwt_throttle

3

Thanks @yawaramin
Is there an example of using the throttle somewhere (I googled and couldn’t find anything)? by reading the documentation I couldn’t figure out how to use it.

4

I use it like this:

module RateLimiter = Lwt_throttle.Make(struct
  type t = string

  let hash = Hashtbl.hash
  let equal s1 s2 = hash s1 = hash s2
end)

let req_per_s = 120
let num_simultaneous_calls = 1
let limiter =
  RateLimiter.create ~rate:req_per_s ~max:req_per_s ~n:num_simultaneous_calls
let channel = "the_api"

let rec get client path =
  let open Lwt.Syntax in
  let* allowed = RateLimiter.wait limiter channel in
  if allowed then
    Client.get client path
  else
    let* () = Lwt_unix.sleep 1. in
    get client path
4 Likes