[Solved]How to make async nested function return only a `Deferred.t`?

[Taki]

I’m trying to use async to make a mini-crawler, when I send a request by get_body to the target site A, it returns its body with string Deferred.t type.

And I wrote another function fn_on_soup in order to get the wikilink on it.

let get_bd_wiki_abst (keyword:string)  =
  let%map body = get_body keyword in
  let soup = parse body in  
  let wikilink = fn_on_soup soup in
  wikilink

The wikilink will just relocate to another site with a 302 header, so here’s what I did to get the real link:

let get_302_hdr (url:string)  =
  let uri = Uri.of_string url in
  let%map resp_head = Cohttp_async.Client.head uri in
  let hdrs = Response.headers resp_head in
  match Header.get hdrs "location" with
  | Some location -> location
  | None -> "No redir"

Now I combine these two functions as follows:

let get_real_link =
  let%map wikilink = get_bd_wiki_abst "soga" in
  let%map real_link = get_302_hdr wikilink in
  real_link

It turns out that the get_real_link has type string Deferred.t Deferred.t, I was wondering how may I make it return only one Deferred.t?

[rgrinberg]

Deferred.join will flatten your deferreds if you want.

But I think what you're looking for is bind rather than map in your last snippet.

[Taki]

@rgrinberg Thank you for the quick reply. I used your great Opium to make a demo, and Deferred.join works.

##web server

(*
 * To compile the code:
     * corebuild -pkg opium.unix websvr.native
 * To run the webserver:
    *  ./websvr.native -p 9001 -d
 * *)

open Opium.Std
open Printf

(*
 * http://127.0.0.1:9001/redir
 * *)
let getid = get "/redir" (fun _ ->
    `String "here we are" |> respond')
(*
 * http://127.0.0.1:9001/
 * *)
let hello = get "/" (fun _ ->
    `String "http://127.0.0.1:9001/redir" |> respond')

let app =
  App.empty
  |> hello
  |> getid

let _ =
  app |> App.run_command

##Testing client

(*corebuild -pkg async,uri,cohttp.async reqsend.native*)

open Core
open Async
open Cohttp
open Cohttp_async

let get_body (url:string) =
  let base_uri = Uri.of_string url in
  let%bind _, body_r = Cohttp_async.Client.get base_uri in
  let%map strings = Cohttp_async.Body.to_string body_r in
  strings

let test_abst  =
  let%map wikilink = get_body "http://127.0.0.1:9001/" in
  let%map real_link = get_body wikilink in
  print_endline real_link

let () =
  Command.async
    ~summary:"Test for async http request"
    Command.Spec.(empty)
    (fun  () -> (Deferred.join test_abst))
  |> Command.run

But I don't get you point of using let%bind instead of let%map, any hint would be appreciated:slight_smile:

[Leonidas]

But I don't get you point of using let%bind instead of let%map, any hint would be appreciated

Sure, here you go:

(* corebuild -pkg async,uri,cohttp.async reqsend.native *)

open Async

let get_body (url:string) =
  let base_uri = Uri.of_string url in
  let%bind _, body_r = Cohttp_async.Client.get base_uri in
  Cohttp_async.Body.to_string body_r

let test_abst () =
  let%bind wikilink = get_body "http://127.0.0.1:9001/" in
  let%map real_link = get_body wikilink in
  print_endline real_link

let () =
  Command.async
    ~summary:"Test for async http request"
    Command.Spec.(empty)
    test_abst
  |> Command.run

The trick is to transform one Deferred.t into a new Deferred.t and not to chain them into each other.

[rgrinberg]

The response from Leonidas is exactly what I meant. By the way, opium is really only Lwt compatible. It will not work well with Async. So if you’d like to use it, you should switch to Lwt.

[Taki]

Thank you for your advice, I’m learning OCaml by reading RWO, all the examples were based on Core/Async, I think I will learn Lwt some day:)

[antron]

You may find this useful if/when you want to learn it:

dkim/rwo-lwt