I’m trying to use async to make a mini-crawler, when I send a request by get_body to the target site A, it returns its body with string Deferred.t type.
And I wrote another function fn_on_soup in order to get the wikilink on it.
let get_bd_wiki_abst (keyword:string) =
let%map body = get_body keyword in
let soup = parse body in
let wikilink = fn_on_soup soup in
wikilink
The wikilink will just relocate to another site with a 302 header, so here’s what I did to get the real link:
let get_302_hdr (url:string) =
let uri = Uri.of_string url in
let%map resp_head = Cohttp_async.Client.head uri in
let hdrs = Response.headers resp_head in
match Header.get hdrs "location" with
| Some location -> location
| None -> "No redir"
Now I combine these two functions as follows:
let get_real_link =
let%map wikilink = get_bd_wiki_abst "soga" in
let%map real_link = get_302_hdr wikilink in
real_link
It turns out that the get_real_link has type string Deferred.t Deferred.t, I was wondering how may I make it return only one Deferred.t?
(*
* To compile the code:
* corebuild -pkg opium.unix websvr.native
* To run the webserver:
* ./websvr.native -p 9001 -d
* *)
open Opium.Std
open Printf
(*
* http://127.0.0.1:9001/redir
* *)
let getid = get "/redir" (fun _ ->
`String "here we are" |> respond')
(*
* http://127.0.0.1:9001/
* *)
let hello = get "/" (fun _ ->
`String "http://127.0.0.1:9001/redir" |> respond')
let app =
App.empty
|> hello
|> getid
let _ =
app |> App.run_command
##Testing client
(*corebuild -pkg async,uri,cohttp.async reqsend.native*)
open Core
open Async
open Cohttp
open Cohttp_async
let get_body (url:string) =
let base_uri = Uri.of_string url in
let%bind _, body_r = Cohttp_async.Client.get base_uri in
let%map strings = Cohttp_async.Body.to_string body_r in
strings
let test_abst =
let%map wikilink = get_body "http://127.0.0.1:9001/" in
let%map real_link = get_body wikilink in
print_endline real_link
let () =
Command.async
~summary:"Test for async http request"
Command.Spec.(empty)
(fun () -> (Deferred.join test_abst))
|> Command.run
But I don’t get you point of using let%bind instead of let%map, any hint would be appreciated:slight_smile:
The response from Leonidas is exactly what I meant. By the way, opium is really only Lwt compatible. It will not work well with Async. So if you’d like to use it, you should switch to Lwt.