Make a json string from an OCaml string

Learning
,
1

“Simple” but blocking problem with a json string I send over http.

EDIT: the issue comes from Yojson.Basic.to_string that delivers an OCaml string but with additional quotes.

let y = Yojson.Basic.to_string x   (* x is a valid Yojson.Basic.t  name/value, here just "foo"  *)
let z = "{ \"session_id\" : \"" ^ y ^ "\" }" 
      (* building json string manually *)

gives (in the browser network tab):

{ "session_id" : ""foo"" }

    let z = `Assoc [ ("session_id", `String y) ] in 
                                 Yojson.Basic.to_string z
                                            (* using Yojson *)

gives

{"session_id":"\"foo\""}

If I write:

"{ \"session_id\" : \" ^ y ^ \" }" )

then y is no more recognized as a string.

Where am I wrong?
(no problem with x-www-form-urlencoded)
Thanks

EDIT
I had a look at RWO where I read the same as my manually build json string

Yojson.Basic.pretty_to_string person ;;
- : string = "{ \"name\": \"Anil\" }"

EDIT2
The question is

2

If I run in ocaml

let y = "foo"
let z = "{ \"session_id\" : \"" ^ y ^ "\" }"

let () = Printf.printf "%s" z

I get { "session_id" : "foo" } so it’s probably a problem later in your program.

3

I get the same as you:

# let y = "foo"
# let z = "{ \"session_id\" : \"" ^ y ^ "\" }"
val z : string = "{ \"session_id\" : \"foo\" }"
# let () = Printf.printf "%s" z
{ "session_id" : "foo" }

But if I insert Printf.printf "%s" in my program, I get a type error because it expects to send as string over http while Printf.print is of type unit:

Error: This expression has type string Lwt.t
       but an expression was expected of type unit Lwt.t
       Type string is not compatible with type unit
4

When you manually build up the json string, what you have is an OCaml string which carries a json encoded string. Therefore, you need to parse it.


(* This is json, you need to parse it as such *)
let manually_created_json_string = "\"foo\""

let foo =
  match Yojson.Basic.from_string manually_created_json_string with
  | `String foo -> foo
  | #Yojson.json as e -> failwithf "Expected string but got: %s" (Yojson.to_string e)

But, it seems like a pretty roundabout way of making your json document. Instead if you already have a string that you want to represent as json, you could just do the following.

let y = "foo"
let z = `Assoc [ ("session_id", `String y) ]
let json = Yojson.Basic.to_string z

Here, json is an OCaml string that looks like "{\"session_id\": \"foo\"}"
and if you print it

print_endline json

it should print out {"session_id": "foo"}

5

As I send the json string over http, I need a string type.

# Yojson.Basic.pretty_to_string person ;;
- : string = "{ \"name\": \"Anil\" }"
# Yojson.Basic.pretty_to_channel stdout person ;;
{ "name": "Anil" }- : unit = ()

Hum As I get exactly the string illustrated in RWO (see my EDIT), it should be something related to the content type of http request.

PS
What is #Yojson.json ?
(maybe you can remove the trailing f in failwith’f’ ?)

6

I misunderstood the example from RWO.

Yojson.Basic.pretty_to_string person;;
- : string = "{ \"name\": \"Anil\" }" 

Yojson.Basic.pretty_to_channel stdout person;;
{ "name": "Anil" }
- : unit = ()

I’m trying to serve a string to a http request.
An OCaml string should use \" to represent a JSON string.

So the question should be:
how do you extract a string from a Yojson.Basic.t object in a string expected by a function to send it as a response over http ? (as with Yojson.Basic.to_file which works well)

7

That’s not exactly this.
So you want to send this text, which is valid json: { "name": "Anil" }
A representation of this text with an ocaml string is

let s = "{ \"name\": \"Anil\" }"

If you send the string s to a file, to a socket or to whatever, you will get exactly { "name": "Anil" }

8

Recently the yojson interface changed a little bit, so my apologies, the code examples I gave above assumed the older interface (and the example I’m including here does as well). You might need to change from_string to of_string and Yojson.Basic.json to Yojson.Basic.t. I’m not totally sure though since I haven’t yet upgraded to the latest version of Yojson (and Core… for that matter).

You asked about the #Yojson.json syntax, basically it just represents “the remaining polymorphic variants I haven’t handled”. You can read a little more about that here https://dev.realworldocaml.org/variants.html if you search on that page for | #color as color ->

Still, I’m not totally sure I’m understanding your question.

Here’s an example of calling out to httpbin and parsing out their echo response.

Filename: main.ml

open Core
open Async

(* To construct JSON you can do something like the following *)
let person name = `Assoc [ "name", `String name ]
let richard = person "Richard"

(* To deconstruct JSON, you have to pattern match *)
let say_hello (json : Yojson.Basic.json) =
  match json with
  | `Assoc [ ("name", `String name) ] -> printf "Hello, %s\n" name
  | #Yojson.Basic.json as e ->
    failwithf
      "Expect json with shape {\"name\", ...}, but got %s"
      (Yojson.Basic.to_string e)
      ()
;;

(* If you want to POST json in an HTTP body *)
let do_post uri (json : Yojson.Basic.json) =
  let body = Cohttp_async.Body.of_string (Yojson.Basic.to_string json) in
  Cohttp_async.Client.post ~body uri
;;

let main () =
  let uri = Uri.of_string "https://httpbin.org/post" in
  do_post uri richard
  >>= fun (_response, body) ->
  Cohttp_async.Body.to_string body
  >>| fun body ->
  printf "HTTPBin Raw Response: %s\n" body;
  let echo_response =
    body
    |> Yojson.Basic.from_string
    |> Yojson.Basic.Util.to_assoc
    |> (fun assoc -> List.Assoc.find_exn assoc "data" ~equal:String.equal)
    |> Yojson.Basic.Util.to_string
  in
  printf "HTTPBin Echo Response: %s\n" echo_response;
  say_hello (Yojson.Basic.from_string echo_response)
;;

let () = 
  Command.Param.return main
  |> Command.async ~summary:"asdf"
  |> Command.run
;;

Filename: dune

(executable
 (name main)
 (libraries core async uri yojson cohttp cohttp-async))

Example output:

$ dune exec ./main.exe
HTTPBin Raw Response: {
  "args": {},
  "data": "{\"name\":\"Richard\"}",
  "files": {},
  "form": {},
  "headers": {
    "Content-Length": "18",
    "Host": "httpbin.org",
    "User-Agent": "ocaml-cohttp/2.0.0"
  },
  "json": {
    "name": "Richard"
  },
  "origin": "123.123.123.123, 123.123.123.123",
  "url": "https://httpbin.org/post"
}

HTTPBin Echo Response: {"name":"Richard"}
Hello, Richard
9

Thanks all for your useful feedback.
I was aware that I had to produce a string with escaped quotes to feed the http program with, like the following:

let s = "{ \"name\": \"Anil\" }"

That’s why I wrote:

let y = Yojson.Basic.to_string x   
        (* y has type Yojson.Basic.t 
           (and NOT x as mentioned in the question *)

let z = "{ \"session_id\" : \"" ^ y ^ "\" }" 
      (* building json string manually *)

which gives (as seen in the browser network tab):

{ "session_id" : ""foo"" }

Using Yojson to build a Yojson.Basic.t type gives the same output:

let y = Yojson.Basic.from_string x

let z = `Assoc [ ("session_id", `String y) ] in 
                           Yojson.Basic.to_string z
                           (* using Yojson *)

But in fact, x was not the string "foo" I thought it was, but a string already processed with Yojson (I did not mentioned that in my question): "\"foo\"" hence the issue of getting additional quotes.

# let y = "foo"
# let s = "{ \"id\" : \"" ^ y ^ "\" }" 
# print_endline s
{ "id" : "foo" }  (* OK *)
- : unit = ()

# let y = "\"foo\"" 
(* in fact, this is the string outputed by Yojson.Basic.to_string *)
# let s = "{ \"id\" : \"" ^ y ^ "\" }";
# print_endline s
{ "id" : ""foo"" } (* NOT OK *)
- : unit = ()

So, if y is "foo" and not "\"foo\"", I can build manually a valid JSON string like that:

"{ \"session_id\" : \"" ^ y ^ "\" }"

A better solution is to build an object of type Yojson.Basic.t then to output it with valid escaped quotes.

# let y = `String "foo" (* this is what I extracted with Yojson functions *)
val y : [> `String of string ] = `String "foo"

# let s = `Assoc [ ("id", y) ]
val s : [> `Assoc of (string * [> `String of string ]) list ] =
  `Assoc [("id", `String "foo")]

# Yojson.Basic.to_string s
- : string = "{\"id\":\"foo\"}" (* OK *)

It’s useful to remember that Yojson.from_string needs a string with espaced quotes:

let y = Yojson.Basic.from_string "foo"
Exception: Yojson.Json_error "Line 1, bytes 0-3:\nInvalid token 'foo'".

let y = Yojson.Basic.from_string "\"foo\"" 
val y : Yojson.Basic.t = `String "foo"

I should have started with that.
The issue is closed.
Thanks.

1 Like
10

With OCaml 4.07.1, Core v0.12.4 and Yojson 1.7.0, the example could be interpreted/compile with:

Yojson.Basic.from_string
Yojson.Basic.t

With Yojson.Basic.json , I just got a warning:

Warning 3: deprecated: Yojson.Basic.json
json types are being renamed and will be removed in the next Yojson major version. Use type t instead

Thanks again for your detailed example.

BTW, have you ever heard about a possibly old Yojson syntax that should look like:

type foo =  { err_msg : string }
let f x = Yojson.to_string<foo> { x }

Or maybe that writing <foo> is an Ocaml feature to say that the argument is of type foo, but I’ve never used it and I didn’t find it in refman.

11

Assuming standard OCaml syntax, this means:

let f x =
  (Yojson.to_string < foo) > { x }

which is ill-typed unless you redefine the < and > operators to something strange.

12

I wrote let f x = Yojson.to_string<foo> { x }
( with type foo = { err_msg : string } )

I don’t understand why you wrote an expression that is not well-formed (right paren inside < > ).
let f x = (Yojson.to_string < foo) > { x }

Searching an explanation to my side question, I found that:
1/ { err_msg : string } is simply an OCaml record to encapsulate an error message
2/ <foo> was used by camlp4 before ppx landed (the “new camlp4, 2007 now deprecated” - different from the “genuine camlp4, renamed camlp5”, still maintained).

BTW, thank you for your nice camlp5 tutorial How to customize the syntax of OCaml, using Camlp5 - Everything you always wanted to know, but were afraid to ask

13

Did you try copy-pasting it into utop? You’ll get an error that is not a syntax error, meaning its syntax is correct. < and > are the standard comparison operators.

14

Drifting to a discussion about comparison operators is interesting but seems to be out of the scope of the question. Or maybe I don’t understand what you mean and how it can bring an answer to the question about the meaning of <x> in the expression let f x = Yojson.to_string<foo> { x } ?

15

There is no syntax like Yojson.to_string<foo> in OCaml.

@mjambon is saying that what you’ve typed is by coincidence syntactically valid, because the < and > can be interpreted as comparison operators. The code fragment

let f x = (Yojson.to_string < foo) > { x }

is just the same as your original fragment, except with some extra parentheses to make it more clear how OCaml parses this.

1 Like
16

I understand what Martin said. But it didn’t answer my side question about the “possibly old yojson preprocessing syntax”.

The expression I gave is supposed to be parsed first by camlp4, not by OCaml. Now, ppx_yojson_deriving brings a slightly different syntax.
Here is the whole example code:

(* syntax with (supposed) camlp4 preprocessing *)
type err_msg = { err : string; } deriving (Yojson)

let json err = Yojson.to_string<err_msg> { err } 

(* syntax with ppx_yojson_deriving preprocessing *)
type err_msg = { err : string; } [@@deriving yojson]
val err_msg_to_yojson : err_msg -> Yojson.Safe.t = <fun>
val err_msg_of_yojson :
  Yojson.Safe.t -> err_msg Ppx_deriving_yojson_runtime.error_or = <fun>
val _ : Yojson.Safe.t -> err_msg Ppx_deriving_yojson_runtime.error_or = <fun>

let json err = Yojson.Safe.to_string (err_msg_to_yojson {err} )
val json : string -> string = <fun>

(* example *)
json "abc";;
- : string = "{\"err\":\"abc\"}" (* ready to be sent over http *)

I still haven’t found documentation about the “old syntax”. I just expected from Martin that he could help me in my archaeological work related to his library yojson.
Anyway ppx_yojson_deriving does the job, and there is even ppx_yojson_conv.

17

Here’s a simplified timeline of the json-related events I was part of (from memory):

  • 2007: I implement a json parser and printing library called json-wheel similar to the future yojson. There was another json library at the time but it was reading objects into a hashtbl, therefore losing the ordering of the fields.
  • Later in 2007: I implement json-static, a camlp4 “syntax extension” which reads OCaml type definitions and translates them to code that converts between these types and the json-wheel AST.
  • Not long after came the “new camlp4”: I translate json-static to the new camlp4 because it is incompatible with the legacy camlp4.
  • A bunch of people are unhappy with the situation since old camlp4 extensions need to be ported and the new camlp4 has bugs, has no documentation, and is slower.
  • At some point the old camlp4 gets resurrected by its author and becomes camlp5.
  • 2010: I start the implementation of a camlp4-free and faster equivalent of json-static, called atdgen. The yojson library is released separately as the runtime for atdgen, and is widely equivalent to json-wheel. Unlike json-wheel, yojson allows atdgen to directly call parsing functions, skipping the intermediate representation as an AST, resulting in better performance.

Later came the ppx system, which I haven’t touched. Yojson and atd are now maintained by other people, for which I am immensely grateful.

Based on this timelime, you can see that having yojson combined with a camlp4 extension is a bit anachronistic. The code sample you have here uses the camlp4 syntax extension called “deriving”. It’s like json-static but supports multiple back-ends. As you mention, you can use its ppx equivalent or atd if you want something similar without camlp4 preprocessing.