Easiest way to convert Logs.Tag.t values into strings?

Learning
1

I have a log backend that doesn’t use stdout (it uploads JSON to Datadog), and I’m trying to figure out the best way to make Logs tags show up.

Basically I have a Format.formatter -> 'a -> unit which prints the value into a formatter, and I have a value, and I want to convert the value to a string.

You can see the exact types in the documentation

What I have now is:

let tag_to_key_value tag =
  let Logs.Tag.V (def, value) = tag in
  let name = Logs.Tag.name def
  and value =
    let printer = Logs.Tag.printer def
    and buffer =
      Buffer.create 80
    in
    let formatter = Format.formatter_of_buffer buffer in
    printer formatter value;
    Format.pp_print_flush formatter ();
    Buffer.contents buffer
  in
  name, value

This seems… really complicated, so I’m wondering if there’s a Format.print_to_string printer value or something that I’m missing.

2

Ah, someone else at my company came up with a solution right after I posted this. The function I wanted was asprintf

  let tag_to_key_value tag =
    let (Logs.Tag.V (def, value)) = tag in
    let name = Logs.Tag.name def
    and value =
      let printer = Logs.Tag.printer def in
      Format.asprintf "%a" printer value
    in
    name, value

Although the "%a" argument still makes me think I’m doing something wrong…

1 Like
3

Indeed, your solution is a good one :slight_smile:

If you wanted to avoid using asprintf (and the performance cost of interpreting a format string), it’s also possible to print a value to Format.str_formatter and then get the string with Format.flush_str_formatter. The implementation of this would then be nearly identical to your original solution, but more ergonomic.

The str_formatter is occasionally useful when you don’t have a convenient handle on a specific pretty-printer and value. I also think you are right to be suspicious of the single "%a" format string; I’ve come to see this as a code-smell that indicates there’s a nicer solution somewhere.

3 Likes
4

Is it possible to share for you? We could collaborate or least share the implementation.

Was about to write a json log reporter for the same library. Datadog as target too. And it is just overhead for me currently.

5

Note that format strings are parsed at compile time, e.g.

utop # ("%s = %a" : _ format);;
- : (string -> ('a -> 'b -> 'c) -> 'b -> 'c, 'a, 'c) format =
CamlinternalFormatBasics.Format
 (CamlinternalFormatBasics.String (CamlinternalFormatBasics.No_padding,
   CamlinternalFormatBasics.String_literal (" = ",
    CamlinternalFormatBasics.Alpha CamlinternalFormatBasics.End_of_format)),
 "%s = %a")
1 Like
6

Indeed. By ‘interpreting’ I was thinking of pattern matching on the underlying GADT :slight_smile:

7

Our log integration is currently tied into a lot of stuff. I’m hoping to make it a lot more generic where we’re just setting up outputs for other log libraries but right now it’s not really something we can easily separate.

We basically have our own entire log system and just add integrations for others that redirect to it. We also have a thread-local tag system so we can do something like:

Logger.with_tags [ "url", "https://www.example.com" ] @@ fun () ->
Log.Global.info "This will have the tag created above"

In case it’s useful to you, our Async.Log integration looks like this:

  Log.Global.set_transform
    (Some
       (fun msg ->
         (* synchronously add thread-local tags to the message. If we do this later, we will frequently
            get tags from the wrong async context.
            See https://github.com/janestreet/async_unix/issues/16#issuecomment-620830885 *)
         let open Log.Message in
         let level = level msg
         and raw_message = raw_message msg
         and tags = tags msg |> Logger0.merge_thread_local_tags
         and time = time msg in
         Log.Message.create ?level ~tags ~time raw_message));
  Log.Global.set_level Config.(get log_level |> log_level_to_async_log_level);
  Log.Output.create
    ~flush:(fun () -> Lazy.force Writer.stdout |> Writer.flushed)
    (fun msgs ->
      Queue.iter msgs ~f:(fun msg ->
          let module Message = Log.Message in
          let time = Message.time msg
          and message = Message.message msg
          and type_ =
            Message.level msg
            |> Option.map ~f:log_level_to_async_log_level
            |> Option.value ~default:Info
          and tags = Message.tags msg in
          save_and_log ~type_ ~time ~tags "%s" message)
      |> return)
  |> List.return
  |> Log.Global.set_output;

And our Logs integration looks like:

  let report src level ~over k msgf =
    msgf
    @@ fun ?header:_ ?(tags = Logs.Tag.empty) fmt ->
    Format.kfprintf
      (fun fmt ->
        Format.pp_print_flush fmt ();
        over ();
        k ())
      (Format.make_formatter
         (fun str pos len ->
           let type_ =
             match level with
             | Logs.Error -> Error
             | Warning -> Warn
             | App | Info -> Info
             | Debug -> Debug
           and tags =
             let tags =
               Logs.Tag.fold
                 (fun (Logs.Tag.V (def, value)) acc ->
                   let name = Logs.Tag.name def
                   and value =
                     let printer = Logs.Tag.printer def in
                     Format.asprintf "%a" printer value
                   in
                   (name, value) :: acc)
                 tags
                 []
             in
             if List.exists tags ~f:(fun (key, _) -> String.(key = "source"))
             then tags
             else ("source", Logs.Src.name src) :: tags
           in
           save_and_log ~type_ ~tags "%s" (String.sub str ~pos ~len))
         ignore)
      fmt
  in
  Logs.set_reporter { report };
  Config.(get log_level)
  |> type_to_logs_level
  |> Option.return
  |> Logs.set_level ~all:true

At some point we’ll probably change our log system to be based on Logs and just have an integration for Async.Log, but it’s not a high priority for us at the moment.

1 Like