Using Format/Fmt/Lwt_fmt to write on several formatters?

Learning
1

Hello,

I’m trying to improve a weird logging library.
Here’s how it works :

  • there are several logging levels, let’s just assume 2 (Normal and Verbose) for the sake of simplicity.
  • when i call log_normal <string> it writes the string in the normal log file.
  • when I call log_verbose <string> it writes the string in the verbose log file AND the normal log file.

So instead giving the verbosity level as a parameter, the logging library just writes in several files and then the user can chose which to open. (Consider that choice out of my reach)

However, building the strings is extremely long. So there’s a trick involving lazy strings, and a flag that silences the logs. Therefore the strings are only built when it’s not on silent mode. When the logs are active, the program takes at least twice the time to run than in silent mode. It can go up to 5 times depending…

So I want to move to :

  • using Fmt instead of stringifiers so that we are not building huge strings by concatenation but simply writing as we go
  • using Lwt_fmt to have some non-blocking wirting for free.

The issue is, in the previous version, once the string is built, we can write it in several files. So there is a single call to the stringifiers.
In this version, I don’t see how I could call the pretty-printers a single time to write into several files using Lwt_fmt.fprintf. I could write into a string and then write it in several files but I would lose the interest…

If anyone has an idea, it would be great ! Thanks :slight_smile:

2

I don’t understand exactly what you loose here. Basically use Format.kasprintf and use the log in the continuation. Doesn’t that work ?

3

You might be right, I’m not entirely clear on what’s happening there.
But if I use kasprintf, then it will write the entire thing into a string, and then the entire string into several files. There’s an additionnal step that we would like to avoid no ?
If I use fprintf and give the (Format.formatter_of_out_channel oc_normal), it will skip the first step of writing into a string and directly write into the file, won’t it ?

4

Well somehow you don’t really have the choice.

Either you perform the formatting (which can be expensive) once and then write it to multiple files. In which case you use a temporary string.

Or you perform the formatting directly on the files. But then you perform it on as much output files as there are. I suspect this will be much slower (but do measure).

Btw. you may want to inspire yourself from what is done in Logs. The logging scheme used there may feel a bit weird but it was carefully measured by @drup and @yallop to be as low-overhead as possible.

2 Likes
5

Thank you very much for your answers ! I got the answers I needed

I am aware of Logs, and in an ideal world I would love to use it directly ! But I think my team is too used to our current logging system to change to that.

1 Like
6

In case you are interested here is the discussion that lead to Logs’ logging structure.

3 Likes
7

Oh cool !! Thank you !

8

If someone ever reads this thread and is interested in a follow-up :

  • The overhead cost of the Logs-style continuation trick is smaller than the lazy trick. But not by far, about 2-3%.
  • The cost when logging is not silenced seems to be much much more expensive however…
2 Likes