Hello! I am looking to use logs and am wondering if there’s any way to print a log message without a newline? It seems to be (by default) automatically ending all messages with a newline. My use case is that I want to log HTTP requests for a server, and I log part of the request as soon as I parse it, and the other part later after I have the response. For example:
GET /header/user-agent 200 127.0.0.1
\ / \ /
------------------- ----------
print first print later
Thanks. If I understand you correctly, I would create a ‘no newline’ reporter, set it as the reporter for the first portion of the log message, print that, then set the standard reporter as the reporter for the second portion of the log message, then print that?
No you create a no newline reporter and write newlines in your log messages to have control over newlines.
That being said I doubt a bit the original aim. I’m not sure who is going to watch these “partial” log lines (besides you are going to get into trouble as soon as some form of concurrency quicks in).
Ah, good point. I can explicitly put in the newlines.
And, I’m not too sure about how logs works but the way I think of them, they’re not actually ‘partial’ log lines, they’re full log lines which are printed out piecemeal. Re: concurrency, I believe I have that under control (currently using Printf.printf) but as for the overall design, maybe it can be improved to not need this piecemeal logging.
I’m assuming that you just don’t have a convenient place to assemble a complete log-line? That is, for modularity purposes, the various bits of data that need to be put together in a log-line, don’t share a sort of “context”? Perhaps you might think of using threadlocal state for that, then? So you initialize it at the beginning of processing … actually, -before- processing begins (so you don’t have to be really specific as to where) and then after processing is finished, you flush that threadlocal buffer to the logger ?
An important thing about logging has always been that log-lines are terminated somehow, so that they can be inspected/processed by third-party tools. Breaking that would be … fine for a development hack, but bad for any sort of code/system that is intended to be consumed by others. Logging is for debugging (amongst other things) so the last thing you want, is the possibility of producing malformed log-lines.
Currently my ‘context’ is the standard output stream By using Lwt_io.printf/print/print_endline, I can easily print logs piecemeal and terminate the log-line precisely by endling the final piece with a newline character.
Exploring logs to see if it provides something I really need; right now I’m not sure what that could be other than, maybe, more granular logging levels than just ‘info’ and ‘error’. I really like the simplicity of stdout and stderr, though…
Using logs has other benefits too though. Consumers of your library can control the log reporters, what log levels get filtered out, etc (These are some decisions that are better made by the application, instead of a library in my opinion).
Logs can be a little granular though. As an application consumer, one can decide to add their own metadata to the libraries log output by combining tags + reporters, use different reporters for different libraries, all without having the library needing to be changed to meet these needs.
That being said, i’m not trying to convince you to switch, it was just an attempt to bring up some things that a library like logs brings over just using printf/print_endline
I guess you can take this for what it’s worth; I offer it only in the spirit of “old guy who got burned a lot …”
The probity (trustworthiness) and (controllable) copiousness of log-lines is absolutely essential for debugging. For my entire commercial career, I never used a debugger except on perl scripts – because in all the runtimes that matters (e.g. Java/J2EE) if you used a debugger, you’d already perturbed your program’s execution enough that the bug vanished. For many, many significant runtimes, controllable-at-runtime logging (so, you can turn on and off log-lines at runtime without recompiling) is the only tool for debugging and program-understanding. Even when a program is run on the command-line, logging is an invaluable program-understanding tool.
Viz. recently I was trying to figure out where opam cached its tarballs. I was unable to figure it out, partially b/c I couldn’t coax opam to log the (nontrivial) UNIX commands it executed (e.g. tar, copying tarballs, etc). That was disappointing.
There’s an excellent paper from Google, about the Dapper system, which is worth reading. Dapper is basically “controlled logging” + “very interesting post-processing”.
To sum up: it’s really, really important that log-lines be trustworthy. 100% trustworthy. The standard way to ensure this, is to mandate that each log-line be presented all-at-once, and to format it into a buffer before writing it to storage/network And to ensure that logfiles can be -parsed- without recourse to anything special. Parsed without errors, and this in the face of faults of all kinds. If your logs aren’t trustworthy in faulty environments, they’re useless.
This is a very compelling argument. Thank you. Any serious codebase does end up full of ‘debug’ level log messages. And it’s true that on my project, I spent a fair bit of time debugging with print statements.
If you’re going to be building a system for anything close to real use, I feel like there’s one other piece of advice I ought to offer:
“The test of a really good log system is the cost for a log-line that is -not- triggered at runtime, but -is- available to be enabled at runtime without recompilation”. The idea being:
(1) if the cost of a log-line that is compiled-in, but not enabled, is very low
then (2) you can put these ALL OVER your program – all over.
and (3) you can then enable them when you need to debug.
So for instance, if a “disabled log line” requires a string-comparison to decide not to log, that’s a non-starter. If it’s an integer comparison, that’s great!
Of course, this sort of assumes that one can enable/disable log-lines on a module-by-module basis, not merely by verbosity level. The “glog” package (in several languages IIRC, but the one I looked at was in golang (spit, sigh)) was quite good on the above-described metric.
You have to enable verbose mode (--verbose, -v, -vv, etc.), and opam displays all commands it launches. First level is just for commands, second one with their output, etc.
And, if you didn’t get an answer/find yet, tarball cache is in <opamroot>/download-cache directory, stored by their hash —not human friendly to dig in.
