I have noticed that recent versions of Lwt issue a deprecation warning about the
Alert deprecated: Lwt_unix.set_default_async_method
Will be a no-op in Lwt >= 5.0.0. See
My use case for applying that function is to set the default async method for blocking system calls to
Lwt_unix.Async_none in some utility scripts which happen to use Lwt but which launch new processes using
Unix.create_process. This is because when I last checked (which admittedly was some time ago)
Unix.create_process calls up some functions between the fork and the exec which are not async-signal-safe and so is not thread-safe according to POSIX.
Does this deprecation imply that in OCaml-5 or recent versions of OCaml4,
Unix.create_process will now be thread safe? If not, is there intended to be any work-around for this case? I accept there won’t be many programs which use
Unix.create_process and Lwt at the same time, if only because blocking system calls and Lwt don’t make a good fit, but I am in the unfortunate position of having a couple of them of my own which I use, where the blocking occurs at times which are not problematic to the Lwt main loop.
For anyone else encountering this issue, with Lwt version 5.6.1 at least, setting the thread pool size to 0 with
Lwt_unix.set_pool_size seems to do what I want. That is to say, it seems to do pretty much the same as what
Lwt_unix.(set_default_async Async_none) used to do in Lwt < 5.
Lwt has Module Lwt_process, AFAICT it calls only
close between fork+exec, which is on the list of async-signal-safe functions (see fork and the table at the end of this section: General Information). Can you use that?
Also the latest version of
posix_spawn (and it is then up to your
libc to implement that in an async-signal-safe way). But even the one that doesn’t only calls
close, assuming you have an
execvpe in libc, and the only potentially async-signal-unsafe function would be in the execvpe emulation code. If you’ve spotted some other unsafe functions then that would probably be a bug worth fixing.
Thanks for your help.
My comments in the scripts in question from at least four years ago tell me that Unix.execv at that time allocated memory, which is unlikely to be async-signal-safe, and at that time I must presumably have concluded that Unix.create_process either applies that function or applies another Unix.exec* function which does so. I’ll look at the code again and see if it has changed – its seems from what you say that it probably has.
I cannot now remember why the Lwt_process module didn’t suit me. Possible reasons lost in the mists of time are that (i) the documentation on what amounts to a high level wrapper was somewhat lacking and required too much effort to sort out, whereas I am comfortable with wiring up pipes by hand, (ii) one of the processes to be launched involved mailx which required some fiddling with the closing of descriptors at the right time in order to get it to send correctly, or (iii) I wasn’t satisfied that Lwt_process’s spawning was POSIX compliant either.
Anyway, setting the number of threads in the pool to 0 seems to do the trick nicely. It would be good if Lwt continued to allow it: there can be other reasons why you don’t want some technically blocking but relatively transient operation to cause a thread to be started, particularly when (as in the case of my scripts) most of the code was i/o which doesn’t block but relies instead on Lwt’s poll/epoll wrapper operating on non-blocking descriptors. I can’t actually understand why it was necessary to fiddle with this, even after reading Deprecate the Lwt_unix.async_method machinery and eventually remove it · Issue #572 · ocsigen/lwt · GitHub .
 Obviously, the problem at that time was with OCaml’s wrapping of C’s execv, not with C’s execv. I think it constructed an OCaml string, or something of that kind, which would have required allocation.
On looking again at the source code, in OCaml-4.14,
Unix.execv and cognates are implemented by the stub functions
unix_execv() and equivalents which call up various functions which seem to end up at
caml_stat_alloc_noexc(), which in turn may call up
malloc() which is definitely not async-signal-safe as a matter of generality (although actually glibc will allow a call to
fork() and exec, whereas musl won’t).
I might well have missed something as I haven’t spent much time on this, but OCaml-4.14 doesn’t appear to use
posix_spawn to implement
Unix.create_process. That function appears instead to be implemented via the stub
unix_spawn(). That doesn’t use unix_exec* in the child process but instead calls up C directly except in one case (the OS not providing
execvpe) where it may apply
unix_execvpe_emulation() which applies
malloc() (the case you mentioned). So in the general case
Unix.create_process may be thread-safe, even though
Unix.execv and cognates in a child process are not. What do you think?
4.12 and later should use
posix_spawn: Reimplement Unix.create_process in C, using posix_spawn if possible · ocaml/ocaml@ba528fe · GitHub and thus
Unix.create_process should be safe there.
But you raise a good point:
Unix.execve is not thread safe, and that is exactly what Lwt_process uses: lwt/lwt_process.cppo.ml at master · ocsigen/lwt · GitHub, so using
Unix.create_process (provided you don’t hit the emulated path) should be safer than using
There is an issue already in the Lwt repo that Lwt_process is not multi-domain safe, I just added a comment now that it isn’t traditional multi-thread either given the analysis in this thread: Support `Lwt_process` in multi-domain settings · Issue #959 · ocsigen/lwt · GitHub
Would be good if a solution was provided by Lwt itself here (at least in the form of documentation) on what to use to be safe in a multi-threaded environment (which Lwt itself is, unless you use one of the 2 workarounds you mentioned - setting the default async method, or setting thread count to 0).
Also OCaml library : Unix should probably document the thread safety. It is quite a common trap to fall into if one just thinks that the usual C async-signal-safety rules apply for OCaml stubs of the same name.
A project I contributed to has a similar bug in its use of
Unix.execve as Lwt does:
xen-api/child.ml at master · xapi-project/xen-api · GitHub
ocamlnet does provide an implementation of spawn too: http://projects.camlcity.org/projects/dl/ocamlnet-3.6.1/doc/html-main/Netsys_posix.html (although it has a fallback to
fork+exec which is again quite a lot of code that I haven’t reviewed whether it is all safe).
There is also a
spawn package on OCaml that might be useful, but unfortunately it doesn’t call
posix_spawn, but reimplements it: spawn/spawn_stubs.c at master · janestreet/spawn · GitHub. That is quite a lot of C code, and I haven’t reviewed whether all the functions called there are async-signal-safe.
Your response is most informative. First, my conclusion now is that
Unix.create_process since OCaml-4.12 is thread-safe, though
Unix.create_process_env might not be if the OS does not provide
execvpe and doesn’t provide
posix_spawn. (I had missed the fact that there is a conditional compilation of
unix_spawn dependent on whether
posix_spawn is provided.)
Unix.exec* appears not to be thread-safe and this is not documented, and that function remains in use in the
So, dare I say it, it seems to me to be a good idea if
Lwt_unix.(set_default_async Async_none) were returned to its former glory, which is where this started!
In something like a webserver you want to avoid all blocking calls because it could cause delays to other clients trying to connect. In something like a library you want to avoid all blocking calls because your final user might be something like a webserver.
But in a script, you can just call a blocking function from
Unix and it’s fine. It’ll just block your program for however long it takes, which does prevent all other promises from making progress, even the ones that would be resolved by the OS (like
read). If that’s ok with you (i.e., if the blocking is short enough, if the other promises don’t do time-sensitive things) then go on and call those blocking functions.
This is not the same as setting
Async_none, but it could be a substitute in some cases.
Yes, in the script in question I tried that, but
ps axH at the command line showed that something was causing Lwt to launch a worker thread and rather than spend time tracking it down
Lwt_unix.(set_default_async Async_none) resolved the issue. Now that
Unix.create_process is thread-safe the point would be less pressing.
This github issue appears to be another example of a case where
Lwt_unix.(set_default_async_method Async_none) is necessary: forking breaks lwt-io if happens after Lwt_main.run · Issue #970 · ocsigen/lwt · GitHub
More generally, given that Lwt by default starts worker threads on encountering blocking system calls, and
Lwt_unix.(set_default_async_method Async_none) has been removed, what is the point of