The Unix System Programming in OCaml book mentions that signal handlers installed by the user will only be invoked at certain “checkpoints” corresponding to, among other things, system calls. In practice, different system calls interact with signals in different ways. For example:
Long-running Unix.select calls fail with EINTR after a signal handler runs
Unix.sleep calls continue to run concurrently with signal handlers
Blocking Mutex.lock calls prevent signal handlers from running until after the mutex is acquired
Of these, the first two are in line with C’s select and sleep calls. However, the last behavior is surprising to me: unless I’m mistaken, C allows signal handlers to run concurrently with pthread_mutex_lock calls, just like with sleep.
I have some signals that need to be handled in a timely fashion, so I’ve resorted to wrapping locks with Unix.sigprocmask to make sure that they don’t get queued up behind a long mutex wait:
Threads and signals tend to get messy (IIRC there’s no guarantee to which thread a signals gets delivered if more than one thread accepts them). In general I think you are better off choosing one thread (e.g. the main one or a dedicated one like in this example) that will handle them. So I’d say yes.
Signal handlers may not necessarily run on the thread that received them, OCaml records that a signal was received, and the handler is run whenever any OCaml thread reaches a point where it checks for pending signals (even if you specifically target a signal to a particular thread by carefully manipulating signal masks).
So even though your thread A is blocked waiting on a mutex, that should imply that there is another OCaml thread B that is currently holding that mutex… and eventually there’ll be an OCaml thread C that is holding a lock and is either running, blocked in a system call, or blocked in a C call. If it is blocked by a system call, that should get interrupted, which would give the handler a chance to run (even if it is Unix.sleep, it should check for pending signals before it loops around on EINTR). If it is running then when it’ll eventually check for pending signals and run its handler. Long running C library calls can be more problematic if they handle EINTR internally without calling any of the OCaml runtime functions that check for pending signals.
I think the only way to know for sure that there will be an OCaml thread available to handle signals is to have a dedicated thread that is always idle and just waits for pending signals (e.g. call Unix.pause in a loop).
I agree with having a spare/dedicated thread to ensure signals can be handled quickly.
The thread signal mask is not so straight foward: due to the way OCaml handles signals it can in fact slow things down because then you may need to wait up to the 50ms timeslice expiry before OCaml switches to executing your thread and running the handler, whereas otherwise – when not blocked by a mutex – it’ll handle it sooner, so it is probably better to allow all OCaml threads to handle the signal.
This is of course all theory, it’d be interesting to measure the delay between:
receiving a signal, and caml_record_signal running
caml_record_signal and the actual signal handler running
I would’ve suggested to use perf probe -x to measure those, but I just tried and it doesn’t work at all on OCaml programs (the probe gets registered, but the perf event is never triggerred), so might have to measure this some other way (e.g. record a high precision monotonic clock timestamp when sending the signal in one process, and in the handler in the other process).
If you can describe your use-case, it might be interesting for us to learn about. I know that in the Java world, using signals at all was highly frowned-upon (for the reasons @edwin mentions, but more generally b/c signals are a generalization of the control-C at the terminal, and not really a proper way to do IPC of any sort). In a multi-threaded runtime, they’re really pretty frowned-upon, at least in my experience. That is to say, the only good thing you can really do when you get a signal, is to die as hastily as possible.
This makes sense, but it seems like the runtime isn’t doing a great job with system calls that do not get interrupted by signals, including the ones backing Domain.join and Mutex.lock (both these pages have a note about EINTR specifically). Or maybe I’m barking up the wrong tree entirely, but at any rate, there’s something going on with my code that makes signals get handled faster when I block them strategically. I put up a snippet here that illustrates the issue, though I can only hope it will reproduce on any system but mine.
I am learning about OCaml 5 features by writing my own toy actor-model-style scheduler and effects-based IO system. The scheduler spends its idle time in select calls, so I have a SIGCHLD listener that does “the self-pipe trick” to let the system know when it needs to go clean up a child. But I can achieve the same thing by dedicating a thread to calling Unix.wait in a loop. Until now I’ve been avoiding dedicating threads to the scheduler (the worker threads all operate cooperatively on shared data structures) but I don’t think that actually achieves anything other than making me feel clever.
Maybe I am wrong, but it seems to me that, when you are not blocking signals on the mutex-waiting threads, one of them might end up receiving the signal (instead of the select one). Since, as you noticed, during the process of mutex locking, signals are not handled by the operating system, it means that the signal has to wait until the locking completes.
Something to note is that the thread running the OCaml signal handler in general need not be the same as the one running the POSIX signal handler (unless you use signal masks to direct your signal). (They do, however, share the same notion of signal mask.) Temporarily blocking signals during Mutex.lock is therefore not supposed to result in improved behaviour in general. In both cases the OCaml signal handler should be executed by any thread making progress and not masking the signal.
However, you mention that you are using OCaml 5.1.1. There were bug regarding this, that we fixed in OCaml 5.2. Do you still observe the same behaviour with OCaml 5.2?
Domain.join and Mutex.lock are not interruptible, probably because the POSIX spec does not make this easy to implement in the OCaml runtime (they restart automatically on EINTR). This is unlike Unix.sleep that was made interruptible by reimplementing it with nanosleep (which returns EINTR). (Also perhaps because it is not necessary to make join and lock interruptible for the use-case of “let’s send Ctrl-C to all threads” without preexisting programming errors.) It should be possible to implement interruptible mutexes using binary semaphores—but this is probably not what you need here.
I think it’s a bit different. pthread_mutex_lock does run C signal handlers while waiting, letting the OCaml runtime record the signal. But this might be the 5.1 bug whereby only the current domain is notified of the signal, or another bug or limitation (fixed or not since). In this case, it can take up to a full minor cycle to run the OCaml signal handler from another domain. (This is consistent with the signal being treated slowly, rather than not being treated at all.)
I still see the behavior on 5.2, yes (my apologies that you had to ask - I thought I told opam to upgrade me to latest before I posted anything). Should I open an issue on GitHub?
Thanks, it helps knowing that this behaviour happens with 5.2. I read your snippet more carefully this time and it seems that all three threads are blocked inside system calls. Let me know if I missed anything, but in this case there is no thread making progress that could run the OCaml signal handler. Now you have a main domain waiting inside lock/join when it receives SIGCHLD. With the signal mask, you are instead directing the signal towards the select command, which now gets interrupted with EINTR.
I agree with the two initial replies:
Blocking signals during non-interruptible calls (namely lock and join) is not too bad as a way to ensure that another blocked system call receives EINTR. (This does not solve the problem if all threads are inside non-interruptible system calls—but with lock and join getting into this situation is a programming error.) You can call trylock first to avoid extra system calls in the fast path. Note also that sigmask runs the signal handlers it unmasked, so there is no additional delay in the worst case.
Having a dedicated thread as explained above.
In addition:
There is a third solution for this use-case (writing to a pipe), which is to set a POSIX signal handler directly from C, bypassing the OCaml signal handling mechanism. There is nothing wrong with this approach, especially if one has POSIX semantics in mind.
As said, it is possible to implement an interruptible mutex, that runs OCaml signal handlers. But I do not know how to implement an interruptible join.
for forked child processes from a heavily multithreaded process, can I suggest use the “spawner” trick? I first encountered this working on one of the first multithreaded web servers (circa 1997) but Stu Feldman informed me that he did the same thing for make back in the day (which was a -long- time ago, or so I remember).
The idea is you fork/exec a process (think “remora fish”) early in the lifetime of your big multithreaded process (think “whale shark”), and the remora communicates with the whale shark. When the whale shark wants to fork/exec a process, it communicates with the remora, sending it the args, maybe sockets for communications (using PASS_FD) and the remora does the fork/exec. Since the remora is single-threaded and small, it’s efficient and doesn’t involve having to “park” a bunch of threads possibly across multiple cores.
for I/O I don’t recall ever using SIGPIPE: always, my memory is you arrange for sockets to return an EPIPE error on failed write.
Oh, I understand better now - I was thinking that if the runtime were smarter it could, as part of its internal signal handling, direct all signals away from threads making uninterruptible calls, ensuring the EINTR result without manual blocking. But now I realize that EINTR can only be generated by the system’s runtime, not OCaml’s, so it’s already too late to interrupt the select by the time things have gotten that far.
Both of your proposed workarounds are interesting, thank you! Stdlib semaphores appear to be implemented on top of mutexes so I’ll need to learn how to write and call external C functions in either case, but I guess that’s what this project is for.
I was thinking that if the runtime were smarter it could, as part of its internal signal handling, direct all signals away from threads making uninterruptible calls
It could be smarter, if we had the idea of blocking signals during these calls.
I don’t understand the tradeoff here: why is “a bunch of threads” the alternative? Why not implement the remora as a single thread in the main process where it already has access to your file descriptors and whatnot?
At least back when I was doing this stuff in anger, a fork() requires that the parent process “park” (that is, stop) all its threads, then execute the address-space clone, then get its threads moving again. It’s a heavyweight operation. By moving that fork() to a small child process, you remove all that work. It makes fork()s cheaper and faster. As a side-effect, the main process sees child-process-death as an I/O event, not a signal.
That probably reflects the fact that the only thing a child process can do after a multi-threaded parent forks is apply async-signal-safe functions and then replace its process image with exec*(). That’s in C. In ocaml you can’t do that. According to the documentation for Unix.fork:
It fails if the OCaml process is multi-core (any domain has been spawned). In addition, if any thread from the Thread module has been spawned, then the child process might be in a corrupted state.
If you are on Linux you can also use signalfd to turn a signal into an event on a file descriptor, you can probably find bindings for it in extunix.
And when you’re not on Linux you could spawn the extra thread that does the self pipe trick.
Although I’d avoid using Unix.select: it’ll fail if your program wants to use more than 1024 file descriptors. Better use something like iomux which supports poll (and ppoll, which allows you to manipulate the thread’s signal mask if needed, although as discussed it shouldn’t be), otherwise it’ll be harder to redesign your scheduler later around efficient use of poll-like functions.
Not sure what “that” refers to, but probably to “park all threads” ? It’s true that the child process inherits only the main thread, not all the other threads. Hence, unless there’s only one thread, the child address-space is almost certainly corrupt in some way, and the only useful thing it can do is to immediately exec().
Yes that’s what “that” referred to. Incidentally, apart from the problem with Unix.fork a child process of a multi-threaded OCaml program cannot apply Unix.exec* either, because the implementation of those can apply malloc(), which is not async-signal-safe.
Unix.create_process in a multi-threaded or multi-domained process is OK though. But I think this may be straying off topic.
Sure, that makes sense (“Unix.exec” and malloc()). I was speaking in terms of the C syscalls, not the OCaml entry points. AFAIR(emember) exec(2) and variants don’t malloc. But hey, I could be misremembering.
