Per-domain memory allocation limits?

My understanding is that garbage collection (and thus, allocation?) is largely isolated within each domain in multicore OCaml. To quote @kayceesrk et al.'s Retrofitting Parallelism onto OCaml paper:

Sweeping is not idempotent. Instead, we ensure that the areas swept by different domains are
disjoint: each domain sweeps only the memory that it allocated, keeping the interaction between
sweeping and allocation local to a domain.

Given this, I would like to be able to place limits on the amount of memory that a domain can allocate.

For context, I have a set of OCaml libraries/programs that I would like to delegate work to, but in a supervised fashion such that they do not exceed reasonable limits, especially with regard to memory allocation. The realities of malicious external inputs and extant bugs make unsupervised operation an obvious safety risk, especially in a public / multitenant context.

I could use something like e.g. systemd-run (or cgroups, or any of another dozen operating system-level facilities) to set these constraints on a per-process basis, but of course that entails all the usually clumsy IPC that goes along with invoking external processes. Since OCaml domains track the memory they allocate, and of course have ownership of their underlying system thread, it seems plausible that one could spawn domains with constraints set on thread priority and heap allocation (similar to e.g. Erlang’s spawn_opt), and thus avoid the friction of the aforementioned IPC.

I’m sure something like this has been considered somewhere at some point; I wonder if anyone has concretely experimented along these lines?

In the meantime, I see that Gc.allocated_bytes has been extended to be domain-aware, though its documentation is confusing to me:

Return the number of bytes allocated by this domain and potentially a previous domain.

Could someone clarify what “and potentially a previous domain” means? I interpret this to mean that domain A might “take ownership” of memory allocated by another domain B after the latter has completed via Domain.join, but that’s just speculative. If that’s the case, it seems that one could supervise memory allocation and terminate domains that exceed some threshold (although that begs the question of domain termination, which I gather remains an open subject itself).

Could someone clarify what “and potentially a previous domain” means?

Currently, there is a fixed number of domain slots that are available in the compiler. A running domain must inhabit one of those slots. By default, the compiler imposes a limit of 128 domain slots. This determines the maximum number of domains that can run concurrently. However, when a domain terminates, then a new domain may reuse the slot used by the terminated domain. This allows the program to spawn more than 128 domains if not all of them are active at the same time. The GC stats associated with the domain are sticky to the slot so that the stats remain even if the domain has terminated. Hence, the comment.

There is ongoing work to remove the restriction on the maximum number of domains and the ability to (eventually) have per domain stats. See

tl;dr there is more work necessary before we can implement per domain memory limit.

1 Like