No at_init in Stdlib.Domain?

Sorry to come up late with some design opinion, but:
There is Domain.at_exit, but not the opposite at_init.
at_exit is a “kind of per process” cleanup function.
So, the “kind of per process” init function is not provided.

For example, some users like to setup logging differently in each parallel worker.
Maybe @Julia_Lawall will have an opinion on this design decision.

Cf. Parmap for init and finalize functions.

We removed the corresponding function (Domain.at_each_spawn) when we realized that its specification was tricky and that we did not know how to use it correctly. (The one use to setup threads was in fact buggy.)

stdlib: remove `Domain.at_each_spawn` by Octachron · Pull Request #11595 · ocaml/ocaml · GitHub

We can consider adding things back in further 5.x releases, but they should come with a good argument that they ar needed.

Instead of at_each_spawn it is often possible to use the Domain.DLS for domain-local storage. if domains use per-domain resources, their state can be stored in domain-local storage, and DLS provides an API to compute a new domain’s state either eagerly (from the parent, when spawning) or lazily (from the child, on first request0.

Note: Domain.at_exit only runs the callback at the exit of the current domain, it does not affect other domains. There is no symmetric function at_init because it does not make sense to try to run a function when the current domain starts, it has already started.

The documentation for DLS says this, which is ok:

 Note that the [split_from_parent] call is computed in the parent
        domain, and is always computed regardless of whether the child domain
        will use it. If the splitting function is expensive or requires
        client-side computation, consider using ['a Lazy.t key].

But then the docs of Lazy say they are not thread or domain-safe:

  Note: [Lazy.force] is not concurrency-safe. If you use this module with
    multiple fibers, systhreads or domains, then you will need to add some
    locks. The module however ensures memory-safety, and hence, concurrently
    accessing this module will not lead to a crash but the behaviour is
    unspecified.

And it was my general understanding before as well that Lazy is best avoided in multi-threaded programs unless you’re really really careful. Maybe I am missing something, but how can a lazy value be used safely here, because you’d have to setup (create) the lazy value in the parent domain, and force it (access it) in the child domain, which would seem to be against the only 1 thread/domain can safely access a lazy value restriction.
Although the value is returned by the split_from_parent function and only forced in the child domain, there might still be some lingering references in the parent domain’s minor heap for example that I wouldn’t have any control over as a developer and the child domain may update the lazy value concurrently with the parent domain’s (minor) heap being sweeped.

It might be better if the lazy value is managed internally by the OCaml runtime, e.g. through a 'a -> unit -> 'a style API, and the implementation of domain.ml can do the necessary things to convert that to a lazy value, and ensure the lazy value obeys all the safety rules required.

Or does the OCaml runtime move any (lazy) value returned by `split_from_parent’ from the parent domain’s (minor) heap to the child domain’s heap, including any transitive references, such that the parent has no more references to the lazy value at all?

If I have to do such lazy initialization today, then I think might use an 'a option ref' instead, which although has more overhead, I think I'd understand how to safely use in split_from_parent`.

It’d be nice to have a safe lazy initialization in DLS (or the above at_init function, but they’re not completely equivalent).

When we say that a value is not concurrency-safe, we mean that it is unsafe to access it concurrently from two domains, one of the accesses being a mutation. Using it from (only) one domain at first, and then from (only) another domain later is always safe, for any OCaml value. This is an ownership-passing idiom.

In the case of Lazy, it would be unsafe to call Lazy.force concurrently on the same value from several domains. If you use Lazy.force (Domain.DLS.get key), then only one domain forces each domain-local value, so this is safe.

Extra comments.

@edwin you seem to have a mental model where each domain has its private minor heap that other domains should not access. This was the design in the beginning of the Multicore OCaml project, but now this is not the case anymore, all domains can access minor values in all other domains, so you can think of the minor heap as global (even if each domain works on a separate portion of it). This should be irrelevant to reason about the correctness of manipulating OCaml values from OCaml programs, however.

there might still be some lingering references in the parent domain’s minor heap for example that I wouldn’t have any control over as a developer and the child domain may update the lazy value concurrently with the parent domain’s (minor) heap being sweeped

It is the runtime’s responsibility, not the programmer’s responsibility, to ensure that concurrent accesses caused by the runtime itself are safe. If a domain runs, say, a part of a major GC, and another domain acts on the values being traversed, any safety violation is a runtime bug. (There could very well be some such bugs left!) There is in fact careful logic to handle the “tag update” that happens on Lazy.force concurrently with GC marking.

Thanks for pointing that out, not the first time I made that mistake. I’ll go back and read the docs available in the current official 5.0 tree, and raise a separate topic to discuss any misconceptions (I’ll try to keep some notes along the way about things I thought were true, but no longer are, even though the internal implementation may still evolve to reason about safety of various pieces of code, and e.g. C bindings it is useful to understand how it all actually works, and what the latest memory model is).

Would the following be a safe way to use lazy with domains?

type 'a t

val dls_lazy_init: (unit -> 'a) -> 'a t

val dls_lazy_get: 'a t -> 'a
type 'a t = 'a Lazy.t Domain.DLS.key
let dls_lazy_init constructor =
    let split_from_parent _ =
        Lazy.from_fun constructor
    in
    Domain.DLS.new_key ~split_from_parent @@ fun () -> Lazy.(from_fun constructor)

let dls_lazy_get t =
    t |> Domain.DLS.get |> Lazy.force

This might be useful if you have a logger library like in the original question and you want to perform some per-domain initialization, but only if that logger is used in that domain.

I don’t have a too hard feeling on this one.
If it’s not there, people will learn to live without the functionality.

A good argument that such a function is generally needed would be if you
see it in the OpenMP or MPI specification.
Because, both of these libraries allow to do parallel programming and have
undergone heavy use and significant refinements.
I will ask some HPC people.

@edwin yes, this is correct, but you don’t need to do this. Just do not pass a split_from_parent value to the new_key call, and you will get a non-inheriting DLS value that is just initialized independently on each domain, on each access. split_from_parent is only useful if the value for a child domain depends somehow on the value of the parent domain.

(I sent a PR to clarify the documentation of DLS.new_key: clarify the doc of Domain.DLS.new_key by gasche · Pull Request #11814 · ocaml/ocaml · GitHub )