Hopefully this is a pretty simple question, with the first hunk of multicore support being submitted upstream in mid-2019, where is OCaml multicore development happening? Is it still at the ocamllabs repo? The last commit there is 2 months ago and looks like it might be in need of a rebase, which makes me a bit skeptical I fully understand the development picture here.
Thanks MLers!
There is a multicore repo over here
But I’m sure you meant something else. 
Indeed, I mentioned that in my original post. 
@brettgilio My theory is that you may (also) be asking politely about the status of multicore 
Indeed things seem to have significantly quietened on the ocaml github repo (at least from an external perspective) as far as multicore is concerned since https://github.com/ocaml/ocaml/pull/8713 was merged.
Regarding https://github.com/ocaml-multicore/ocaml-multicore I think this is still where multicore development is taking place because I see a recent PR submitted here:
https://github.com/ocaml-multicore/ocaml-multicore/pull/270
Request @stedolan @kayceesrk @gasche to add a few words if they feel so inclined – they definitely would be able to give an authoritative answer.
P.S. I’m not associated with the multicore effort so I’m not in a position to give any reliable answers
Here are two relevant posts which should answer all your questions:
Multicore is a really big change to the code base and cannot be merged in one go. This sort of change takes time.
Thanks for sharing these links @kit-ty-kate – I think @brettgilio may already be aware of this. The Multicore prerequisite patches appearing in released OCaml compilers now post is really mainly about https://github.com/ocaml/ocaml/pull/8713 (already mentioned above by me) but I think it would be useful to know if there is anything else after that.
Then again, this is an open source effort and I leave it to the people in-the-know to add any additional info if they can.
(I don’t personally have any more information than what was given in this thread. @kayceesrk has moved to IIT Madras and there seems to be new activity from students there working on the multicore-ocaml runtime; it may be that directing this work has taken some time away from upstreaming work.)
Greetings! I have indeed moved to IIT Madras as an Assistant Professor in Computer Science and Engineering. The demands of the new job meant that I could not contribute more actively to multicore. OTOH, I am building a nice team to hack on multicore at IITM along with the existing team spread across Cambridge and Paris.
Currently, our aim is to get the domains-only multicore (no effect handlers yet) upstreamed. PR#8713 is a step towards that. Our strategy is to carve out self-contained pieces of multicore runtime and upstream it to trunk in such a way that it is independently useful. No one likes to review thousands of lines of highly interconnected PR. However this has meant that there is more to do for the multicore developers to cleanly separate out the features from the multicore runtime to make it self-contained (and hence, more easily reviewable).
Here are some of the things we have been working on, which isn’t quite visible externally:
Event tracing: The aim of domains-only multicore is to make your programs go faster. If it doesn’t behave as you would expect, then we would like the developers to have tools to be able to debug it. Multicore has a great tracing framework aimed at performance debugging. There is an in-progress PR aimed at upstreaming this feature: https://github.com/ocaml/ocaml/pull/9082. You can see that this PR has been quite active. This work is being led by @Engil from Tarides, Paris.
Synchronized minor GCs: Designing performant GCs is a tricky business. Multicore OCaml uses a scheme where each domain can independently garbage collect its own minor heap without having to synchronize with other domains. While this is highly desirable, the downside is that reading object fields become a GC safe-point (a point where the runtime might potentially have to do a GC). This complicates the C API since every Field access now has to be a safe point. Hence, we are exploring an alternative GC design that performs synchronized, parallel, minor GCs. This design should obviate the need to change the C API, but also is expected to have better behavior in programs that share small sized objects between domains frequently. This work is being lead by Tom Kelly and Sadiq Jaffer from OCaml Labs, Cambridge. Some of the open PRs in multicore repo is due to this.
Mid-sized Benchmarks: In order to evaluate the performance of different GCs, and multicore in general, we’ve been adding new benchmarks to the sandmark benchmarking suite. We’re in the process of enhancing the sandmark suite to support the questions that we might want to ask about the multicore programs. See Sandmark repo for details: https://github.com/ocaml-bench/sandmark. This is led by my student Atul Dhiman at IITM.
Better Safepoints: We’re working on addressing issues with timely response to runtime interrupts: https://github.com/ocaml-multicore/ocaml-multicore/issues/187. A PR should be open upstream towards the end of Jan. This is led by Anmol Sahoo at IITM.
There are other developments RWO style chapter on multicore programming, fixing marshaling in trunk to be compatible with multicore, parallel version of Irmin, which are currently in development at IITM, for which I don’t have a link or a PR to show.
On the whole, while the developments may not be concentrated on the multicore repo, there is quite a good momentum building around the multicore ecosystem. I’m eager to see the domains-only multicore across the line in the coming months.
This being a part of the reason why I proposed for OCaml project to use GitHub Projects feature in their repository - #8734.
Moreover, if you move multicore projects under main OCaml GitHub organization, you can use the cross-repository, organisation-wide project boards, which exactly a good fit for Multicore, since requires many changes in the compiler itself and the ecosystem.
@kayceesrk, if I remember correctly, the safe-point only happens with mutable fields, right? So maybe it makes sense to have a minor heap that’s just for immutable values and runs asynchronously, and then a second minor heap for mutable values that collects synchronously? That way, the common allocations (from lists and such) don’t require full synchronization.
I would like to echo @XVilka’s perspective here. Is there a reason why the development isn’t more integrated in this fashion? @kayceesrk.
Thanks to everybody for the great detailed answers! I ask because I am an aspiring compiler hacker, with experience in other high level multicore languages and would like to dedicate some of my free time to helping here.
We’ve considered this exact proposal. We need to collect a bit more numbers about the distribution of immutable and mutable objects in representative OCaml programs.
Currently, the compiler has information about mutable fields, but does tag objects with mutability information (an object should be marked immutable if it does not have any mutable fields). Once we have this, we can analyse the distribution of objects to see if the proposed scheme would be profitable. I would use the Sandmark suite of benchmarks as representative benchmarks (and if you think this isn’t representative, you should contribute benchmarks! ).
I don’t expect the twin minor heap scheme to be slower, but we should know whether benefit we get from this is worth the additional code complexity. In any case, we only need to consider this if the current synchronized minor GC scheme does exhibit slowdowns.
People use tooling if they feel like it adds something to their personal work. I can’t help feeling it slightly inappropriate to ask other people to adopt new tools to make our lives easier rather than theirs.
(Personally I think that Github Projects would be fine if the multicore people wished to use them, but still not nearly as convenient as getting a good-quality status report every six months or so, as KC has just done. If someone is willing to maintain a wiki page or discuss post or reddit topic somewhere with links to the most recent reports, that would be just fine.)
Those are good ideas, @gasche. Thank you.
And reading the ocamlverse post again reminded me that there is a github project for multicore.
The items there are still relevant, but they need a bit of spring cleaning to make them organized. Also, internally, we’ve agreed to post monthly updates regarding multicore progress for the benefit of the community. We will start posting updates from the end of this month.
The update for January has been posted here: Multicore OCaml: January 2020 update