# Dune and Multicore **URL:** \ **Category:** Community\ **Tags:** multicore, dune\ **Created:** [February 10, 2019, 4:49am UTC](https://discuss.ocaml.org/t/dune-and-multicore/3315 "2019-02-10T04:49:57Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![jhw](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/jhw/32/1639_2.png) [@jhw](https://discuss.ocaml.org/u/jhw)\ **Post date:** [February 10, 2019, 4:49am UTC](https://discuss.ocaml.org/t/dune-and-multicore/3315/1 "2019-02-10T04:49:57Z")
For grins, I updated my _multicore_ OPAM switch and started looking into what is currently keeping my Orsetto project from compiling there. At the moment, the answer is that Dune doesn’t compile. Which I suppose is not really noteworthy, because multicore is still way way off, but I was vastly amused by the log message that appeared on my console: > I must not segfault. Uncertainty is the mind-killer. Exceptions are the little-death that brings total obliteration. I will fully express my cases. Execution will pass over me and through me. And when it has gone past, I will unwind the stack along its path. Where the cases are handled there will be nothing. Only I will remain. I don’t know who wrote this, but it has me rolling on the floor— especially given the irony of this happening on a compiler that’s trying to introduce algebraic effects. Just wanted to raise a glass for that here. Bravo, whoever you are. You have won the Internet today. ---
**Author:** ![rgrinberg](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/rgrinberg/32/40_2.png) [@rgrinberg](https://discuss.ocaml.org/u/rgrinberg)\ **Post date:** [February 10, 2019, 5:46am UTC](https://discuss.ocaml.org/t/dune-and-multicore/3315/2 "2019-02-10T05:46:25Z")
I believe the credit goes to someone at Jane Street (@jeremiedimino should know who). Recent versions of dune do not work on the multicore fork because it doesn’t have threads AFAIK. Dune 1.2.1 is the last version to work on the multicore fork. ---
**Author:** ![jeremiedimino](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/jeremiedimino/32/699_2.png) [@jeremiedimino](https://discuss.ocaml.org/u/jeremiedimino)\ **Post date:** [February 11, 2019, 8:12am UTC](https://discuss.ocaml.org/t/dune-and-multicore/3315/3 "2019-02-11T08:12:15Z")
Credit goes to [David Powers](https://blog.janestreet.com/author/dpowers/) 🙂 BTW, you the version of OCaml used to build dune doesn’t need to match the version of OCaml you want to build projects with. In short, you can build a dune with a vanila 4.07 compiler and use it to build OCaml code using a multicore OCaml. Not sure this will work with opam though, you might have to setup your own workspace by cloning all the various dependencies and letting dune build everything at once. ---
**Author:** ![jhw](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/jhw/32/1639_2.png) [@jhw](https://discuss.ocaml.org/u/jhw)\ **Post date:** [February 11, 2019, 2:42pm UTC](https://discuss.ocaml.org/t/dune-and-multicore/3315/4 "2019-02-11T14:42:58Z")
Yeah, one could do that. I feel like the best way I can help the multicore effort is to help make patches to OPAM packages as necessary to adapt to the forthcoming language changes and stage them in the multicore repository. I did that already for some of the PPX tools while Dune was still working. Next I’ll need to think about whether to make Dune work by pinning it or patching it. Not sure how much of OPAM is already dependent on versions of Dune that are broken under multicore. ---
**Author:** ![jeremiedimino](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/jeremiedimino/32/699_2.png) [@jeremiedimino](https://discuss.ocaml.org/u/jeremiedimino)\ **Post date:** [February 11, 2019, 4:05pm UTC](https://discuss.ocaml.org/t/dune-and-multicore/3315/5 "2019-02-11T16:05:44Z")
It seems to me that the best way to go about it would be to get the `threads` library to work with multicore, then everything using threads, in particular dune will work again. ---
**Author:** ![dra27](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/dra27/32/5294_2.png) [@dra27](https://discuss.ocaml.org/u/dra27)\ **Post date:** [February 12, 2019, 7:56am UTC](https://discuss.ocaml.org/t/dune-and-multicore/3315/6 "2019-02-12T07:56:37Z")
Odd though it may sound, getting `threads` working is low down the multicore roadmap because it will always be a bad library to use! I spent a few delightful hours on Sunday hacking both Dune and multicore - Dune would need a small `select`ed backend to switch between `Domain.create` and `Thread.create` and multicore needs `Unix.open_process*` and `Unix.create_process` not to use `Unix.fork`. I made preliminary versions of both these changes, but I’ve hit a segfault when the process watcher domain attempts to report back to the main domain that a process has terminated. @stedolan and I are going to have a poke at that later in the week, hopefully… ---
**Author:** ![jeremiedimino](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/jeremiedimino/32/699_2.png) [@jeremiedimino](https://discuss.ocaml.org/u/jeremiedimino)\ **Post date:** [February 12, 2019, 9:32am UTC](https://discuss.ocaml.org/t/dune-and-multicore/3315/7 "2019-02-12T09:32:13Z")
Getting Dune well integrated with multicore seems good. I do feel like the same problem will arise with many other projects after Dune though 😕 > multicore needs Unix.open\_process\* and Unix.create\_process not to use Unix.fork Is it ok if the forking is done in the C stubs though? I’ve been thinking of vendoring [spawn](https://github.com/janestreet/spawn) into dune for a while, it might be worth looking into that. ---
**Author:** ![c-cube](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/c-cube/32/1727_2.png) [@c-cube](https://discuss.ocaml.org/u/c-cube)\ **Post date:** [February 12, 2019, 2:54pm UTC](https://discuss.ocaml.org/t/dune-and-multicore/3315/8 "2019-02-12T14:54:12Z")
What do you mean, a “bad library to use”? What about the programs which already use it (and don’t care about launching 50,000 threads)? I have several programs that use `Thread` because it’s currently the only way to have a modicum of concurrency without sprinkling a monad in 90% of your code, and I’d like these programs to continue working in the future… edit: sorry, that was unecessarily harsh. I overreacted. ---
**Author:** ![dra27](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/dra27/32/5294_2.png) [@dra27](https://discuss.ocaml.org/u/dra27)\ **Post date:** [February 12, 2019, 3:03pm UTC](https://discuss.ocaml.org/t/dune-and-multicore/3315/9 "2019-02-12T15:03:24Z")
Those programs will work - just not as well as if they were written to use multicore’s domains, but the intention (obviously) is that the performance would be no worse than if you weren’t using multicore OCaml. ---
**Author:** ![c-cube](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/c-cube/32/1727_2.png) [@c-cube](https://discuss.ocaml.org/u/c-cube)\ **Post date:** [February 13, 2019, 12:44am UTC](https://discuss.ocaml.org/t/dune-and-multicore/3315/10 "2019-02-13T00:44:23Z")
follow up question: could `Thread` become a thin layer on top of multicore’s domains (which are heavy threads, if I understand correctly) while keeping the legacy interface? ---
**Author:** ![Freyr666](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/freyr666/32/591_2.png) [@Freyr666](https://discuss.ocaml.org/u/Freyr666)\ **Post date:** [February 13, 2019, 6:54am UTC](https://discuss.ocaml.org/t/dune-and-multicore/3315/11 "2019-02-13T06:54:47Z")
The multicore roadmap says they want to reimplement threads on top of domains. Though is it safe, are the current programs rely on GIL? ---
**Author:** ![c-cube](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/c-cube/32/1727_2.png) [@c-cube](https://discuss.ocaml.org/u/c-cube)\ **Post date:** [February 13, 2019, 2:49pm UTC](https://discuss.ocaml.org/t/dune-and-multicore/3315/12 "2019-02-13T14:49:18Z")
You can only use the global lock as a substitute for proper mutexes if your critical section doesn’t allocate (or call functions?) at all, I think, since that’s where preemption can occur. In practice I use `Mutex.t` or `Condition.t` (or sane wrappers around them) whenever `Thread` is involved. ---
**Author:** ![Freyr666](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/freyr666/32/591_2.png) [@Freyr666](https://discuss.ocaml.org/u/Freyr666)\ **Post date:** [February 13, 2019, 3:22pm UTC](https://discuss.ocaml.org/t/dune-and-multicore/3315/13 "2019-02-13T15:22:25Z")
Does `ref` imply memory barriers? If so does OCaml has at least informal memory model? Locking everywhere in absence of atomics looks bad. ---
**Author:** ![jjb](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/jjb/32/2713_2.png) [@jjb](https://discuss.ocaml.org/u/jjb)\ **Post date:** [February 13, 2019, 3:38pm UTC](https://discuss.ocaml.org/t/dune-and-multicore/3315/14 "2019-02-13T15:38:35Z")
Re the memory model, see: kcsrk.info/papers/pldi18-memory.pdf ---
**Author:** ![bluddy](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/bluddy/32/104_2.png) [@bluddy](https://discuss.ocaml.org/u/bluddy)\ **Post date:** [February 13, 2019, 4:22pm UTC](https://discuss.ocaml.org/t/dune-and-multicore/3315/15 "2019-02-13T16:22:39Z")
Also see [this](https://github.com/ocamllabs/ocaml-multicore/issues/100). ---
**Author:** ![jhw](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/jhw/32/1639_2.png) [@jhw](https://discuss.ocaml.org/u/jhw)\ **Post date:** [March 14, 2019, 10:01pm UTC](https://discuss.ocaml.org/t/dune-and-multicore/3315/16 "2019-03-14T22:01:07Z")
For the record, pinning Dune down at 1.2.1 is a sadness for me, because too many of the things I care about have dependencies on things that need Dune 1.6 or later. Looks like if I want to mess around with multicore then I need to sort out this Dune and Multicore problem myself. Grmf. ---
**Author:** ![jhw](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/jhw/32/1639_2.png) [@jhw](https://discuss.ocaml.org/u/jhw)\ **Post date:** [March 17, 2019, 12:09am UTC](https://discuss.ocaml.org/t/dune-and-multicore/3315/17 "2019-03-17T00:09:57Z")
Note well: making `Unix.open_process_full` and `Unix.create_process_env` not use `Unix.fork` is one thing, but dealing with the lack of `execvpe(3)` on Darwin (macOS and iOS) leads down an interesting dark alley. I have not yet managed this, but I’m close. The logic I have passes all the tests in the multicore test suite, but Dune still manages to expose some error or another I haven’t identified. On the plus side, I’ve implemented in terms of `Domain` and `Atomic` most of the parts of `Thread`, `Mutex` and `Condition` that Dune needs. Still some weirdness with `Thread.wait_signal` that I don’t have wired down yet, and I’m working on that too. And I agree, `Thread` is a bad library, and everyone should want a good alternative. I’m hopeful we can get that soon, but in the mean time, there is a giant catastrophic pile of already running logic that uses it. So, having a working approximation of it might be nice. ---
**Author:** ![jhw](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/jhw/32/1639_2.png) [@jhw](https://discuss.ocaml.org/u/jhw)\ **Post date:** [March 17, 2019, 4:10pm UTC](https://discuss.ocaml.org/t/dune-and-multicore/3315/18 "2019-03-17T16:10:55Z")
> could `Thread` become a thin layer on top of multicore’s domains? I believe so. Moreover, I believe it could be decoupled from the compiler distribution and offered as a separate package. In my copious spare time, after I overcome this Dune compatibility skew problem, I hope to do that and make it available on the [multicore-opam](https://github.com/ocamllabs/multicore-opam) repository as a first step toward helping sort out this disorder. ---
**Author:** ![jeremiedimino](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/jeremiedimino/32/699_2.png) [@jeremiedimino](https://discuss.ocaml.org/u/jeremiedimino)\ **Post date:** [March 18, 2019, 10:42am UTC](https://discuss.ocaml.org/t/dune-and-multicore/3315/19 "2019-03-18T10:42:55Z")
BTW, in general it takes time for the community to completely switch to a new compiler and drop compatibility with older ones. That means that after multicore is released, it will take a long time before existing projects can start switching from `threads` to the better replacement provided by multicore. So another option would be to provide an API that can be implemented either on top of the current `threads` library or on top the new multicore features. Then existing projects could start migrating to the new APIs right now. ---
**Author:** ![jhw](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/jhw/32/1639_2.png) [@jhw](https://discuss.ocaml.org/u/jhw)\ **Post date:** [March 19, 2019, 7:18am UTC](https://discuss.ocaml.org/t/dune-and-multicore/3315/20 "2019-03-19T07:18:00Z")
> The logic I have passes all the tests in the multicore test suite, but Dune still manages to expose some error or another I haven’t identified. […and there is still] some weirdness with `Thread.wait_signal` that I don’t have wired down yet, and I’m working on that too. I had some time sitting in a waiting room this evening that afforded me the opportunity to work on this a bit more. I’ve got it all working now. I’ll steal some more time at some point soon and push my branches to Github. [Next page](https://discuss.ocaml.org/t/dune-and-multicore/3315.md?page=2)