For grins, I updated my multicore OPAM switch and started looking into what is currently keeping my Orsetto project from compiling there. At the moment, the answer is that Dune doesn’t compile. Which I suppose is not really noteworthy, because multicore is still way way off, but I was vastly amused by the log message that appeared on my console:
I must not segfault. Uncertainty is the mind-killer. Exceptions are the little-death that brings total obliteration. I will fully express my cases. Execution will pass over me and through me. And when it has gone past, I will unwind the stack along its path. Where the cases are handled there will be nothing. Only I will remain.
I don’t know who wrote this, but it has me rolling on the floor— especially given the irony of this happening on a compiler that’s trying to introduce algebraic effects. Just wanted to raise a glass for that here. Bravo, whoever you are. You have won the Internet today.
I believe the credit goes to someone at Jane Street (@jeremiedimino should know who). Recent versions of dune do not work on the multicore fork because it doesn’t have threads AFAIK. Dune 1.2.1 is the last version to work on the multicore fork.
BTW, you the version of OCaml used to build dune doesn’t need to match the version of OCaml you want to build projects with. In short, you can build a dune with a vanila 4.07 compiler and use it to build OCaml code using a multicore OCaml. Not sure this will work with opam though, you might have to setup your own workspace by cloning all the various dependencies and letting dune build everything at once.
Yeah, one could do that. I feel like the best way I can help the multicore effort is to help make patches to OPAM packages as necessary to adapt to the forthcoming language changes and stage them in the multicore repository. I did that already for some of the PPX tools while Dune was still working. Next I’ll need to think about whether to make Dune work by pinning it or patching it. Not sure how much of OPAM is already dependent on versions of Dune that are broken under multicore.
It seems to me that the best way to go about it would be to get the threads library to work with multicore, then everything using threads, in particular dune will work again.
Odd though it may sound, getting threads working is low down the multicore roadmap because it will always be a bad library to use!
I spent a few delightful hours on Sunday hacking both Dune and multicore - Dune would need a small selected backend to switch between Domain.create and Thread.create and multicore needs Unix.open_process* and Unix.create_process not to use Unix.fork. I made preliminary versions of both these changes, but I’ve hit a segfault when the process watcher domain attempts to report back to the main domain that a process has terminated. @stedolan and I are going to have a poke at that later in the week, hopefully…
Getting Dune well integrated with multicore seems good. I do feel like the same problem will arise with many other projects after Dune though
multicore needs Unix.open_process* and Unix.create_process not to use Unix.fork
Is it ok if the forking is done in the C stubs though? I’ve been thinking of vendoring spawn into dune for a while, it might be worth looking into that.
What do you mean, a “bad library to use”? What about the programs which already use it (and don’t care about launching 50,000 threads)? I have several programs that use Thread because it’s currently the only way to have a modicum of concurrency without sprinkling a monad in 90% of your code, and I’d like these programs to continue working in the future…
edit: sorry, that was unecessarily harsh. I overreacted.
Those programs will work - just not as well as if they were written to use multicore’s domains, but the intention (obviously) is that the performance would be no worse than if you weren’t using multicore OCaml.
follow up question: could Thread become a thin layer on top of multicore’s domains (which are heavy threads, if I understand correctly) while keeping the legacy interface?
You can only use the global lock as a substitute for proper mutexes if your critical section doesn’t allocate (or call functions?) at all, I think, since that’s where preemption can occur. In practice I use Mutex.t or Condition.t (or sane wrappers around them) whenever Thread is involved.
For the record, pinning Dune down at 1.2.1 is a sadness for me, because too many of the things I care about have dependencies on things that need Dune 1.6 or later. Looks like if I want to mess around with multicore then I need to sort out this Dune and Multicore problem myself. Grmf.
Note well: making Unix.open_process_full and Unix.create_process_env not use Unix.fork is one thing, but dealing with the lack of execvpe(3) on Darwin (macOS and iOS) leads down an interesting dark alley. I have not yet managed this, but I’m close. The logic I have passes all the tests in the multicore test suite, but Dune still manages to expose some error or another I haven’t identified.
On the plus side, I’ve implemented in terms of Domain and Atomic most of the parts of Thread, Mutex and Condition that Dune needs. Still some weirdness with Thread.wait_signal that I don’t have wired down yet, and I’m working on that too. And I agree, Thread is a bad library, and everyone should want a good alternative. I’m hopeful we can get that soon, but in the mean time, there is a giant catastrophic pile of already running logic that uses it. So, having a working approximation of it might be nice.
could Thread become a thin layer on top of multicore’s domains?
I believe so. Moreover, I believe it could be decoupled from the compiler distribution and offered as a separate package. In my copious spare time, after I overcome this Dune compatibility skew problem, I hope to do that and make it available on the multicore-opam repository as a first step toward helping sort out this disorder.
BTW, in general it takes time for the community to completely switch to a new compiler and drop compatibility with older ones. That means that after multicore is released, it will take a long time before existing projects can start switching from threads to the better replacement provided by multicore. So another option would be to provide an API that can be implemented either on top of the current threads library or on top the new multicore features. Then existing projects could start migrating to the new APIs right now.
The logic I have passes all the tests in the multicore test suite, but Dune still manages to expose some error or another I haven’t identified. […and there is still] some weirdness with Thread.wait_signal that I don’t have wired down yet, and I’m working on that too.
I had some time sitting in a waiting room this evening that afforded me the opportunity to work on this a bit more. I’ve got it all working now.
I’ll steal some more time at some point soon and push my branches to Github.
