Multicore compatibility

multicore
#1

So I finally reached a good stopping point with my other side projects and spent some time yesterday investigating what it would take to bring them up on the multicore variant of the OCaml toolchain.

Answer: it looks tantalizingly possible. It wasn’t hard to fix OMake, for example, not to use effect as a variable name. Other things look more troubling. The tagged 4.06.1 release appears not to include an important C-language header file in its installation, and I think that might be fixed on trunk already. Other things look moderately difficult for a newcomer to approach, but I feel like I might be able to wrestle them to the ground if nobody better at these jobs have time for them.

For example, I noticed that the ocaml-migrate-parsetree project needs to be improved with comprehension of the new effect syntax. I spent some time yesterday investigating that, and I haven’t cracked it yet. Nevertheless, it still seems like something I can do provided with enough time on the calendar— my day job is intense and I have to steal time at home after everyone in the house is asleep to work on my side projects.

Question: what is everyone else working on to make multicore compatibility with the popular packages in the default OPAM repository happen? I’d like to contribute where I can, and I don’t want to duplicate work that somebody else might be able to get done faster.

#2

Update: I have the ocaml-migrate-parsetree project updated and passing its tests. Now to tackle the ppx_tools project, which looks like it has similar issues related to the introduction of the new effect keyword.

#3

Thanks, this is really helpful! Bear in mind that the multicore repository is still a work-in-progress and not an official part of the language. Having said that, it’s extremely useful to be able to differentially compile the stable opam universe against the multicore one, and avoiding the use of the effect keyword is one good fix that doesn’t adversely affect anything.

Things might get more complex when it comes to adding experimental features to ocaml-migrate-parsetree – it might be better to do a lightweight fork there and maintain a patch in an opam-multicore remote.

#4

Yeah, I know it’s work in progress. Did I mention that I am REALLY champing at the bit to have algebraic effects in the language?

So I opened an issue on the opam-multicore remote to cover this problem of how to stage the required changes. Due to the design of the ocaml-migrate-parsetree project, it would be inappropriate to pull the necessary changes into its master until the multicore variant has a release vehicle planned in the OCaml project.

My preferred approach would be to keep the necessary changes on a development branch and put a package for it into the opam-multicore remote.

#5

I’ve now opened pull requests for the opam-multicore remote repository that adds multicore variants for the ppx_tools and omake projects (and their dependencies). The omake patch was a little interesting because it make fairly extensive use of the C foreign-function interface, and the new C primitive API changes are not so well documented yet.

#6

Considering how far we are from mainlining of the multicore branch I’m not sure if there is any sense in porting something to multicore (beyond benchmarking purposes). I’m pretty sure your experience could be very helpful in finishing up the multicore branch itself.

#7

Once it becomes certain that effect will indeed become a new keyword in a not too distant future version of the compiler, I reckon it would help the transition if upcoming releases before multicore would issue a warning if effect were used as an identifier.

Or will there be no more major releases after 4.07 and before multicore? :slight_smile:

#8

Depends on what is considered “distant”. KC’s later tweet states “I am hopeful that we can get all of items sorted by the end of the year.” some it’s quite far even from an alpha release.

#9

I don’t know when all of the multicore efforts might land in the main OCaml distribution, but my experience with the remote that’s been baselined on 4.06.1 has been pretty positive so far.

One thing I noticed today is that PR#1003 to bring the C interface changes into mainline seems a bit stalled. Apparently, the documentation needs updating (and I might be able to help out with that) and there were also apparently some efforts underway that might make some of the breaking changes unnecessary (which I’d be curious to see an update here about that).

Certainly it would be nice to have the OCAML_API_VERSION macro available so that we can begin preparing external projects for any breaking changes in the C interface going forward. I’d be thrilled to have just that in the OCaml 4.07 release.