A parallel and shared memory library based on Hack's implementation

Hack_parallel is a standalone parallel and shared memory library based on the components used in FB tools Hack, Flow, and Pyre. The internals of these components are little known outside of these projects, yet stand to benefit the OCaml community more generally, both for practical use and educational purposes. So I extracted the core parts and packaged it up; more information can be found in the project README.

I put together an example project to illustrate the basic interface.

15 Likes

Exciting stuff!

But how seriously should I take these warnings?

2 Likes

From some more details:

You can ignore the scary THIS CODE ONLY WORKS WITH HACK. The implementation works generally (it’s used in Flow/Pyre and my own projects without problems). The warning is overcompensating so that people do not treat it like a “generic atomic hashtable for OCaml”. The data structure and constraints on memory operations are explained well here. I may document the points in the video in future, but hope that helps for now.

3 Likes

Came here after googling around. This lib does not work on FreeBSD, for your information. :wink: When starting Flow (must be run as root because of access to /dev/null, for whatever reason), it fails with “Out of shared memroy”.

1 Like

If you can show me that your library goes significantly faster than this one:

Then, and only then, I will have a serious look at it. :wink:

2 Likes

Open an issue in their bug tracker man…

1 Like

What is the difference between parany and parmap by dicosmo?
If you rely on fork to process things in parallel, then here you have your difference: the hacks’s parallel library shines if you need shared memory. It’s like a poor’s man multicore while waiting for the official multicore in OCaml 5.0.

Parany can process in parallel an infinite stream.
Parmap cannot.