# Parallel performance of an OCaml program: ocaml-5.0.0 Vs 4.13.1 **URL:** \ **Category:** Community\ **Tags:** multicore-ocaml, parallel-programming\ **Created:** [January 24, 2023, 6:23am UTC](https://discuss.ocaml.org/t/parallel-performance-of-an-ocaml-program-ocaml-5-0-0-vs-4-13-1/11233 "2023-01-24T06:23:00Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![UnixJunkie](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/unixjunkie/32/638_2.png) [@UnixJunkie](https://discuss.ocaml.org/u/UnixJunkie)\ **Post date:** [January 24, 2023, 6:23am UTC](https://discuss.ocaml.org/t/parallel-performance-of-an-ocaml-program-ocaml-5-0-0-vs-4-13-1/11233/1 "2023-01-24T06:23:00Z")
I ran some test 5 times. I show the average +/- one standard deviation. ![xp_par_encode](https://us1.discourse-cdn.com/flex020/uploads/ocaml/original/2X/8/88d208627791694f9104312d97a78835cdfb9e26.png) I was running this program: [GitHub - tsudalab/ACP4: AutoCorrelation of Pharmacophore Features](https://github.com/tsudalab/ACP4) The peak performance on this computer is reached by ocaml-4.13.1. The program is parallelized using parany, so in the 4.13.1 case, the parallelism is fork-based and values are serialized by Marshal. In 5.0.0, it uses Domains and Domainslib.Chan (Marshal not needed). I don’t claim this is the best way to write this program in parallel. I don’t claim this is reflective of other users use-cases. What the program does, in essence: - read some file - apply an autocorrelation function to each element read from that file - write the resulting vector for each element into an output file I had to do some work so that the program parallelizes better. In essence, I make the parallel work function do more things (coarser grain), the input function should do the least work; same for the output function. I might test later this week on another computer, with more computing threads. For my personal records, so that I can rerun this later: ```bash for j in `seq 1 5`; do for i in `(echo 1; seq 2 2 24) | sort -n -r`; do echo "NCORES: "$i; (acp4 -i data/IDH1_1conf.ph4 -o idh1_par.txt -np $i -csize 50 2>&1) | grep Hz; done; done ``` ---
**Author:** ![Chet\_Murthy](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/chet_murthy/32/1501_2.png) [@Chet\_Murthy](https://discuss.ocaml.org/u/Chet_Murthy)\ **Post date:** [January 24, 2023, 6:37am UTC](https://discuss.ocaml.org/t/parallel-performance-of-an-ocaml-program-ocaml-5-0-0-vs-4-13-1/11233/2 "2023-01-24T06:37:48Z")
Is this workload intrinsically partitionable ? If so, then it shouldn’t be surprising that 4.13.1 beats 5.0, since _partitioning_ will always incur less overhead than multithreading for truly partitionable workloads, right? ---
**Author:** ![UnixJunkie](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/unixjunkie/32/638_2.png) [@UnixJunkie](https://discuss.ocaml.org/u/UnixJunkie)\ **Post date:** [January 24, 2023, 6:44am UTC](https://discuss.ocaml.org/t/parallel-performance-of-an-ocaml-program-ocaml-5-0-0-vs-4-13-1/11233/3 "2023-01-24T06:44:02Z")
The input file can be cut into independent chunks, if this is what you are asking for. ---
**Author:** ![Chet\_Murthy](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/chet_murthy/32/1501_2.png) [@Chet\_Murthy](https://discuss.ocaml.org/u/Chet_Murthy)\ **Post date:** [January 24, 2023, 7:26am UTC](https://discuss.ocaml.org/t/parallel-performance-of-an-ocaml-program-ocaml-5-0-0-vs-4-13-1/11233/4 "2023-01-24T07:26:41Z")
In that case, it seems likely you cannot improve over just plain partitioning. ---
**Author:** ![octachron](https://avatars.discourse-cdn.com/v4/letter/o/49beb7/32.png) [@octachron](https://discuss.ocaml.org/u/octachron)\ **Post date:** [January 24, 2023, 8:04am UTC](https://discuss.ocaml.org/t/parallel-performance-of-an-ocaml-program-ocaml-5-0-0-vs-4-13-1/11233/5 "2023-01-24T08:04:34Z")
> [@UnixJunkie](#): > > I ran some test 5 times. > I show the average +/- one standard deviation. You cannot use 5 sample points and expect to measure a meaningful standard deviation even if the distribution of your sampled data is _exactly_ a gaussian. More generally, standard deviation is not that meaningful without the hypothesis that the underlying distribution is a gaussian, and in presence of scheduling I expect this hypothesis to be _false_. But yes, overall if you don’t need synchronisation between processes, process parallelism should be faster than shared memory parallelism since it means that you don’t have any synchronisation overhead. ---
**Author:** ![Chet\_Murthy](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/chet_murthy/32/1501_2.png) [@Chet\_Murthy](https://discuss.ocaml.org/u/Chet_Murthy)\ **Post date:** [January 24, 2023, 8:19am UTC](https://discuss.ocaml.org/t/parallel-performance-of-an-ocaml-program-ocaml-5-0-0-vs-4-13-1/11233/6 "2023-01-24T08:19:05Z")
Also, when reporting parallel results, it’s worth reporting the CPU type, # cores, hyperthreads-per-core, etc. Basically, the contents of /proc/cpuinfo , or output of `lscpu` . Also, to follow along the path @octachron started down, you might want to construct a series of test inputs, each one doubling in size, and then run each scenario a large # of times on the smaller inputs (which will not take very long) and then as the inputs get larger, you will know whether you can reduce the # of runs, by watching the spread (variance? Not a prob/stats guy) in results. ---
**Author:** ![UnixJunkie](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/unixjunkie/32/638_2.png) [@UnixJunkie](https://discuss.ocaml.org/u/UnixJunkie)\ **Post date:** [January 24, 2023, 9:18am UTC](https://discuss.ocaml.org/t/parallel-performance-of-an-ocaml-program-ocaml-5-0-0-vs-4-13-1/11233/7 "2023-01-24T09:18:18Z")
I was not expecting the variance to be so huge and I wanted to minimize the number of experiments, so that the bench runs reasonably fast; hence the “only” 5 experiments per data point. But, we are not here to discuss the fine points of statistics, what I see is what I call a “performance regression bug”. And, this is not good news. If other people have parallel programs that they can run end-to-end, and test against ocaml-5 and before, this might be useful data. ---
**Author:** ![octachron](https://avatars.discourse-cdn.com/v4/letter/o/49beb7/32.png) [@octachron](https://discuss.ocaml.org/u/octachron)\ **Post date:** [January 24, 2023, 9:26am UTC](https://discuss.ocaml.org/t/parallel-performance-of-an-ocaml-program-ocaml-5-0-0-vs-4-13-1/11233/8 "2023-01-24T09:26:16Z")
> [@UnixJunkie](#): > > what I see is what I call a “performance regression bug”. And, this is not good news. You are not reporting a performance regression since you are comparing different parallelization methods? Process parallelism is expected to be faster in absence of communications and synchronization between processes and it is still available in OCaml 5. ---
**Author:** ![UnixJunkie](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/unixjunkie/32/638_2.png) [@UnixJunkie](https://discuss.ocaml.org/u/UnixJunkie)\ **Post date:** [January 24, 2023, 9:27am UTC](https://discuss.ocaml.org/t/parallel-performance-of-an-ocaml-program-ocaml-5-0-0-vs-4-13-1/11233/9 "2023-01-24T09:27:47Z")
There are 1-to-N and N-to-1 communications in this program. Hence, the domains (or process) are synchronized by exchanging “messages”, if you want. ---
**Author:** ![bluddy](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/bluddy/32/104_2.png) [@bluddy](https://discuss.ocaml.org/u/bluddy)\ **Post date:** [January 24, 2023, 9:44am UTC](https://discuss.ocaml.org/t/parallel-performance-of-an-ocaml-program-ocaml-5-0-0-vs-4-13-1/11233/10 "2023-01-24T09:44:50Z")
How long do these programs run? ---
**Author:** ![Chet\_Murthy](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/chet_murthy/32/1501_2.png) [@Chet\_Murthy](https://discuss.ocaml.org/u/Chet_Murthy)\ **Post date:** [January 24, 2023, 10:09am UTC](https://discuss.ocaml.org/t/parallel-performance-of-an-ocaml-program-ocaml-5-0-0-vs-4-13-1/11233/11 "2023-01-24T10:09:43Z")
TL;DR if your workload is “embarrassingly partitionable”, it is highly unlikely you will be able to do better using multithreading, than you can do using process parallelism. This is known, and is the case for basically every language runtime in the universe: OCaml isn’t special here. Can you describe the pattern of communication ? From your description, it seems like it might be something like “scatter-gather” – that is, a main thread scatters work units to workers, who process and return the results to the main thread, who assembles them? If 1. the work required to scatter, and then to reassemble, is small compared to the work to process each work-unit 2. if there is no other mutable shared state 3. there isn’t some massive data-set that needs to be shared among all workers (even this can be accommodated, if that data-set can be preloaded before workers are forked) then it can often be more-efficient (and more-performant) to use process-parallelism. These sorts of workloads have a name: “embarrassingly partitionable”, and it is because it doesn’t take much to convert them to run well using multiple processes. And if you can do that, it will always, always be faster than using multithreading, and for many, many reasons. Commercial transaction-processing software vendors have explored many of these issues, as have HPC code implementors. Multithreading isn’t actually the best way to get peak performance: it’s just often a good way to get decent performance out of an SMP without working insanely hard. Even here, I would note that there’s a difference between SMP and “multicore”. The latter term (in the commercial world) is synonymous with “NUMA” (non-uniform memory access) and naive multithreading (any form of multithreading that doesn’t take into account which memory banks are used to store various data, and their relation to the processor sockets where threads run to access that data) is known to perform suboptimally on NUMA machines. And these days, there are a whole lot more NUMA machines that you might think, b/c Moore’s Law for frequency scaling has stopped. ---
**Author:** ![hyphenrf](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/hyphenrf/32/2733_2.png) [@hyphenrf](https://discuss.ocaml.org/u/hyphenrf)\ **Post date:** [January 24, 2023, 11:07am UTC](https://discuss.ocaml.org/t/parallel-performance-of-an-ocaml-program-ocaml-5-0-0-vs-4-13-1/11233/12 "2023-01-24T11:07:50Z")
I was always told “fork is faster than smp”, almost as conventional wisdom. It’s nice to see it visualized! ---
**Author:** ![hyphenrf](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/hyphenrf/32/2733_2.png) [@hyphenrf](https://discuss.ocaml.org/u/hyphenrf)\ **Post date:** [January 24, 2023, 11:22am UTC](https://discuss.ocaml.org/t/parallel-performance-of-an-ocaml-program-ocaml-5-0-0-vs-4-13-1/11233/13 "2023-01-24T11:22:03Z")
BTW I just remembered something I wrote two years ago: it’s for the “programming language drag race” and I wrote it because OCaml’s score felt way too low. Never got to actually submitting it though. It further confirms this observation (but not visually): > (feel free to tear apart 😁) ---
**Author:** ![gadmm](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/gadmm/32/877_2.png) [@gadmm](https://discuss.ocaml.org/u/gadmm)\ **Post date:** [January 24, 2023, 4:49pm UTC](https://discuss.ocaml.org/t/parallel-performance-of-an-ocaml-program-ocaml-5-0-0-vs-4-13-1/11233/14 "2023-01-24T16:49:49Z")
I do not read too much into the chart. There could be improvements to make to the program, or to the multicore runtime implementation, or both. However, I see proposed around a lot the explanation of inherent synchronization costs of SMP runtimes. This should not be used to discard legitimate questions and explorations about consequences of the design choices of the multicore runtime (for instance, a stop-the-world collector that empties _all_ minor heaps whenever _one_ minor heap is full). ---
**Author:** ![Niklaus](https://avatars.discourse-cdn.com/v4/letter/n/5f8ce5/32.png) [@Niklaus](https://discuss.ocaml.org/u/Niklaus)\ **Post date:** [January 24, 2023, 5:21pm UTC](https://discuss.ocaml.org/t/parallel-performance-of-an-ocaml-program-ocaml-5-0-0-vs-4-13-1/11233/15 "2023-01-24T17:21:30Z")
I think it is a valid point. I very much do not embrace the argument saying « choose the right architecture for the job » here. If one has to resort to forks for a simple performant map/reduce it means Ocaml multicore abstraction leaks (in time). It would be interesting to understand the _Intrinsic Parallel Efficiency_ of an Ocaml program with respect to at least the GC (minor and major) and the major allocator. As far as I understand the minor GCs work in 1) lock-step and are as a consequence is 2) tail sensitive (to the most allocating among N). Also the major allocator is a global parallel allocator which has to cope with generational copy. Intrinsic Parallel Efficiency: **Maybe** one way to characterize would be: - an embarrassing parallel program (map only, N partitions) , no sync, with an as much as possible identical sequential version (easy for map) - some functions with computation designed to generate, _per run_, an increasingly number of minor GC/sec (no major) - a variant with spill out to major, increasingly - a variant with unbalanced allocation (1 tail among N) - using single threaded (4.x) vs multi runtime (5.x) PE (Parallel Efficiency): sequential-time-workload-sizeN / (parallel-time-workload-size1 \* N) SR (Sequential Regression): sequential-time-workload-5.x / sequential-time-workload-4.x (assuming no other unrelated optimization) **IPE** of the _Ocaml Multicore runtime_: PE \* SR Also mentioned in another discussion about domains: the tail latency (wrt lock steps) induced by thread scheduling, which may suggest a sensitivity to _noisy neighbors_ / too many domains. What is not part of IPE is synchronisation nor cache costs nor NUMA, nor // algorithm efficiency, as they are concerns regardless of the language. And to end on a definitely positive note: effects and local DRF will allow for a great parallel programing model (green threads, sync primitives, performant and bug-free lock-free algorithms, actor/CSP model?…) It is still early days, Ocaml team has done an incredible job with multicore, we will all benefit from gaining an understanding in the characteristics and inevitable trade-offs made during this journey. ---
**Author:** ![UnixJunkie](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/unixjunkie/32/638_2.png) [@UnixJunkie](https://discuss.ocaml.org/u/UnixJunkie)\ **Post date:** [January 25, 2023, 1:18am UTC](https://discuss.ocaml.org/t/parallel-performance-of-an-ocaml-program-ocaml-5-0-0-vs-4-13-1/11233/16 "2023-01-25T01:18:42Z")
If you have proper statistics of a parallel implementation comparing ocaml-5 to ocaml\<5, it would be interesting to share them. ---
**Author:** ![UnixJunkie](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/unixjunkie/32/638_2.png) [@UnixJunkie](https://discuss.ocaml.org/u/UnixJunkie)\ **Post date:** [January 25, 2023, 9:59am UTC](https://discuss.ocaml.org/t/parallel-performance-of-an-ocaml-program-ocaml-5-0-0-vs-4-13-1/11233/18 "2023-01-25T09:59:33Z")
I need to share my test data file, that’s the only obvious thing missing. This is a regular opam package, so installing it should be a matter of opam pin to the git repository. I am not particularly interested in people trying to optimize the program. Rather, I would be more interested by stories of other parallel programs which have an ocaml\<5 and an ocaml-5 version. ---
**Author:** ![gasche](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/gasche/32/4_2.png) [@gasche](https://discuss.ocaml.org/u/gasche)\ **Post date:** [January 25, 2023, 10:24am UTC](https://discuss.ocaml.org/t/parallel-performance-of-an-ocaml-program-ocaml-5-0-0-vs-4-13-1/11233/19 "2023-01-25T10:24:01Z")
> I am not particularly interested in people trying to optimize the program. Hot take: the problem with your benchmark is that it is not clear that you used Domains correctly, or whether you made programming mistakes that limit the scalability of the Domains version. For example, you mention that the benchmark displays on the x axis the number of compute domains, but that there are also 2 mostly-idle domains used for control. This is a performance antipattern for Domains programming, and it is hard to know whether this invalidates the performance results or not. I have already made this point last week: [No Domain.maximum\_domain\_count() in the stdlib - #28 by gasche](https://discuss.ocaml.org/t/no-domain-maximum-domain-count-in-the-stdlib/11173/28). This benchmark is a comparison of a finely-tuned multi-process version developed over time by a domain expert (you) and a hastily-, naively-written multi-domain version developed by a newbie (you). Good luck interpreting that. What I gather from this interaction and your experiment is that: - It may very well be the case that, for many sorts of embarrassingly-parallel problems, multiprocess concurrency works just as well or better as shared-memory concurrency. Others have said this in the thread before. In fact, the _whole OCaml community_ has been saying this for the _last 20 years_: before the Multicore OCaml effort, we had clear explanations on the fact that the absence of a shared-memory concurrent runtime was not an issue for good parallel performance for some common categories of programs. These explanations _remain valid_ even now that people have poured the considerable work of moving to a concurrent runtime. It is still interesting to get yet another confirmation of this (modulo the doubts about your Domains-using code which may be too naive), and maybe it can help recalibrate people’s expectations about the performance benefits of Multicore. This also explains why upstream-OCaml insisted so much on compatibility of sequential performance, imposing many difficult/painful performance constraints onto the Multicore OCaml developers. If multiple sequential processes are faster for your use-case, then just keep using that with OCaml 5. - The programming model for Domains is not as easy as it looks from a distance. People like @UnixJunkie with previous experience in parallel programming (much more than I do) seem to get it wrong. There are two things to distinguish here: ---
**Author:** ![gasche](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/gasche/32/4_2.png) [@gasche](https://discuss.ocaml.org/u/gasche)\ **Post date:** [January 25, 2023, 10:34am UTC](https://discuss.ocaml.org/t/parallel-performance-of-an-ocaml-program-ocaml-5-0-0-vs-4-13-1/11233/20 "2023-01-25T10:34:49Z")
Note: we know that multi-processing can be as fast or faster as multi-domain programming, but ideally multi-domain programming should not be too much slower. Provided good benchmarks that show a larger-than-expected performance difference, we could study the multi-domain performance and try to improve the Multicore runtime – surely there are many areas for improvement left. It is not clear to me that it’s a valuable use of our time to discuss _bad_ benchmarks such as yours. On one hand, maybe we should try to focus first on code that follows the recommended Domain programming advice. On the other hand, people in the wild write wild code, and it’s less hassle for everyone if that also runs fast enough. My personal intuition is that we should still focus on “good” benchmarks first, and figure out library designs that make it easy to write “good” programs. ---
**Author:** ![hyphenrf](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/hyphenrf/32/2733_2.png) [@hyphenrf](https://discuss.ocaml.org/u/hyphenrf)\ **Post date:** [January 25, 2023, 3:27pm UTC](https://discuss.ocaml.org/t/parallel-performance-of-an-ocaml-program-ocaml-5-0-0-vs-4-13-1/11233/21 "2023-01-25T15:27:28Z")
I haven’t gathered meaningful statistics unfortunately. I remember this was somewhere between 4.10/4.10+domains and 4.12 for each, respectively. So, by that time, multicore was still WIP but it was nearing the end of its development. I’m curious if 4.14’s prefetching would improve this workload, or if the released multicore fares better, or if using e.g. domainslib can be more optimal. The files are self-contained low-dependency and near-identical, and the measurement code is in the Makefile, so hopefully it’s easier to reproduce and improve for anyone interested. I wanted to functorize but that seems to have made the compiler lay out the code slightly worse and created a bit of a slowdown. [Next page](https://discuss.ocaml.org/t/parallel-performance-of-an-ocaml-program-ocaml-5-0-0-vs-4-13-1/11233.md?page=2)