Welcome to the July 2021 Multicore OCaml monthly report! This month’s update along with the previous updates has been compiled by me, @ctk21, @kayceesrk and @shakthimaan. As August is usually a period of downtime in Europe, the next update may be merged with the September one in a couple of months (but given our geographically diverse nature now, if enough progress happens in August I’ll do an update).
The overall status of the multicore efforts are right on track: our contributions to the next OCaml release have been incorporated in 4.13.0~alpha2, and our focus remains on crushing incompatibilities and bugs to generate domains-only parallelism patches suitable for upstream review and release. As a lower priority activity, we continue to develop the experimental “effects-based” IO stack, which will feature in the upcoming virtual OCaml Workshop at ICFP in August 2021.
4.12.0+domains trees continue to see a tail of bugs being steadily fixed. After last month’s call, we saw a number of external contributors step up to submit fixes in addition to the multicore and core OCaml teams. We would like to acknowledge and thank them!
@emillon (Etienne Millon) for running the Jane Street
corev0.14 test suite with 4.12.0+domains and sharing the test results (and finding a multicore GC edge case bug while at it).
@Termina1 (Vyacheslav Shebanov) for testing the compilation of
batteries3.30 with Multicore OCaml 4.12.0+domains.
@nbecker (Nils Becker) for reporting on
- Filip Koprivec for identifying a memory leak when using
All of these fixes, combined with some big-ticket compatibility changes (listed below) are getting me pretty close to using 4.12.0+domains as my daily OCaml opam switch of choice. I encourage you to also give it a try and report (good or bad) results on the multicore OCaml tracker. If these sorts of problems grab your attention, then Segfault Systems is hiring in India to work with @kayceesrk and the team there on multicore OCaml.
For benchmarking, the Jupyter notebooks for the Sandmark nightly benchmark runs have been updated, and we continue to test the Sandmark builds for the 4.12+ variants and 4.14.0+trunk. Progress has been made to integrate
current-bench OCurrent pipeline with the Sandmark 2.0 -alpha branch changes to reproduce the current Sandmark functionality, which will allow GitHub PRs to be benchmarked systematically before being merged.
As always, the Multicore OCaml ongoing and completed tasks are listed first, which are then followed by the updates from the Ecosystem libraries. The Sandmark nightly build efforts, benchmarking updates and relevant current-bench tasks are then mentioned. Finally, the update on the upstream OCaml Safepoints PR is provided for your reference.
Inclusion of most of OCaml headers results in requiring pthread
The inclusion of multiple nested header files requires
caml_young_endis not a
An inconsistency observed in the CI where
caml_young_endis now a
char *instead of
Parmap testsuite crash
Parmapis causing a segfault when its testsuite is run against
Crash running Multicore binary under AFL
bunpackage crashes with Multicore OCaml 4.12+domains, but,
builds fine on 4.12.
lablgtk’s example segfaults
An ongoing effort to compile lablgtk with OCaml and Multicore OCaml.
core v0.14: test triggers a segfault in the GC
A segfault caused by running
core.v0.14test suite with Multicore
OCaml 4.12.0+domains as reported by @emillon.
Cannot compile batteries on OCaml Multicore 4.12.0+domains
An effort by Vyacheslav Shebanov (@Termina1) to compile
batteries.3.30with Multicore OCaml 4.12.0+domains variant.
Backport trunk safepoints PR to multicore
The Safepoints implementation is being backported to Multicore
OCaml. The initial test results of running Sandmark on a large Xen2
box are shown below:
Some of the compatibility macros are not placed in the same headers as in upstream OCaml
The introduction of a compatibility layer for GC statistics need to
be consistent with trunk.
Review io.c for thread-safety and add parallel tests
The thread-safety fixes in io.c requires a review and additional
tests need to be added for the same.
A draft PR to support
caml_channel_mutex_*interfaces from trunk
to Multicore OCaml.
Add std gnu11 common cflags
The configure.ac file has been updated to use
common_cflagsfor both GCC and Clang.
Destroy channel mutexes after fork
A discussion on resetting and reinitializing mutexes after fork in
Expose functions to program with effects
A draft PR to enable programmers to write programs that use effects
without explicitly using the effect syntax.
Set resource Limit
A query to use
setrlimitin Multicore OCaml, similiar, to
Core.Unix.RLimit.setfrom Jane Street’s core library.
Domain better participants
0(n_running_domains)from domain creation and the iterations
0(Max_domains)from STW signalling have been removed.
Eventog event for condition wait
A new event has been added to indicate when a domain is blocked at
Condition.wait. This is useful for debugging any imbalance in task
distribution in domainslib.
stdlib/.dependto cover the recent developments in stdlib.
prevent leaks with leftover continuations.
Modernise signal handling
The Multicore OCaml signals implementation is now closer to that of
Expose a few more GC variables in headers
variables have now been defined in the runtime.
Make intern and extern work with Multicore
The upstream changes to intern and extern have now been incorporated
to work with the Multicore OCaml runtime.
A patch that fixes a locking bug with global roots observed on a Mac
OS CI with
encode_terminal_statusdoes not set all fields
A minor fix for the error caused when moved from using
In link_channel, channel->prev should be set to NULL
A PR to fix the memory leak when using
reported by Filip Koprivec.
Backtrace last exn is val unit
A fix for the crash reported on running core’s test suite by
Update to cstruct 6.0.1
ocaml-uring is now updated to use
Cstruct.shiftvwith the upgrade
A request by @nbecker to provide a
arrays having the following signature:
val parallel_map : Domainslib.Task.pool -> ('a -> 'b) -> 'a array -> 'b array
parallel_scan rejects arrays not larger than pool size
An “index out of bounds” exception is thrown for
Task.parallel_scanwith arrays not larger than the pool size as
reported by @nbecker.
lib/consts.mlfile in eventlog-tools now includes the
Fix initial value accounting in
The initial value of
parallel_for_reducehas been fixed so as to
not be accounted multiple times.
eio library provides an effects-based parallel IO stack for
WIP: Add eio_luv backend
A work-in-progress to use
luvthat provides OCaml/Reason bindings
to libuv for a cross-platform backend for eio.
Update to latest MDX to fix exception reporting
Dune has been updated to 2.9 along with necessary changes for
exception reporting with MDX.
A documentation update specifying the following steps required to
manually pin the effects version of
opam switch create 4.12.0+domains+effects --repositories=multicore=git+https://github.com/ocaml-multicore/multicore-opam.git,default opam pin add -yn ppxlib 0.22.0+effect-syntax opam pin add -yn ocaml-migrate-parsetree 2.1.0+effect-syntax
Improvements to traceln
tracelnto make it an Effect along with changes to
trace output and addition of tests.
Add Flow.read_methods for optimised reading
The addition of
Flowmodule as a faster
alternative to reading into a buffer.
Allow cancelling waiting for a semaphore
lib_eio/semaphore.mlto allow cancel waiting for a
Add more generic exceptions
The inclusion of generic exceptions to avoid depending on
backend-specific exceptions. The tests have also been updated.
Parallel notebook pausetimes graphing for navajo results throws an error
The parallel Jupyter notebook for pausetimes throws a ValueError
that needs to be investigated.
Status of disabled benchmarks
js_of_ocamlbenchmark results that
were disabled from the Jupyter notebooks have to be tested with
recent versions of Multicore OCaml.
Parallel scalability number on navajo look odd
The parallel performance numbers on the navajo build server for
scalability will need to be reviewed and the experiments repeated
col_wrapas 3 instead of 5 in the normalised results in parallel notebook
For better readability, it is recommended to use col_wrap as 3 in
the normalised results in the parallel notebook.
View results for a set of benchmarks in the nightly notebooks
A feature request to filter benchmarks by name or by tags when used
with Jupyter notebooks.
Static HTML pages for the recent results
The benchmark results from the most recent build runs should be used
to generate static HTML reports for review and analysis.
Timestamps are not sorted in the parallel_nightly notebook
The listing of timestamps in the drop-down option is now sorted.
Add irmin tree benchmark
A request to add the Irmin tree.ml benchmark to Sandmark, including
necessary dependencies and data files.
An update to dune.2.9.0 in order to build coq with Multicore OCaml
Sandmark breaks on OCaml 4.14.0+trunk
The Sandmark build for OCaml 4.14.0+trunk needs to be resolved as we
begin upstreaming more Multicore OCaml changes.
coq fails to build
coqpackage is failing to build with 4.12.0+domains+effects
with Sandmark on navajo server.
Update pausetimes_multicore to fit with the latest Multicore changes
The Multicore pausetimes have now been updated for the 4.12.0
upstream and 4.12.0 branches which now use the new Common Trace
Update selected benchmarks as a set for baseline benchmark
You now have the option to only filter from the user selected
variants in the Jupyter notebooks.
Run sandmark_nightly on a larger machine
The Sandmark nightly builds now run on a 64+ core machine to benefit
from the improvements to Domainslib.
Add navajo specific parallel config.json file
A navajo server-specific run_config.json file has been added to
Sandmark to run Multicore parallel benchmarks.
Add commentary on grammatrix
A documentation update for the grammatrix benchmark on customised
task distribution via channels and the use of
Add chrt to pausetimes_multicore wrapper
The use of
chrt -r 1in paramwrapper is required with
pausetimes_multicoreto use the taskset arguments.
Add trunk build to CI
The .drone.yml file has now been updated to include 4.14.0+stock
trunk build for the CI.
Read stderr from the docker container
We are able to run Sandmark-2.0 -alpha branch with current-bench
now, and it is useful to view the error output when running with
Replicate ocaml-bench-server setup
A request to dynamically pass the Sandmark benchmark target commands
to current-bench in order to create pipelines.
The PR has been cherry-picked on 4.13 and finally merged with
We would like to thank all the OCaml users, developers and contributors in the community for their valuable time and support to the project. Stay safe and have a great summer if you are northern hemispherically based!
- AFL: American Fuzzy Lop
- CI: Continuous Integration
- CTF: Common Trace Format
- GC: Garbage Collector
- GCC: GNU Compiler Collection
- GTK: GIMP ToolKit
- HTML: HyperText Markup Language
- IO: Input/Output
- OPAM: OCaml Package Manager
- OS: Operating System
- PR: Pull Request
- STW: Stop The World