Multicore OCaml: October 2021

Community
,
1

Welcome to the October 2021 Multicore OCaml monthly report! The previous updates along with
this update have been compiled by me, @ctk21, @kayceesrk and @shakthimaan.

As @octachron announced last month, the core team has committed to an OCaml 5.0 release next year with multicore and the effects runtime. This month has seen tremendous activity in our multicore trees to prepare an upstream-friendly version, with a number of changes made to make the code ready for ocaml/ocaml and reduce the size of the diff. Recall that we have been feeding in multicore-related changes steadily since way back in OCaml 4.09, and so we are now down to the really big pieces. Therefore the mainline OCaml trunk code is now being continuously being merged into our 5.00 staging branch, and test coverage has increased accordingly.

In the standard library, we continue to work and improve on thread safety by default. Since effect handlers are confirmed to go into 5.0 as well, they now have their own module in the stdlib as well. The multicore library ecosystem is also evolving with the changes to support OCaml 5.00, and in particular, Domainslib has had significant updates and improvements as more usecases build up. The integration of the Sandmark performance harness with current-bench is also actively being worked upon.

We would like to acknowledge the following people for their contribution:

  • Török Edwin was able to reproduce the bug in Task.pool management
    Domainslib#43, and has also provided a PR to fix the same.
  • Sid Kshatriya has created
    PR#83 for Eio to use the Effect Handlers module.

Our focus in November is going to continue to be on relentlessly making a 5.0 staging tree, and we are preparing for a series of working groups with the core OCaml teams (taking up an entire week) to conduct preliminary code review on the full patchset. Stay tuned for how that has gone by the start of December!

As always, the Multicore OCaml updates are listed first, which contain the upstream efforts, merges with trunk, updates to test cases, bug fixes, and documentation improvements. This is followed by the ecosystem updates on Domainslib, Tezos, and Eio. The Sandmark and current-bench tasks are finally listed for your reference.

Multicore OCaml

Ongoing

Upstream

Testsuite

  • ocaml-multicore/ocaml-multicore#656
    Core testsuite workflow

    A draft PR to implement a workflow to run the Core’s testsuite once
    a day.

  • ocaml-multicore/ocaml-multicore#720
    Improve ephemerons compatibility with testsuite

    The PR imports upstream fixes to make ephemerons work with infix
    objects, and provides a fix for weaktest.ml.

  • ocaml-multicore/ocaml-multicore#722
    Testsuite: Re-enable signals_alloc testcase

    The signals_alloc testcase has been enabled, and the PR also
    attempts to ensure the bytecode interpreter polls for signals.

  • ocaml-multicore/ocaml-multicore#723
    beat.ml failure on GitHub Action MacOS runners

    An investigation on the beat.ml test failure in the testsuite for
    the CI execution runs.

Sundries

  • ocaml-multicore/ocaml-multicore#669
    Set thread names for domains

    A patch that implements thread naming for Multicore OCaml. It
    provides an interface to name Domains and Threads differently.

  • ocaml-multicore/ocaml-multicore#698
    Return free pools to the OS

    The pool_release is in the shared_heap and does not return memory
    to the OS. An ongoing discussion on how much memory to hold, and to
    reclaim with space overhead setting.

  • ocaml-multicore/ocaml-multicore#703
    Possible loop in caml_enter_blocking_section when no domain can handle a blocked signal

    A scenario that can be triggered when a domain that blocks a
    specific set of signals exists where no other domain can process the
    signal, and can be caused by a loop in
    caml_enter_blocking_section.

  • ocaml-multicore/ocaml-multicore#725
    Blocked signal infinite loop fix

    A monotonic recorded_signals_counter has been introduced to fix
    the possible loop in caml_enter_blocking_section when no domain
    can handle a blocked signal.

  • ocaml-multicore/ocaml-multicore#726
    Marshalling of concurrently-modified objects is unsafe

    The marshalling of objects being mutated on a different domain must
    be handled correctly and should be safe. It should not cause a
    segmentation fault or crash.

Completed

Upstream

Build

  • ocaml-multicore/ocaml-multicore#662
    Disable changes check on 5.0

    The .github/workflows/hygiene.yml has been updated to disable
    check on 5.00 to avoid noise on the change entries.

  • ocaml-multicore/ocaml-multicore#676
    Fix 5.00 install

    The caml/byte_domain_state.tbl has been removed, and README.adoc
    has been renamed to README.stock.adoc in order to build cleanly
    with OCaml 5.00 branch.

Change

  • ocaml-multicore/ocaml-multicore#675
    Align Bytes.unsafe_of_string / Bytes.unsafe_to_string to OCaml trunk

    The Pbytes_to_string / Pbytes_of_string use in
    bytecomp/bytegen.ml are now aligned with upstream OCaml.

  • ocaml-multicore/ocaml-multicore#677
    Remove debugging nop

    The debugging nop primitive is not required for upstreaming and has
    been cleaned up. The PR also fixes check-typo whitespace in
    emit.mlp to match that trunk.

  • ocaml-multicore/ocaml-multicore#679
    Remove caml_read_field

    The use of caml_read_field has been removed as the existing
    Field provides all the necessary information making it closer to
    upstream OCaml.

  • ocaml-multicore/ocaml-multicore#681
    Revert to ocaml/trunk version of otherlibs/unix

    unixsupport.c, cstringv.c and files in otherlibs/unix have
    been updated to be similar to ocaml/ocaml.

  • ocaml-multicore/ocaml-multicore#684
    Remove historical for_handler and Reperform_noloc in lambda/matching

    The for_handler function and Reperform_noloc in
    lambda/matching.ml{,i} are not required to be upstreamed and hence
    have been removed.

  • ocaml-multicore/ocaml-multicore#685
    Remove Init_field from interp.c

    The interp.c file has been updated to be closer to
    ocaml/ocaml. The check-typo errors have been fixed, and the
    Init_field macro has been cleaned up.

  • ocaml-multicore/ocaml-multicore#704
    Remove Sync.poll and nanoseconds from Domain

    The Domain module has been updated to include only the changes
    required for upstreaming. Domain.Sync.poll, and
    Domain.nanosecond have been removed. Domain.Sync.cpu_relax has
    been renamed to Domain.cpu_relax. platform.h has been updated
    with fixes for check-typo.

  • ocaml-multicore/ocaml-multicore#706
    Revert otherlibs/win32unix to ocaml/trunk

    The otherlibs/win32unix/* files have been updated to be closer to
    ocaml/ocaml.

  • ocaml-multicore/ocaml-multicore#708
    Remove maybe stats

    The caml_maybe_print_stats primitive to output statistics and the
    s option to OCAMLRUNPARAM have now been removed.

  • ocaml-multicore/ocaml-multicore#724
    Runtime: Remove unused fields from io.h

    Remove revealed and old_revealed from runtime/caml/io.h as
    they have also been removed from ocaml/ocaml.

Diff
Merge

  • ocaml-multicore/ocaml#2
    Update trunk to the latest upstream trunk

    The PR is an attempt to help with the OCaml 5.0 difference
    output. With the changes, you can successfully do make && make tests. The summary of the results is provided below:

    Summary:
2918 tests passed
 40 tests skipped
  0 tests failed
105 tests not started (parent test skipped or failed)
  0 unexpected errors
3063 tests considered

  • ocaml-multicore/ocaml#3
    Latest 5.00 Commits

    The recent commits from trunk have now been merged to the
    ocaml-multicore 5.00 branch.

  • ocaml-multicore/ocaml-multicore#718
    Deprecate Sync and timer_ticks from Domain

    The patch synchronizes the changes to 4.12.0+domains+effects with
    the mainline 5.00 branch.

Thread Safe

  • ocaml-multicore/ocaml-multicore#632
    Str module multi domain safety

    The
    PR#635
    makes lib-str domain safe to work concurrently with Multicore
    OCaml.

  • ocaml-multicore/ocaml-multicore#672
    Codefrag thread safety

    The PR introduces a lock-free skiplist to make codefrag thread
    safe. The code fragments cannot be freed as soon as they are
    removed, but, they are added to a list and cleaned up during a later
    stop-the-world pause.

Fixes

Testsuite

  • ocaml-multicore/ocaml-multicore#654
    Enable effects tests

    The effect handler tests have now been re-added since the syntax
    support has been added to Multicore OCaml.

  • ocaml-multicore/ocaml-multicore#658
    Enable last dynlink test

    The lib-dynlink-private test has now been enabled to run in the CI.

  • ocaml-multicore/ocaml-multicore#659
    Reimport the threadsigmask test and remove systhread-todo test directory

    The lib-systhreads-todo test on signal handling and tick thread
    missing from systhreads has been reactivated in the the CI.

  • ocaml-multicore/ocaml-multicore#660
    Fixups and housekeeping for testsuite/disabled file

    The check-typo problems for 80 character line, and unnecessary
    test/promotion in testsuite/disabled have been fixed.

  • ocaml-multicore/ocaml-multicore#661
    Testsuite: Re-enable pr9971

    The pr9971 test has been re-enabled to run in the CI.

  • ocaml-multicore/ocaml-multicore#688
    Better signal handling in systhreads

    Improvements to the signal handling in systhreads that fixes the
    threadsigmask testcase failure in the CI.

  • ocaml-multicore/ocaml-multicore#712
    Otherlibs: Unix.kill should check for pending signals

    The unix_kill test case has been re-enabled to ensure that
    Unix.kill checks for pending signals on return.

Documentation

  • ocaml-multicore/ocaml-multicore#672
    Check-typo fixes for major_gc, so the changes in #672 don’t get clobbered

    A patch that fixes check-typo issues in runtime/major_gc.c.

  • ocaml-multicore/ocaml-multicore#696
    Stdlib: Fix typos in effectHandlers.mli

    A few typos in stdlib/effectHandlers.mli have been fixed.

  • ocaml-multicore/ocaml-multicore#697
    Remove dead code and clear up comments in the minor gc

    A non-functional change that clears up the comments in the minor and
    major GC files.

  • ocaml-multicore/ocaml-multicore#699
    Cleanup fiber implementation and add documentation

    The unused code in amd64.S has been removed and formatting has
    been fixed. The addition of 24 bytes at the top of the stack for an
    external call is no longer needed and has been removed.

  • ocaml-multicore/ocaml-multicore#713
    Clarify documentation of Lazy wrt. RacyLazy and Undefined exceptions.

    The documentation in stdlib/lazy.mli has been updated to clarify
    on the behaviour of try_force and thread safety.

  • ocaml-multicore/ocaml-multicore#717
    Tighten code comments in minor_gc.c

    The PR explains promotion of ephemeron keys to avoid introducing a
    barrier, and uses /* ... */ style comments.

  • ocaml-multicore#docs
    Docs

    A documentation repository for OCaml 5.00 that contains the design
    and proposed upstreaming plan.

Effect Handlers

Sundries

Ecosystem

Ongoing
Domainslib

  • ocaml-multicore/domainslib#43
    Possible bug in Task.pool management

    Török Edwin has reproduced the segmentation fault using
    4.12.0+domains with domainslib 0.3.1 on AMD Ryzen 3900X CPU, and has
    also provided a draft PR with a fix!

  • ocaml-multicore/domainslib#46
    Provide a way to iterate over all the pools

    A requirement to be able to iterate over all the pools created in
    domainslib. A use case is to tear down all the pools. A weak hash
    set can be used to store a weak pointer to the pools.

  • ocaml-multicore/domainslib#47
    Task.await deadlock (task finished but await never returns)

    A query on nesting Task.await inside Task.async, and
    Task.async inside Task.async. A sample code snippet, stack trace
    and platform information have also been provided to reproduce a
    deadlock scenario.

  • ocaml-multicore/domainslib#48
    Move ws_deque to lockfree

    A request to move the work-stealing deque in domainslib to
    ocaml-multicore/lockfree, and make domainslib depend on this new
    lockfree implementation.

  • ocaml-multicore/domainslib#49
    Should we expose multi-channel from the library?

    A query on whether Multicore OCaml users will find Non-FIFO
    multi-channel implementation useful. Domainslib already provides
    FIFO channels.

  • ocaml-multicore/domainslib#50
    Multi_channel: Allow more than one instance per program with different configurations

    A draft PR contributed by Török Edwin in lib/multi_channel.ml and
    lib/task.ml to remove use of a global key with a per-channel key.

  • ocaml-multicore/domainslib#51
    Utilise effect handlers

    The tasks are now created using effect handlers, and a new
    test_deadlock.ml tests the same. The change will work only with
    4.12+domains and 5.00. The performance results from the Turing
    machine (Intel Xeon Gold 5120 CPU @ 2.20 GHz, 28 isolated cores) is
    shown below:

    Domainslib-PR-51-performance

Sundries

  • ocaml-multicore/tezos#8
    ci.Dockerfile throws warning

    The ci.Dockerfile on Ubuntu 20.10 throws C99 warnings on _Atomic
    with GCC 10.3.0.

  • ocaml-multicore/tezos#10
    Fix make build-deps, fix NixOS support

    conf-perl is no longer required upstream and has been removed from
    the tezos-opam-repository. The patch also fixes make build-deps/build-dev-deps.

  • ocaml-multicore/ocaml-uring#39
    Test failures on NixOS

    The ocaml-uring master branch is showing test failures with dune runtest on NixOS.

  • ocaml-multicore/eio#85
    Any plans on supporting js_of_ocaml?

    A query by Konstantin A. Olkhovskiy (Lupus) on whether EIO can
    compile to JavaScript backend, assuming that js_of_ocaml gets
    support for effects.

Completed
Domainslib

  • ocaml-multicore/domainslib#45
    Add named pools

    An optional argument is now added to name a pool during setup. This
    name can be used to retrieve the pool later.

  • ocaml-multicore/domainslib#52
    Use a random number as the cache prefix to disable cache in CI

    The cache-prefix now uses a random number in
    .github/workflows/main.yml to disable cache in the CI.

  • ocaml-multicore/domainslib#53
    Make domainslib build/run with OCaml 5.00 after PR#704

    The CI has been updated to now build and run with OCaml 5.00 branch.

  • ocaml-multicore/domainslib#54
    Use last 4.12+domains+effects hash as the cache-key

    The cache-key now uses the last commit hash from OCaml Multicore in
    order to invalidate the cache in the CI.

Sundries

Benchmarking

Sandmark

Ongoing

  • ocaml-bench/sandmark#248
    Coq fails to build

    A new Coq tarball,
    coq-multicore-2021-09-24,
    builds with Multicore OCaml 4.12.0+domains, but, stdio.v0.14.0
    fails to build cleanly with 4.14.0+trunk because of a dune
    issue     that has been
    reported.

  • ocaml-bench/sandmark#260
    Add 5.00 branch for sequential run. Fix notebook.

    A new 5.00 OCaml variant branch has been added to Sandmark to
    track sequential benchmark runs in the CI.

Completed

  • ocaml-bench/sandmark#256
    Remove old variants

    The older variants, 4.05.*, 4.06.*, 4.07.*, 4.08.*,
    4.10.0.* have now been removed from Sandmark.

  • ocaml-bench/sandmark#258
    Document Makefile variables in README

    The README now contains documentation on the various Makefile
    variables that are used during building and execution of the
    benchmarks in Sandmark.

current-bench

Ongoing

  • ocurrent/current-bench#117
    Read stderr from the Docker container

    We would like to see any build failures from the benchmark execution
    inside the Docker container for debugging purposes.

  • ocurrent/current-bench#146
    Replicate ocaml-bench-server setup

    The TAG and OCaml variants need to be abstracted from the Sandmark
    Makefile to current-bench in order to be able to run the benchmarks
    for different compiler versions and developer branches.

Completed

(see the PR full for the full set of graphs, including major words and time taken)

Our special thanks to all the OCaml users, developers and contributors in the community for their valuable time and continued support to the project. Stay safe!

Acronyms

  • AMD: Advanced Micro Devices
  • CI: Continuous Integration
  • CPU: Central Processing Unit
  • DLS: Domain Local Storage
  • FIFO: First In, First Out
  • GB: Gigabyte
  • GC: Garbage Collector
  • GCC: GNU Compiler Collection
  • IO: Input/Output
  • OPAM: OCaml Package Manager
  • OS: Operating System
  • PR: Pull Request
  • TB: Terabyte
  • URL: Uniform Resource Locator
44 Likes