Multicore OCaml: August 2021

Multicore OCaml: August 2021

Welcome to the August 2021 Multicore OCaml monthly report! The following update and the previous updates have been compiled by me, @ctk21, @kayceesrk and @shakthimaan. This month’s update is a bit quieter as August is also a period of downtime in Europe (and our crew in India also took well-deserved time off), but we all participated in the online OCaml Workshop which was held virtually this year, so there are plenty of videos to watch!

The multicore effort is all on track for integration into OCaml 5.0 early next year, with the core team currently organising the upstreaming code review strategy over the coming winter months. Meanwhile, there are some blog posts and videos from the OCaml Workshop which give more detailed updates on both domains-parallelism and effects.

  • Adapting the OCaml ecosystem for Multicore OCaml
  • Parafuzz coverage guided Property Fuzzing for Multicore OCaml programs
    • We develop ParaFuzz, an input and concurrency fuzzing tool for Multicore OCaml programs. ParaFuzz builds on top of Crowbar which combines AFL-based grey box fuzzing with QuickCheck and extends it to handle parallelism.
    • Video
  • Experiences with Effects
    • The multicore branch of OCaml adds support for effect handlers. In this talk, we report our experiences with effects, both from converting existing code, and from writing new code.
    • Video

As always, the Multicore OCaml updates are listed first, which are then followed by the updates from the Ecosystem libraries and Sandmark benchmarking.

Multicore OCaml

  • ocaml-multicore/ocaml-multicore#632
    Str module multi domain safety

    An issue on stdlib safety in the OCaml Str module to work
    concurrently with Multicore OCaml.

  • ocaml-multicore/ocaml-multicore#633
    Error building 4.12.0+domains with no-flat-float-arrays

    A linker error observed by Adrián Montesinos González (debugnik)
    when installing 4.12.0+domains no-flat-float-arrays.

  • ocaml-multicore/ocaml-multicore#634
    Strange type errors from merlin (This expression has type string/1)

    Type errors reported from merlin (4.3.1-412) when using the effects
    version of the Multicore OCaml compiler.

  • ocaml-multicore/ocaml-multicore#624
    core v0.14: test triggers a segfault in the GC

    The root cause of the segfault when running core.v0.14 test suite
    with Multicore OCaml 4.12.0+domains has been identified.

  • ocaml-multicore/ocaml-multicore#573
    Backport trunk safepoints PR to multicore

    This is an on-going effort to backport the Safepoints implementation to Multicore OCaml.

Ecosystem

Ongoing

  • ocaml-multicore/parallel-programming-in-multicore-ocaml#10
    Edited for flow/syntax/consistency

    The Parallel Programming in Multicore OCaml chapter has been updated
    for consistency, syntax flow and grammar.

  • ocaml-multicore/retro-httpaf-bench#15
    Optimise Go code

    The nethttp-go/httpserv.go benchmark has been optimised with use
    of Write instead of fmt.Fprintf, and the removal of yield().

    retro-httpaf-bench-go-optimise

  • ocaml-multicore/ocaml-uring#37
    poll_add test hangs on s390x

    The use of poll_add causes a hang on s390x architecture. A
    backtrace with GDB is provided for reference:

    (gdb) bt
     #0  0x000003ffb63ec01e in __GI___libc_write (nbytes=<optimized out>, buf=<optimized out>, fd=<optimized out>)
         at ../sysdeps/unix/sysv/linux/write.c:26
     #1  __GI___libc_write (fd=<optimized out>, buf=0x3ffffdee8e0, nbytes=1) at ../sysdeps/unix/sysv/linux/write.c:24
     #2  0x000002aa0dbb0ca2 in unix_write (fd=<optimized out>, buf=<optimized out>, vofs=<optimized out>, vlen=<optimized out>) at write.c:44
     #3  0x000002aa0dbd4d3a in caml_c_call ()
    
  • ocaml-multicore/domainslib#37
    parallel_map

    @UnixJunkie has provided a simplified version for the interface for
    scientific parallel programming as recommended by the parany library.

    val run:
    ?csize:int ->
    ~nprocs: int ->
    demux:(unit -> 'a) ->
    work:('a -> 'b) ->
    mux:('b -> unit) -> unit
    
  • ocaml-multicore/domainslib#39
    Add a fast path in parallel scan

    A patch that performs a sequential scan when the number of elements
    is less than or equal to the pool size or if the number of domains
    is one.

  • ocaml-multicore/domainslib#40
    Parallel map

    A PR that implements parallel_map in lib/task.ml that includes an
    optional chunk size parameter.

Completed

  • ocaml-multicore/retro-httpaf-bench#13
    Update EIO for performance improvements, multiple domains

    httpf-eio has been enhanced with performance improvements when
    running with multiple domains. The results on an 8-core VM with 100
    connections and 5 second runs is shown below:

    retro-httpaf-bench-100-connections-5-seconds

    The following illustration is from a VM for 1000 connections and 60
    second runs:

    retro-httpaf-bench-1000-connections-60-seconds

    The results with GOMAXPROCS=3 for three OCaml domains is as follows:

    retro-httpaf-bench-three-domains-15

  • ocaml-multicore/ocaml-uring#36
    Update to cstruct 6.0.1

    ocaml-uring now uses Cstruct.shiftv and has been updated to use cstruct.6.0.1.

  • ocaml-multicore/domainslib#41
    Use the master branch in the link to usage examples

    The README.md file has been updated to point to the sample programs in the master branch that use the new num_additional_domains argument label.

  • The Multicore OCaml concurrency bug detection tool named ParaFuzz is now available in GitHub as Free/Libre and Open Source Software.

  • Tezos is a proof-of-stake distributed consensus platform designed to evolve, and is written in OCaml. The version of the Tezos daemon that now runs on Multicore OCaml is also available in GitHub as a work-in-progress fork.

Eio

The eio library provides an effects-based parallel IO stack for
Multicore OCaml.

Completed
  • ocaml-multicore/eio#68
    Add eio_luv backend

    We now use luv, which has OCaml/Reason bindings to libuv, to
    provide a cross-platform default backend for eio.

  • ocaml-multicore/eio#72
    Add non-deterministic to abstract domain socket test

    The inclusion of non-deterministic=command to disable a regular
    dune runtest for the failing abstract domain socket test.

  • ocaml-multicore/eio#73
    Work-around for io_uring bug reading from terminals

    A work-around to fix IORING_OP_READ that causes io_uring_enter
    to block the entire process when reading from a terminal.

  • ocaml-multicore/eio#74
    Don’t crash when receiving a signal

    A patch to receive a signal and not crash in
    lib_eio_linux/eio_linux.ml.

  • ocaml-multicore/eio#75
    Add Eio.Stream

    The Stream module has been added to Eio that implements bounded
    queues with cancellation.

  • ocaml-multicore/eio#76
    Link to some eio examples

    The README.md has been updated to point to existing eio example
    project sources.

  • ocaml-multicore/eio#77
    Disable opam file generation due to dune bug

    The opam file generation with dune.2.9.0 is broken as dune does not
    have the subst --root option. Hence, the same is now disabled in
    the eio build steps.

  • ocaml-multicore/eio#79
    Initial edits for consistency, formatting and clarity

    Changes in the README.md file for consistency, syntax formatting and
    for clarity.

Benchmarking

Sandmark

  • ocaml-bench/sandmark#251
    Update dependencies to work with 4.14.0+trunk

    A series of patches that update the dependencies in Sandmark to
    build 4.14.0+trunk with dune.2.9.0.

  • We are continuing to integrate and test building of 4.12.0 OCaml
    variants with Sandmark-2.0 with current-bench for both sequential
    and parallel benchmarks.

Our thanks to all the OCaml users, developers and contributors in the
community for their valuable time and support to the project. Stay
safe!

Acronyms

  • GC: Garbage Collector
  • GDB: GNU Project Debugger
  • HTTP: Hypertext Transfer Protocol
  • ICFP: International Conference on Functional Programming
  • IO: Input/Output
  • PR: Pull Request
  • OPAM: OCaml Package Manager
  • VM: Virtual Machine
42 Likes

It’s great to see a timeframe of “early next year” be mentioned. Finally there is a date even though it may be approximate and non-binding :slight_smile:

Looking forward to seeing multicore land!

2 Likes

I want to give props to @talex5 for the Experiences with Effects presentation which I found interesting, honest and to the point. On top of that you’re a very clear communicator!

5 Likes

Oh man that was such a fun and easy 18 mins to watch. I hadn’t had a walkthrough yet of the actual ergonomics of writing effects. Saw slides, but this was more tangible and understandable.

1 Like

Really stoked to see it happening. Congrats y’all, looking forward to catching up on those videos and tutorials.

1 Like