I was trying to build dune-configurator-3.15.3 using an OCaml 5.2+tsan switch.
After working around a TSAN bug (c - FATAL: ThreadSanitizer: unexpected memory mapping when running on Linux Kernels 6.6+ - Stack Overflow had to set ASLR bits to 28) I got this error report from inside the OCaml runtime:
==================
WARNING: ThreadSanitizer: data race (pid=1422866)
Atomic write of size 8 at 0x7f248fdff518 by main thread (mutexes: write M0):
#0 oldify_one runtime/minor_gc.c:250 (dune+0xf72325) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#1 caml_do_local_roots runtime/roots.c:64 (dune+0xf77476) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#2 caml_thread_scan_roots /home/edvint/.opam/5.2+tsan/.opam-switch/build/ocaml-variants.5.2.0+options/otherlibs/systhreads/st_stubs.c:179 (dune+0xf2370d) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#3 caml_empty_minor_heap_promote runtime/minor_gc.c:615 (dune+0xf735c0) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#4 caml_stw_empty_minor_heap_no_major_slice runtime/minor_gc.c:746 (dune+0xf739e5) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#5 caml_stw_empty_minor_heap runtime/minor_gc.c:777 (dune+0xf73c92) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#6 caml_try_run_on_all_domains_with_spin_work runtime/domain.c:1576 (dune+0xf4f533) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#7 caml_try_empty_minor_heap_on_all_domains runtime/minor_gc.c:808 (dune+0xf73dca) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#8 caml_empty_minor_heaps_once runtime/minor_gc.c:829 (dune+0xf73dca)
#9 caml_poll_gc_work runtime/domain.c:1745 (dune+0xf4ee10) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#10 caml_handle_gc_interrupt runtime/domain.c:1778 (dune+0xf4fba5) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#11 caml_do_pending_actions_exn runtime/signals.c:341 (dune+0xf7cc34) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#12 caml_alloc_small_dispatch runtime/minor_gc.c:851 (dune+0xf73e4c) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#13 caml_garbage_collection runtime/signals_nat.c:84 (dune+0xf89e10) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#14 caml_call_gc <null> (dune+0xf87ad1) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#15 camlDune_rules__Scope.fun_5068 src/dune_rules/scope.ml:105 (dune+0x898582) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#16 camlDune_rules__Lib.resolve_name_6800 src/dune_rules/lib.ml:1135 (dune+0x8b0873) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#17 camlDune_rules__Lib.find_even_when_hidden_7883 src/dune_rules/lib.ml:1871 (dune+0x8bb2d6) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#18 camlDune_rules__Lib.get_compile_info_7913 src/dune_rules/lib.ml:1901 (dune+0x8bbd70) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#19 camlDune_rules__Lib_rules.rules_5492 src/dune_rules/lib_rules.ml:648 (dune+0x7c23b8) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#20 camlDune_rules__Gen_rules.if_available_buildable_2319 src/dune_rules/gen_rules.ml:102 (dune+0x76bdd7) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#21 caml_apply2 <null> (dune+0x6e7227) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#22 camlFiber__Scheduler.advance_1192 vendor/fiber/src/scheduler.ml:194 (dune+0xd6d2be) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#23 camlFiber.loop_391 vendor/fiber/src/fiber.ml:15 (dune+0xd62aeb) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#24 camlDune_engine__Scheduler.run_3186 vendor/fiber/src/fiber.ml:17 (dune+0xcc9d26) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#25 camlDune_engine__Scheduler.run_and_cleanup_3379 src/dune_engine/scheduler.ml:1141 (dune+0xcca4be) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#26 camlDune_engine__Scheduler.go_inner_5501 src/dune_engine/scheduler.ml:1363 (dune+0xcccb5f) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#27 camlCmdliner_term.fun_639 vendor/cmdliner/src/cmdliner_term.ml:24 (dune+0xe32da2) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#28 camlCmdliner_eval.run_parser_576 vendor/cmdliner/src/cmdliner_eval.ml:34 (dune+0xe2a865) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#29 camlCmdliner_eval.eval_value_inner_1700 vendor/cmdliner/src/cmdliner_eval.ml:202 (dune+0xe2c0e4) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#30 camlMain.entry bin/main.ml:106 (dune+0x6e7a39) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#31 caml_program <null> (dune+0x6df539) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#32 caml_start_program <null> (dune+0xf87e43) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#33 caml_startup_common runtime/startup_nat.c:132 (dune+0xf875ec) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#34 caml_startup_common runtime/startup_nat.c:88 (dune+0xf875ec)
#35 caml_startup_exn runtime/startup_nat.c:139 (dune+0xf87697) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#36 caml_startup runtime/startup_nat.c:144 (dune+0xf87697)
#37 caml_main runtime/startup_nat.c:151 (dune+0xf87697)
#38 main runtime/main.c:37 (dune+0x6dbbd5) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
Previous read of size 8 at 0x7f248fdff518 by thread T4:
#0 caml_unix_open /home/edvint/.opam/5.2+tsan/.opam-switch/build/ocaml-variants.5.2.0+options/otherlibs/unix/open_unix.c:79 (dune+0xf29dfb) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#1 caml_c_call <null> (dune+0xf87d27) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#2 camlDune_digest.file_787 src/dune_digest/dune_digest.ml:24 (dune+0xda6d3c) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#3 camlDune_shared_cache.digest_1966 src/dune_digest/dune_digest.ml:87 (dune+0x760c8b) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#4 camlStdune__Exn_with_backtrace.try_with_430 otherlibs/stdune/src/exn_with_backtrace.ml:9 (dune+0xdc94d8) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#5 camlDune_engine__Scheduler.f_3423 src/dune_engine/scheduler.ml:1153 (dune+0xcca806) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#6 camlDune_thread_pool__Thread_pool.loop_527 src/dune_thread_pool/thread_pool.ml:39 (dune+0xcd0c59) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#7 camlDune_engine__Scheduler.f_1701 src/dune_engine/scheduler.ml:102 (dune+0xcc413d) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#8 camlThread.fun_768 /home/edvint/.opam/5.2+tsan/.opam-switch/build/ocaml-variants.5.2.0+options/otherlibs/systhreads/thread.ml:48 (dune+0xe4f360) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#9 caml_start_program <null> (dune+0xf87e43) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#10 caml_callback_exn runtime/callback.c:201 (dune+0xf4a493) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
#11 caml_thread_start /home/edvint/.opam/5.2+tsan/.opam-switch/build/ocaml-variants.5.2.0+options/otherlibs/systhreads/st_stubs.c:634 (dune+0xf250a7) (BuildId: f5f81ea033312aa0209de3bf7f7d92b167faded7)
Location is stack of thread T4.
The code:
if (!(Is_block(v) && Is_young(v))) {
/* not a minor block */
(* ---> *) *p = v;
return;
}
caml_enter_blocking_section();
(* ---> *) fd = open(p, cv_flags, Int_val(perm));
caml_leave_blocking_section();
And as the message says the race is between the main thread writing to a location that was previously part of thread 4âs stack (last read by this read function).
Now this either refers to some kind of memory corruption where some object in the main threadâs heap is part of thread 4âs stack (I donât really see how, thread 4 mostly has integers on its stack, unless the main thread got a stale/dangling pointer), or memory got reused, and what used to be thread 4âs stack is now in fact the main threadâs heap, but thread sanitizer doesnât know or didnât get told about that, so it doesnât see the main threadâs write as initializing write.
Has someone seen this before? Could this be a side-effect of OCaml switching stacks and tsan getting confused about it?
FWIW rerunning the build succeeds, so it doesnât always reproduce, although if I build enough packages with dune, itâll eventually fail like this again, interestingly the race is always between oldify_one + unix_open.