I’ve written a tutorial on Record/Replay debugging. If you’re interested in reverse debugging please check it out here.
The tutorial teaches you how to record/replay and debug a program using rr.soft through a small example program written in C. Though the tutorial focus is on C, rr.soft is useful for OCaml-ers too. For instance, you can do Record/Replay style debugging of your ocamlc.opt compiled executables. You will be able to place breakpoints on all symbols exposed by the ocaml.opt compiler – this includes C functions + OCaml functions from your OCaml program and do things like reverse-continue.
For more context please see an announcement I made a couple of months ago:
(What’s new in this new discuss.ocaml.org post is to give focus to the tutorial I’ve very recently written)
How can rr / Software Counters mode rr help me in OCaml if I’m not interested in gdb style debugging the OCaml executables ?
There are some other useful things that rr can do for OCaml-ers.
Here is one: Let’s say you want to know what programs have been executed when you compile a ocaml program via dune.
$ dune init project hello_world
$ cd hello_world/
$ rr record -W -- dune b
rr: Saving execution to trace directory `/home/sidk/.local/share/rr/dune-5'.
Now run rr ps. You will get the exact invocations of all the programs that dune build triggered ! This takes you all the way down to showing you ld and ar invocations !
Very useful when you want to know what happened in the whole process tree.
$ rr ps
PID PPID EXIT CMD
28138 -- 0 dune b
28323 28138 0 /usr/bin/nproc
28360 28138 0 /home/sidk/ocaml53/_opam/bin/ocamlc.opt -config
28421 28138 0 /home/sidk/ocaml53/_opam/bin/ocamlc.opt -w @1..3@5..28@31..39@43@46..47@49..57@61..62@67@69-40 -strict-sequence -strict-formats -short-paths -keep-locs -w -49 -nopervasives -nostdlib -g -bin-annot -bin-annot-occurrences -I lib/.hello_world.objs/byte -no-alias-deps -opaque -o lib/.hello_world.objs/byte/hello_world.cmo -c -impl lib/hello_world.ml-gen
28422 28138 0 /home/sidk/ocaml53/_opam/bin/ocamlc.opt -w @1..3@5..28@31..39@43@46..47@49..57@61..62@67@69-40 -strict-sequence -strict-formats -short-paths -keep-locs -g -bin-annot -bin-annot-occurrences -I test/.test_hello_world.eobjs/byte -no-alias-deps -opaque -o test/.test_hello_world.eobjs/byte/dune__exe__Test_hello_world.cmi -c -intf test/test_hello_world.mli
28423 28138 0 /home/sidk/ocaml53/_opam/bin/ocamlopt.opt -w @1..3@5..28@31..39@43@46..47@49..57@61..62@67@69-40 -strict-sequence -strict-formats -short-paths -keep-locs -w -49 -nopervasives -nostdlib -g -I lib/.hello_world.objs/byte -I lib/.hello_world.objs/native -intf-suffix .ml-gen -no-alias-deps -opaque -o lib/.hello_world.objs/native/hello_world.cmx -c -impl lib/hello_world.ml-gen
28454 28423 0 sh -c -- as --debug-prefix-map '/home/sidk/ocaml53/hello_world/_build/default'='/workspace_root' -o 'lib/.hello_world.objs/native/hello_world.o' '/tmp/build_6b9f5e_dune/camlasm077455.s'
28665 28138 0 /home/sidk/ocaml53/_opam/bin/ocamlc.opt -w @1..3@5..28@31..39@43@46..47@49..57@61..62@67@69-40 -strict-sequence -strict-formats -short-paths -keep-locs -g -a -o lib/hello_world.cma lib/.hello_world.objs/byte/hello_world.cmo
28666 28138 0 /home/sidk/ocaml53/_opam/bin/ocamlc.opt -w @1..3@5..28@31..39@43@46..47@49..57@61..62@67@69-40 -strict-sequence -strict-formats -short-paths -keep-locs -g -bin-annot -bin-annot-occurrences -I bin/.main.eobjs/byte -I lib/.hello_world.objs/byte -no-alias-deps -opaque -o bin/.main.eobjs/byte/dune__exe__Main.cmi -c -intf bin/main.mli
28667 28138 0 /home/sidk/ocaml53/_opam/bin/ocamlopt.opt -w @1..3@5..28@31..39@43@46..47@49..57@61..62@67@69-40 -strict-sequence -strict-formats -short-paths -keep-locs -g -I test/.test_hello_world.eobjs/byte -I test/.test_hello_world.eobjs/native -intf-suffix .ml -no-alias-deps -opaque -o test/.test_hello_world.eobjs/native/dune__exe__Test_hello_world.cmx -c -impl test/test_hello_world.ml
28668 28667 0 sh -c -- as --debug-prefix-map '/home/sidk/ocaml53/hello_world/_build/default'='/workspace_root' -o 'test/.test_hello_world.eobjs/native/dune__exe__Test_hello_world.o' '/tmp/build_6b9f5e_dune/camlasm566f3b.s'
28669 28138 0 /home/sidk/ocaml53/_opam/bin/ocamlopt.opt -w @1..3@5..28@31..39@43@46..47@49..57@61..62@67@69-40 -strict-sequence -strict-formats -short-paths -keep-locs -g -a -o lib/hello_world.cmxa lib/.hello_world.objs/native/hello_world.cmx
28670 28669 0 sh -c -- ar rc 'lib/hello_world.a' 'lib/.hello_world.objs/native/hello_world.o'
29001 28138 0 /home/sidk/ocaml53/_opam/bin/ocamlopt.opt -w @1..3@5..28@31..39@43@46..47@49..57@61..62@67@69-40 -strict-sequence -strict-formats -short-paths -keep-locs -g -I bin/.main.eobjs/byte -I bin/.main.eobjs/native -I lib/.hello_world.objs/byte -I lib/.hello_world.objs/native -intf-suffix .ml -no-alias-deps -opaque -o bin/.main.eobjs/native/dune__exe__Main.cmx -c -impl bin/main.ml
29002 29001 0 sh -c -- as --debug-prefix-map '/home/sidk/ocaml53/hello_world/_build/default'='/workspace_root' -o 'bin/.main.eobjs/native/dune__exe__Main.o' '/tmp/build_6b9f5e_dune/camlasma7168c.s'
29003 28138 0 /home/sidk/ocaml53/_opam/bin/ocamlopt.opt -w @1..3@5..28@31..39@43@46..47@49..57@61..62@67@69-40 -strict-sequence -strict-formats -short-paths -keep-locs -g -o test/test_hello_world.exe test/.test_hello_world.eobjs/native/dune__exe__Test_hello_world.cmx
29004 29003 0 sh -c -- as --debug-prefix-map '/home/sidk/ocaml53/hello_world/_build/default'='/workspace_root' -o '/tmp/build_6b9f5e_dune/camlstartup3c9e41.o' '/tmp/build_6b9f5e_dune/camlstartup065684.s'
29005 29003 0 sh -c -- gcc -Wl,-E -o 'test/test_hello_world.exe' '-L/home/sidk/ocaml53/_opam/lib/ocaml' '/tmp/build_6b9f5e_dune/camlstartup3c9e41.o' '/home/sidk/ocaml53/_opam/lib/ocaml/std_exit.o' 'test/.test_hello_world.eobjs/native/dune__exe__Test_hello_world.o' '/home/sidk/ocaml53/_opam/lib/ocaml/stdlib.a' '/home/sidk/ocaml53/_opam/lib/ocaml/libasmrun.a' -lm -lpthread
29036 29005 0 /usr/libexec/gcc/aarch64-redhat-linux/15/collect2 -plugin /usr/libexec/gcc/aarch64-redhat-linux/15/liblto_plugin.so -plugin-opt=/usr/libexec/gcc/aarch64-redhat-linux/15/lto-wrapper -plugin-opt=-fresolution=/tmp/build_6b9f5e_dune/ccBzNoZJ.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s --build-id --no-add-needed --eh-frame-hdr --hash-style=gnu -dynamic-linker /lib/ld-linux-aarch64.so.1 -X -EL -maarch64linux -o test/test_hello_world.exe /usr/lib/gcc/aarch64-redhat-linux/15/../../../../lib64/crt1.o /usr/lib/gcc/aarch64-redhat-linux/15/../../../../lib64/crti.o /usr/lib/gcc/aarch64-redhat-linux/15/crtbegin.o -L/home/sidk/ocaml53/_opam/lib/ocaml -L/usr/lib/gcc/aarch64-redhat-linux/15 -L/usr/lib/gcc/aarch64-redhat-linux/15/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/usr/lib/gcc/aarch64-redhat-linux/15/../../.. -L/lib -L/usr/lib -E /tmp/build_6b9f5e_dune/camlstartup3c9e41.o /home/sidk/ocaml53/_opam/lib/ocaml/std_exit.o test/.test_hello_world.eobjs/native/dune__exe__Test_hello_world.o /home/sidk/ocaml53/_opam/lib/ocaml/stdlib.a /home/sidk/ocaml53/_opam/lib/ocaml/libasmrun.a -lm -lpthread -lgcc --push-state --as-needed -lgcc_s --pop-state -lc -lgcc --push-state --as-needed -lgcc_s --pop-state /usr/lib/gcc/aarch64-redhat-linux/15/crtend.o /usr/lib/gcc/aarch64-redhat-linux/15/../../../../lib64/crtn.o
29067 29036 0 /usr/bin/ld -plugin /usr/libexec/gcc/aarch64-redhat-linux/15/liblto_plugin.so -plugin-opt=/usr/libexec/gcc/aarch64-redhat-linux/15/lto-wrapper -plugin-opt=-fresolution=/tmp/build_6b9f5e_dune/ccBzNoZJ.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s --build-id --no-add-needed --eh-frame-hdr --hash-style=gnu -dynamic-linker /lib/ld-linux-aarch64.so.1 -X -EL -maarch64linux -o test/test_hello_world.exe /usr/lib/gcc/aarch64-redhat-linux/15/../../../../lib64/crt1.o /usr/lib/gcc/aarch64-redhat-linux/15/../../../../lib64/crti.o /usr/lib/gcc/aarch64-redhat-linux/15/crtbegin.o -L/home/sidk/ocaml53/_opam/lib/ocaml -L/usr/lib/gcc/aarch64-redhat-linux/15 -L/usr/lib/gcc/aarch64-redhat-linux/15/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/usr/lib/gcc/aarch64-redhat-linux/15/../../.. -L/lib -L/usr/lib -E /tmp/build_6b9f5e_dune/camlstartup3c9e41.o /home/sidk/ocaml53/_opam/lib/ocaml/std_exit.o test/.test_hello_world.eobjs/native/dune__exe__Test_hello_world.o /home/sidk/ocaml53/_opam/lib/ocaml/stdlib.a /home/sidk/ocaml53/_opam/lib/ocaml/libasmrun.a -lm -lpthread -lgcc --push-state --as-needed -lgcc_s --pop-state -lc -lgcc --push-state --as-needed -lgcc_s --pop-state /usr/lib/gcc/aarch64-redhat-linux/15/crtend.o /usr/lib/gcc/aarch64-redhat-linux/15/../../../../lib64/crtn.o
29158 28138 0 /home/sidk/ocaml53/_opam/bin/ocamlopt.opt -w @1..3@5..28@31..39@43@46..47@49..57@61..62@67@69-40 -strict-sequence -strict-formats -short-paths -keep-locs -g -shared -linkall -I lib -o lib/hello_world.cmxs lib/hello_world.cmxa
29159 29158 0 sh -c -- as --debug-prefix-map '/home/sidk/ocaml53/hello_world/_build/default'='/workspace_root' -o 'lib/hello_world.cmxs.startup.o' '/tmp/build_6b9f5e_dune/camlstartup219492.s'
29160 29158 0 sh -c -- gcc -shared -o 'lib/hello_world.cmxs' '-Llib' '-L/home/sidk/ocaml53/_opam/lib/ocaml' 'lib/hello_world.cmxs.startup.o' 'lib/hello_world.a'
29161 29160 0 /usr/libexec/gcc/aarch64-redhat-linux/15/collect2 -plugin /usr/libexec/gcc/aarch64-redhat-linux/15/liblto_plugin.so -plugin-opt=/usr/libexec/gcc/aarch64-redhat-linux/15/lto-wrapper -plugin-opt=-fresolution=/tmp/build_6b9f5e_dune/cc1SyPyv.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s --build-id --no-add-needed --eh-frame-hdr --hash-style=gnu -shared -X -EL -maarch64linux -o lib/hello_world.cmxs /usr/lib/gcc/aarch64-redhat-linux/15/../../../../lib64/crti.o /usr/lib/gcc/aarch64-redhat-linux/15/crtbeginS.o -Llib -L/home/sidk/ocaml53/_opam/lib/ocaml -L/usr/lib/gcc/aarch64-redhat-linux/15 -L/usr/lib/gcc/aarch64-redhat-linux/15/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/usr/lib/gcc/aarch64-redhat-linux/15/../../.. -L/lib -L/usr/lib lib/hello_world.cmxs.startup.o lib/hello_world.a -lgcc --push-state --as-needed -lgcc_s --pop-state -lc -lgcc --push-state --as-needed -lgcc_s --pop-state /usr/lib/gcc/aarch64-redhat-linux/15/crtendS.o /usr/lib/gcc/aarch64-redhat-linux/15/../../../../lib64/crtn.o
29162 29161 0 /usr/bin/ld -plugin /usr/libexec/gcc/aarch64-redhat-linux/15/liblto_plugin.so -plugin-opt=/usr/libexec/gcc/aarch64-redhat-linux/15/lto-wrapper -plugin-opt=-fresolution=/tmp/build_6b9f5e_dune/cc1SyPyv.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s --build-id --no-add-needed --eh-frame-hdr --hash-style=gnu -shared -X -EL -maarch64linux -o lib/hello_world.cmxs /usr/lib/gcc/aarch64-redhat-linux/15/../../../../lib64/crti.o /usr/lib/gcc/aarch64-redhat-linux/15/crtbeginS.o -Llib -L/home/sidk/ocaml53/_opam/lib/ocaml -L/usr/lib/gcc/aarch64-redhat-linux/15 -L/usr/lib/gcc/aarch64-redhat-linux/15/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/usr/lib/gcc/aarch64-redhat-linux/15/../../.. -L/lib -L/usr/lib lib/hello_world.cmxs.startup.o lib/hello_world.a -lgcc --push-state --as-needed -lgcc_s --pop-state -lc -lgcc --push-state --as-needed -lgcc_s --pop-state /usr/lib/gcc/aarch64-redhat-linux/15/crtendS.o /usr/lib/gcc/aarch64-redhat-linux/15/../../../../lib64/crtn.o
29163 28138 0 /home/sidk/ocaml53/_opam/bin/ocamlopt.opt -w @1..3@5..28@31..39@43@46..47@49..57@61..62@67@69-40 -strict-sequence -strict-formats -short-paths -keep-locs -g -o bin/main.exe lib/hello_world.cmxa bin/.main.eobjs/native/dune__exe__Main.cmx
29164 29163 0 sh -c -- as --debug-prefix-map '/home/sidk/ocaml53/hello_world/_build/default'='/workspace_root' -o '/tmp/build_6b9f5e_dune/camlstartupe085fd.o' '/tmp/build_6b9f5e_dune/camlstartup6ebf4f.s'
29165 29163 0 sh -c -- gcc -Wl,-E -o 'bin/main.exe' '-L/home/sidk/ocaml53/_opam/lib/ocaml' '/tmp/build_6b9f5e_dune/camlstartupe085fd.o' '/home/sidk/ocaml53/_opam/lib/ocaml/std_exit.o' 'bin/.main.eobjs/native/dune__exe__Main.o' 'lib/hello_world.a' '/home/sidk/ocaml53/_opam/lib/ocaml/stdlib.a' '/home/sidk/ocaml53/_opam/lib/ocaml/libasmrun.a' -lm -lpthread
29166 29165 0 /usr/libexec/gcc/aarch64-redhat-linux/15/collect2 -plugin /usr/libexec/gcc/aarch64-redhat-linux/15/liblto_plugin.so -plugin-opt=/usr/libexec/gcc/aarch64-redhat-linux/15/lto-wrapper -plugin-opt=-fresolution=/tmp/build_6b9f5e_dune/ccVb7fsY.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s --build-id --no-add-needed --eh-frame-hdr --hash-style=gnu -dynamic-linker /lib/ld-linux-aarch64.so.1 -X -EL -maarch64linux -o bin/main.exe /usr/lib/gcc/aarch64-redhat-linux/15/../../../../lib64/crt1.o /usr/lib/gcc/aarch64-redhat-linux/15/../../../../lib64/crti.o /usr/lib/gcc/aarch64-redhat-linux/15/crtbegin.o -L/home/sidk/ocaml53/_opam/lib/ocaml -L/usr/lib/gcc/aarch64-redhat-linux/15 -L/usr/lib/gcc/aarch64-redhat-linux/15/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/usr/lib/gcc/aarch64-redhat-linux/15/../../.. -L/lib -L/usr/lib -E /tmp/build_6b9f5e_dune/camlstartupe085fd.o /home/sidk/ocaml53/_opam/lib/ocaml/std_exit.o bin/.main.eobjs/native/dune__exe__Main.o lib/hello_world.a /home/sidk/ocaml53/_opam/lib/ocaml/stdlib.a /home/sidk/ocaml53/_opam/lib/ocaml/libasmrun.a -lm -lpthread -lgcc --push-state --as-needed -lgcc_s --pop-state -lc -lgcc --push-state --as-needed -lgcc_s --pop-state /usr/lib/gcc/aarch64-redhat-linux/15/crtend.o /usr/lib/gcc/aarch64-redhat-linux/15/../../../../lib64/crtn.o
29167 29166 0 /usr/bin/ld -plugin /usr/libexec/gcc/aarch64-redhat-linux/15/liblto_plugin.so -plugin-opt=/usr/libexec/gcc/aarch64-redhat-linux/15/lto-wrapper -plugin-opt=-fresolution=/tmp/build_6b9f5e_dune/ccVb7fsY.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s --build-id --no-add-needed --eh-frame-hdr --hash-style=gnu -dynamic-linker /lib/ld-linux-aarch64.so.1 -X -EL -maarch64linux -o bin/main.exe /usr/lib/gcc/aarch64-redhat-linux/15/../../../../lib64/crt1.o /usr/lib/gcc/aarch64-redhat-linux/15/../../../../lib64/crti.o /usr/lib/gcc/aarch64-redhat-linux/15/crtbegin.o -L/home/sidk/ocaml53/_opam/lib/ocaml -L/usr/lib/gcc/aarch64-redhat-linux/15 -L/usr/lib/gcc/aarch64-redhat-linux/15/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/usr/lib/gcc/aarch64-redhat-linux/15/../../.. -L/lib -L/usr/lib -E /tmp/build_6b9f5e_dune/camlstartupe085fd.o /home/sidk/ocaml53/_opam/lib/ocaml/std_exit.o bin/.main.eobjs/native/dune__exe__Main.o lib/hello_world.a /home/sidk/ocaml53/_opam/lib/ocaml/stdlib.a /home/sidk/ocaml53/_opam/lib/ocaml/libasmrun.a -lm -lpthread -lgcc --push-state --as-needed -lgcc_s --pop-state -lc -lgcc --push-state --as-needed -lgcc_s --pop-state /usr/lib/gcc/aarch64-redhat-linux/15/crtend.o /usr/lib/gcc/aarch64-redhat-linux/15/../../../../lib64/crtn.o