# \[ANN\] A tool to reverse debug OCaml/other binary runs **URL:** \ **Category:** Community\ **Tags:** debugging, announce\ **Created:** [March 24, 2025, 1:08pm UTC](https://discuss.ocaml.org/t/ann-a-tool-to-reverse-debug-ocaml-other-binary-runs/16366 "2025-03-24T13:08:38Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![sid](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/sid/32/1476_2.png) [@sid](https://discuss.ocaml.org/u/sid)\ **Post date:** [March 24, 2025, 1:08pm UTC](https://discuss.ocaml.org/t/ann-a-tool-to-reverse-debug-ocaml-other-binary-runs/16366/1 "2025-03-24T13:08:38Z")
I’d like to announce a debugging tool I’ve built ! It’s called **_Software Counters mode_ `rr`**. It is available at [GitHub - sidkshatriya/rr.soft: Use the rr debugger without HW performance counters !](https://github.com/sidkshatriya/rr.soft) Many of you may have already heard of a debugger called [`rr`](https://github.com/rr-debugger/rr.git) – it allows you to record and replay programs on Linux. It is extremely useful for instance to debug issues with garbage collection or other low level issues in natively compiled OCaml programs. Once you capture a bug during the record phase, that bug can be replayed any number of times during replay. One major limitation of `rr` is that it requires access to CPU Hardware Performance counters which is usually not available in cloud VMs or containers. Sometimes HW counters can be unreliable/high latency for some CPUs (e.g. Zen) or it could just be difficult to get them working for your particular configuration. **_Software Counters mode_ `rr` is a modification of the `rr` debugger that lifts this limitation – access to CPU Hardware Performance counters is not required**. This means you can run `rr` in many more configurations. I’ve been able to successfully record/replay the whole OCaml compiler test suite using _Software Counters_ mode `rr` (Except for a single ocaml test called `pr2195` which exhausts the file descriptors). **I’ve also written a blog post** about record/replay debugging generally and _Software Counters mode_ `rr` in particular. Please see [here](https://github.com/sidkshatriya/me/blob/master/008-rr-everywhere.md). ---
**Author:** ![kayceesrk](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/kayceesrk/32/1936_2.png) [@kayceesrk](https://discuss.ocaml.org/u/kayceesrk)\ **Post date:** [March 24, 2025, 3:04pm UTC](https://discuss.ocaml.org/t/ann-a-tool-to-reverse-debug-ocaml-other-binary-runs/16366/2 "2025-03-24T15:04:37Z")
This is very cool. I’ve been a heavy rr user. I appreciate the work to get the software mode working. How slow is the software mode compared to the hw mode in practice? Are there other limitations? ---
**Author:** ![sid](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/sid/32/1476_2.png) [@sid](https://discuss.ocaml.org/u/sid)\ **Post date:** [March 24, 2025, 3:56pm UTC](https://discuss.ocaml.org/t/ann-a-tool-to-reverse-debug-ocaml-other-binary-runs/16366/3 "2025-03-24T15:56:19Z")
Thank you ! _Software counters mode_ is definitely slower due to overheads from dynamic instrumentation. Some of that overhead can be managed by using (optional) static instrumentation via custom C/C++ compiler plugins (which are provided in the repo). But given that you would probably be using this with OCaml compiler compiled binaries rather than C/C++ binaries the dynamic instrumentation overhead is probably unavoidable. I would love it if you were able to try it out sometime and tell me how it went – Is the performance acceptable for your use case ? Is the debugging experience reliable/robust ? As far as limitations – in general, with dynamic instrumentation, things can be more fragile apart from being slower. But when HW counters are not available the _Software Counters mode_ could be better than no option at all. Additionally on some CPUs _Software Counters mode_ might be a good idea to try out also (On Zen CPUs the HW counters can sometime be unreliable). Also currently I don’t not instrument JITed code (but nevertheless in most cases debugging should work fine when JITs are present also). Given that OCaml does not do JIT this should not be a big concern. One of additional limitations of _Software Counters mode_ rr is that it currently needs to be used in a mainstream Linux distribution (fedora / ubuntu / debian unstable) due to a need for good `debuginfod` support. See [Home · sidkshatriya/rr.soft Wiki · GitHub](https://github.com/sidkshatriya/rr.soft/wiki) for detailed information for why this is the case and other useful information ! ---
**Author:** ![lambda\_foo](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/lambda_foo/32/275_2.png) [@lambda\_foo](https://discuss.ocaml.org/u/lambda_foo)\ **Post date:** [March 24, 2025, 9:56pm UTC](https://discuss.ocaml.org/t/ann-a-tool-to-reverse-debug-ocaml-other-binary-runs/16366/4 "2025-03-24T21:56:54Z")
Adding this to my toolkit, awesome work. Do you think ARM64 support will be difficult to get working? When I last looked `rr` needed Linux to correctly identify the underlying CPUs (e.g. Apple Silicon M3Pro) to setup hardware performance counters. At the time only M1/M2 were supported by Linux. ---
**Author:** ![sid](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/sid/32/1476_2.png) [@sid](https://discuss.ocaml.org/u/sid)\ **Post date:** [March 24, 2025, 10:11pm UTC](https://discuss.ocaml.org/t/ann-a-tool-to-reverse-debug-ocaml-other-binary-runs/16366/5 "2025-03-24T22:11:58Z")
aarch64 support exists for _Software Counters mode_ rr ! I am for instance able to run `rr` in Linux VMs on apple silicon macOS machines which I think is very cool (if I say so myself 🙂 !). There are still some rough edges that need to be resolved but it generally works well. I’ve just not made the code public for aarch64 currently – I’m trying to figure out my strategy for that. As a start I’ve released x86-64 version to the world. ---
**Author:** ![sid](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/sid/32/1476_2.png) [@sid](https://discuss.ocaml.org/u/sid)\ **Post date:** [May 5, 2025, 4:25pm UTC](https://discuss.ocaml.org/t/ann-a-tool-to-reverse-debug-ocaml-other-binary-runs/16366/6 "2025-05-05T16:25:50Z")
BTW aarch64 support was released – forgot to mention it on this thread. ---
**Author:** ![sid](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/sid/32/1476_2.png) [@sid](https://discuss.ocaml.org/u/sid)\ **Post date:** [May 8, 2025, 5:11am UTC](https://discuss.ocaml.org/t/ann-a-tool-to-reverse-debug-ocaml-other-binary-runs/16366/7 "2025-05-08T05:11:40Z")
> [@sid](#): > > One of additional limitations of _Software Counters mode_ rr is that it currently needs to be used in a mainstream Linux distribution (fedora / ubuntu / debian unstable) due to a need for good `debuginfod` support. See [Home · sidkshatriya/rr.soft Wiki · GitHub](https://github.com/sidkshatriya/rr.soft/wiki) for detailed information for why this is the case and other useful information ! BTW this limitation was also removed some time ago i.e. `debuginfod` support in your Linux distribution is not needed. A `flake.nix` is provided so you can get up and running very quickly in case you don’t want to perform the build instructions manually. TL;DR _Software Counters_ mode can run on any recent aarch64/x86-64 Linux distro. ---
**Author:** ![sid](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/sid/32/1476_2.png) [@sid](https://discuss.ocaml.org/u/sid)\ **Post date:** [May 23, 2025, 12:52pm UTC](https://discuss.ocaml.org/t/ann-a-tool-to-reverse-debug-ocaml-other-binary-runs/16366/8 "2025-05-23T12:52:09Z")
I’ve written a tutorial on using both `rr` and its _Software Counters_ mode variant. See [me/009-rr-on-aarch64.md at master · sidkshatriya/me · GitHub](https://github.com/sidkshatriya/me/blob/master/009-rr-on-aarch64.md) The tutorial starts from the basics and covers quite a lot of ground. I would love feedback on the blog post. If you’re interested in reverse debugging please check it out ! @kayceesrk Since you’ve used `rr` quite a lot, I would particularly appreciate your feedback, should you have the time 🙂