[ANN] A dynamic checker for detecting naked pointers

Are they? Some instances of naked pointers have been reported, and the tool to detect naked pointers has only just been released. Important examples includes the usage made by the ancient and netmulticore (ocamlnet) libraries, which was reported early on, and which has no replacement whatsoever in the no-naked-pointers world.

Concerning safety, it is instructive to look at how they avoid the issue of out-of-heap pointers that become in-heap after the heap grows. Ancient uses its own allocator (mmalloc) while netmulticore writes to a shared memory object. Both ultimately come down to allocating with mmap, instead of using the system allocator like OCaml does. By looking at the code, I think that netmulticore is programmed to avoid the issue (only the main process frees some mapping, after the children exit), whereas for Ancient I think it is improbable if not impossible that it happens thanks to mmalloc, where in the former case the fix would be very simple: just keep address space reserved (i.e. instead of munmap use mprotect with PROT_NONE flag, and maybe also madvise with MADV_FREE or MADV_DONTNEED). Note that one can take an off-the-shelf high-performance allocator that offers this feature (jemalloc with option opt.retain).

The reason, I believe, why the issue is a reality concerning C interoperability in current OCaml is that OCaml gets its memory chunks from the system allocator, which is designed to recycle memory.

Concerning maintainer time, I will just note that the no-naked-pointers mode was far from ready for the removal of the page table at the moment the decision to drop naked pointers appears to have been taken, and it took engineering time away with a few PRs, especially regarding the representation of closures which is still not finished, an aspect which by itself slowed down the multicore development because the solution to circumvent the absence of test for in-heap pointers was brittle. Then, once naked pointers are removed, it will be the turn of the community to deal with the consequences, which beyond engineers include academics and open source volunteers, who are not paid for that job and who (I hope) have better things to do. I make my point in more detail in my draft, but I do not think proper care has been taken to evaluate the consequences and to find alternatives.

This reply surprises me since it is the first time I see such a claim proposed to the scientific community (not the failure case, but the reason to remove naked pointers). It appeared in none of the PRs, issues, discuss threads, caml-list messages, etc., from the past 5 years that I have read in detail in the recent weeks. The claim in the last sentence is incorrect, and I would have liked to have a chance sooner to explain what ancient and netmulticore do for this problem.

On the other hand, there have been many mentions on the caml-list and on this discuss showing interest in naked pointers à la ancient and netmulticore, from OCaml users and from scientists. See for instance here and my own.

With my draft proposal, you could remove the hack “force a GC to remove naked pointers”, and you could use jemalloc as your malloc with option opt.retain to prevent its address space from being reused (if there is any risk).

As an update, what I want to do next with the draft is to see if I can fit any of it within the page limit of the ML workshop and to submit it there.