I’m trying to diagnose a memory leak currently, where a server which is accumulating a log from a client has much larger memory usage than the client even though they both have the same persistant state.
The server behavior appears approximately as follows:
type entry = {id : int; payload : bytes} [@@deriving bin_io]
type message = {entries : entry list} [@@deriving bin_io]
type t = {entries : entry list}
(* This is an implementation for an `async_rpc`*)
let receive t (msg : message) =
t.entries <- (hd msg.entries) :: t.entries
The server acts like this irl because when it receives a list of entries, it will discard any duplicates.
If I restrict the updates such that only unique elements get sent, the exceptionally high memory usage disappears. Inspecting the size of the lists, they can be up to 3000 elements to send ~30 unique elements.
The current hypothesis is that the memory leak is due to the sent list not getting garbage collected, due to the elements from it still being in the persisted log. Is there any validity to this hypothesis? If it was an array I can definitely see this being plausible, however I can’t see that it would be with lists.
I do have a solution for the problem of avoiding duplicate transmission in the replication, however the GC issue seems like it will always be a major potential problem.