Memory keeps growing with `Buffer` and js_of_ocaml

Learning
1

with this piece of code:

let () =
  let ob = Buffer.create 2096 in
  let oc = open_out "test-file" in
  let seq =
    Seq.unfold
      (fun b ->
        if b < 100 * 100 * 100 * 100 then Some (b, b + 1) else None)
      0
  in
  let () =
    Seq.iter
      (fun _v ->
        Buffer.add_string ob ("output");
        Buffer.output_buffer oc ob;
        Buffer.clear ob)
      seq
  in
  close_out oc

if I compile it directly, memory output never grows over some MBs. Compiled with js_of_ocaml on the other hand let’s the memory consumption grow unbounded.
I’m not sure if this a bug in js_of_ocaml or if I need to do something different?

2

I’m not familiar with some of the APIs used but one thing which jumps to mind is that while you do Buffer.clear ob, I don’t see an equivalent “flush” for oc. Could it be accumulating in memory in the JSOO version until close_out oc? In other words, are you sure that the buffer is what’s eating up the memory, and not the output channel?

3

Thanks, adding flush oc fixed it.
Not sure why that isn’t needed for the binary version though?

4

I guess now I’ll also need to be smart about when to flush, because adding flush in the iteration makes the code much much slower…

5

I don’t see anything obvious in https://github.com/ocsigen/js_of_ocaml/tree/master/runtime to explain it. Maybe it accumulates on the JS engine’s side and not in JSOO per se. I didn’t even know we could write files in JavaScript…

6

In fact, I think the behavior is clear from the jsoo runtime @VPhantom linked to:

See the last conditional test: this function only flushes the internal channel buffer on newlines. If you print large strings without any newline, it will keep growing the internal buffer without limit.

This is different from the behavior of the upstream runtime (in C), which flushes the buffer as soon as it doesn’t have enough space for the incoming input.

2 Likes