Gynormous memory consumption to compile output of Menhir

Ecosystem
1

I’m hacking on the official OCaml grammar, in a manner that increases the size of the grammar significantly: from 3MB of generated code to 10MB, and I’m seeing the memory footprint of the “ocamlc” bytecode compiler process get to >4GB. For the optimizing compiler, sometimes on my 8GB machine it blows memory completely.

Is there something I can do to decrease memory-consumption? [I’m guessing probably not, but I figure I had to ask.]

In case it’s relevant, the “hacks” are to take things like parsing identifiers (viz “LIDENT”) and replace them with a disjunction of “LIDENT | ANTIQUOTATION”. I’ve tried using a separate nonterminal for these, but in too many cases, it’s not possible (that is to say, the nonterminal needs to be inlined, for the grammar automaton to be conflict-free).

Anybody got any thoughts/advice?

2

If you can run the compilation command by hand, you can use the -dprofile flag to find out which part of the compiler consumes the most memory.
It won’t help much by itself, but at least it might tell you whether the compiler is behaving normally or if one part in particular is blowing up.
Here is what I get for the bytecode compilation of the current parser.ml file (columns are time, allocations, top heap size, absolute top heap size):

7.269s 1.30GB  120MB  123MB parsing/parser.ml
  0.930s 0.27GB 26.1MB 29.3MB parsing
    0.930s 0.27GB 26.1MB 29.3MB parser
  4.065s 0.62GB 75.9MB  105MB typing
  1.783s 0.28GB 5.28MB  110MB transl
  0.490s 0.11GB 13.4MB  123MB generate
  0.001s ------ ------ ------ other
0.027s ------ 3.12MB ------ other

Given these numbers, it seems a bit odd that your example requires more than 4GB.
Even the native compilation only allocates about 2.3GB in total, with a reported top heap size of 210MB; if you simply generate an input file that is roughly 4 times as big, I wouldn’t expect that to blow up your RAM.

3

Here’s the output of the compile command (ocamlc.opt) with -dprofile

96.205s  154GB 4.81GB 4.81GB parser.ml
  02.305s 4.77GB 0.69GB 0.70GB parsing
    02.305s 4.77GB 0.69GB 0.70GB parser
  19.644s 26.4GB 2.20GB 2.90GB typing
  09.283s 4.78GB 0.21GB 3.12GB transl
  64.971s  118GB 1.69GB 4.81GB generate
  00.002s ------ ------ ------ other
00.006s ------ ------ ------ other
```
The original parser.ml was 13M, my modified parser.ml is 33M in size.
The original parser.cmo was 4M, my modified parser.cmo is 11M in size.

As a sanity-check, I compiled the original parser.ml with `-dprofile`:
```
24.196s 44.6GB 1.75GB 1.75GB parser.ml
  00.915s 1.86GB 0.27GB 0.27GB parsing
    00.915s 1.86GB 0.27GB 0.27GB parser
  06.513s 10.4GB 0.85GB 1.13GB typing
  01.205s 1.90GB 0.03GB 1.17GB transl
  15.563s 30.3GB 0.58GB 1.75GB generate
00.016s ------ ------ ------ other
```

That's quite a bit more memory than your run took.  I'm on an AMD Ryzen, Ubuntu 22.04.  And OCaml 5.0.0.

I wonder what's going on ....
4

Did you try to pass the --table option to Menhir ?

2 Likes
5

Damn, I did not know about that. It worked, compiles slicker than snot. Thank you!

1 Like
6

Menhir without --table produces a large number of mutually recursive toplevel functions, and the OCaml compiler currently has a lot of difficulty compiling this kind of code. (That is, its complexity seems nonlinear.) I am told that this may be due to the register allocation algorithm, when it is applied to the module’s initialization code. It would certainly be nice if this was fixed!..

7

Can I just say: the “inline (parameterized) nonterminals” thing is just great. I mean, just great! It made my little project oh-so-much-simpler.