@stedolan was kind enough to provide the answer to this:
the main tool for configuring space use is the space overhead parameter (OCAMLRUNPARAM’s
export OCAMLRUNPARAM="o=20" before running your program
Unlike java, you don’t specify a maximum heap size but instead a relative overhead:
o=80 (the default) means that the heap is allowed to grow to the amount of live data in your program + 80%
Interestingly, @Drup points out that OCAMLRUNPARAM` affects any OCaml program, and since the compiler itself is an OCaml program, it’s affected!
@dww has also opened up a PR for the CircleCI docs for anyone else who runs across this https://github.com/circleci/circleci-docs/pull/1499/files.
EDIT: PR has been merged, the new OCaml/Reason note for CircleCI is live https://circleci.com/docs/1.0/oom/#out-of-memory-errors-in-ocamlreason-builds