I made a port of ocaml bytecode to a platform with 15-bit integers.
As polymorphic variants are internally represented with an hash number, i need to rewrite their identifiers using a short form, i.e. `( [A-Z] | [A-H][_'0-9A-Za-z] | I[_'0-9A-Za-g] ) so that their hash numbers will stay in 15 bits.
Writing a ppx rewriter for a single compilation unit is a naive exercise, but I need to perform a global rewriting on all units, passing the state of renamed identifiers, because a rewriting based on module names will fail if some compilation units contain an “open” directive.
How can I manage that?
Note: no need of ppxlib as all my code is tied to the release 4.14 LTS
Because the port doesn’t compile source files to bytecode, but gets bytecode produced by ocamlc and transforms it to platform-specific bytecode or native code.
You could do it the way that TeX deals with such global data-set issues:
(1) emit “aux” files from the processing of each compilation unit
(2) when they exist, load all the “aux” files and use them as the database you desire
(3) and of course, check that for the variants in the current compilation unit, “computed ab initio” output to the aux file is the same as what’s in the aux file (if present).
That is to say, you run your processor … probably three times:
once to generate AUX files but produce incorrect bytecode (b/c some of the AUX files aren’t present)
once to regenerate those AUX files and also produce what-should-be-correct bytecode
and (maybe?) once more, to verify that you’ve reached a fixpoint
I offer this only as a possibly incorrect suggestion. It might not fit with your problem at all.
Avoiding parallel compilation, only one pass will suffice if the growing database of substitutions is saved in an unique file.
But how can I pass the database file name to the ppx rewriter? I think that the preprocessor executable cannot take command line options, nor access to the argv array. So I’m stuck to use a predefined filename, and that’s not the best way to deal with multiple ocamlc invocations.
Update:
It seems that ocamlc -ppx “./myppx -opt1 -opt2…” (with quotes) will pass opt1 & opt2 to myppx. I’ll try it.
And I just read that ppxlib allows to pass global state via cookies: that’s should be a possible solution.
This answer seems to contradict everything else here: if your port doesn’t handle the compilation of source code to bytecode, but uses ocamlc unmodified, then it’s already too late for any PPX because they are executed at the very beginning of ocamlc.
Or if it’s just this answer that’s out of place and the port actually does handle the compilation (because you can choose what PPX to use), then you can also swap out ocamlc itself for a custom one like @hhugo suggested.
There’s no contradiction: my project is similar to OCaPIC (see GitHub - bvaugon/ocapic: OCaml for PIC microcontrollers · GitHub ): the bytecode produced by ocamlc is adapted for the platform and a new bytecode is emitted, raising an error at “transformation time” if any integer is not in the 15-bit range.
The whole code works perfectly, but I’m stuck on polyvars because of their internal representation. I want to let a programmer use meaningful polyvars, so that `MyPolyvar could be translated by the ppx into `Aa and ocamlc hashes it into a 15-bit int.
The question is: how can a ppx rewrite all polyvars globally for all compilation units?
To be clearer, this is what my code does:
calls ocamlc → translates bytecode to platform-specific bytecode → generate a file with bytecode+interpreter, or native-code
If they can choose a PPX, then they could also just choose to use a custom ocamlc. The compilation scheme for polymorphic variants (described in https://caml.inria.fr/pub/papers/garrigue-polymorphic_variants-ml98.pdf) very intentionally avoids such global state and ocamlc already can do it all locally and avoid collisions at runtime. All you’d have to do is reimplement Btype.hash_variant in the compiler and your problem would be solved.
In fact, the compilation scheme doesn’t even require no collisions to occur within a single module, but just in a single polymorphic variant type. Two different polymorphic variant types are allowed to have colliding hashes. As the paper says:
This also means that the probability of having a conflict is only proportional to the number
of tags in a variant type, rather than the total number of tags in a program.
(I’ve seriously wondered if anyone has really ever managed to hit such a collision, outside of the compiler test suite, where the colliding names are completely artificial, not what any programmer would choose.)
Even if you’d have such a PPX and the programmer can activate it for their own code, then that’d do absolutely nothing to all the polymorphic variants that may be used somewhere in the dependencies being used. It wouldn’t be very straightforward to hook a PPX into every dependency, but it is very simple to just swap out the compiler and compile the dependencies with a compiler that solves the issue across the board.
This just sounds like an XY problem to me.
Of course reducing hash_variant bits from 31 to 15 exponentially increases the collision probability, but that’s all within one type. If the polymorphic variant names used by actual code never combine such constructors into a single type, then there’s no need for anything more complicated. Only if that actually happens would it require a different approach.
Yes, recompiling ocamlc with a new hash function will be the definitive solution, and I could also insert all bytecode transformations into the new compiler so that a whole pass will be skipped. That’s will be a next step in my project.