[ANN] prr, a fork of brr for non-browser JavaScript environments

I’m please to announce the first release of the prr library: prr.0.1.1.

Prr is a fork and a stripped-down version of the widely used JavaScript FFI library brr, which is originally built by @dbuenzli.

There are two major differences between brr and prr:

  1. prr uses dune as its build system. This allows easier vendoring through dune’s vendoring support.
  2. prr includes only non-browser specific components of brr, so that it could be safely used in JavaScript environments like those of Node.js and React Native projects, in addition to targeting web. (In fact this is our motivation to fork brr in the first place.)
    • prr’s README gives an overview of what is included and what is not.

No need to say that we are basically freeloading @dbuenzli and brr contributors’ efforts in building such a wonderful FFI library so special thanks to the contributors of brr.

We will be actively maintaining the compatibility between prr and brr and closely follow the developments of brr.

To install, simple run opam install prr, list prr as a dependency of your project, and open Prr.
Code that previously depends on brr should work without modification if no access to browser specific APIs is used.

Comments and suggestions are mostly welcomed.

6 Likes

pff… :wink:

PS: blague pour les francophones

Looks useful, thanks!

The only question that came to mind is, could brr be packaged in a way where prr isn’t necessary? Not that I don’t appreciate your work, but it will minimize efforts to not not have 2 implementations of the same bindings into 2 repos, for the sake of compatibility with browsers.

Honestly I have no idea what prr is trying to achieve. As I see it now it’s just fragmenting the usage of the brr FFI API.

There should be no problem in using brr in node.js, just don’t use the APIs that are not available (or load a node package that implements them) – if there is one, file an issue.

3 Likes

Indeed if using whole program compilation mode of jsoo, brr is fine for nodejs usage. But that drastically increases compilation time (on our codebase, it’s a difference of between ~3sec and 10~15sec on a M1 Pro MBP).

Also there are other two reasons which led to our decision to fork brr:

  1. to seal the modules brr provides into a toplevel module (if you list brr as a project dependency, modules like Fut, Jv, etc. immediately come into your global namespace among Brr)
  2. to adopt dune for easier vendoring

Since those two are preferences on taste (also changes involving breakage in the case of 1.) to brr, we decided to fork it instead of suggesting upstream changes. Later we thought this might also be useful to others so we published it to OPAM.

I suspect that it also works with separate compilation since 0.0.4 in which toplevel initialization bits were sanitized for WebWorker.

If not, issues are welcome to see what we can do.

Usually the solution to that is to contribute an overlay here.

Finally I will just mention that I usually do not care with what happens with the code I publish as long as the copyright headers are kept and the license terms are satisfied (which prr does) – that’s the reason why I publish my code under unrestrictive licenses in the first place. However in this case I do not condone what has been done here, it needlessly fragments the eco-system.

1 Like

Also note that with the next release of jsoo, separate compilation with behave much more like whole program compilation in term of what gets linked it. See https://github.com/ocsigen/js_of_ocaml/pull/1378

4 Likes

That was an entirely different project :wink: returntocorp/pfff: pfff is mainly an OCaml API to write static analysis, dynamic analysis, code visualizations, code navigations, or style-preserving source-to-source transformations such as refactorings on source code. (github.com)

I suspect that it also works with separate compilation since 0.0.4 in which toplevel initialization bits were sanitized for WebWorker.

We were using brr.0.0.3 and after some long hours of debugging our configuration with separate compilation, we figured out that if we remove all browser-only APIs from brr the separation compilation setup finally began to work. And based on that observation we made the decision to fork.

Now looking back, it seems that it was exactly the same day when we prepared to perform a proper fork and brr was about to have a new release. (Our fork point is exactly the previous commit of the 0.0.4 preparation commit, and our first commit for the forking sits just between…)

Admittedly, I should have checked whether the newest upstream has solved our problem first but I was instead blindly executing our decision to go forward with a fork that is up-to-date with the upstream and does not contain any browser-only APIs (plus some repackaging).

I will be testing whether replacing with brr.0.0.4 solves the problem and report back the result.

I didn’t know that option exist and will definitely look into it.

I am sorry for that I have not seen it as a potential problem, and fragmenting the ecosystem around the brr’s style of JS FFI is definitely not our intention. In fact, prr is intended to purely be a stripped down and repackaged version of brr. The only other thing that we may be adding (to the repository, no to the actual codebase) without upstream consent would be tests that ensure the library works well on the platforms that we’d like it to work on.

In this regard, my understanding is that all real development will not happen at all under prr, and we will pull all upstream updates to prr. In fact we has planned to make a script for mechanizing just that.

Right now this policy is not explicitly stated on the README of prr’s repository, but is certainly our intention and implicitly conveyed by the wording of the current README. We are happy to (and probably should) revise the README and make it clearer.

If there is still a concern of unnecessary fragmentation and it is considered harmful for the ecosystem, we have no problem to take the package off opam (if that could be done and is an option). We would like to keep the fork at least internally though because we do benefit materially from this repackaging in our internal projects (especially the change to have a single toplevel module for the avoidance of name clashing).

I’m glad to know that! I will be very eager to try it out once it gets released (maybe even before that).

To add more about our situation: after enabling the separation compilation, the building time went down while the final js file’s size went up, which drastically increased the loading time on the JavaScript’s tooling side. We have mitigated most of them and have achieved a much pleasant working flow compared to before, but we were not able to make ts-jest any faster so we are now performing a separate whole program compilation to run the Jest test suite. It would be nice to be able to consolidate them.

For if sharing the numbers would be helpful to anyone, when using js_of_ocaml.4.1.0, the separate compilation results in a js bundle of about 17.1M and the whole program compilation mode results in about 2.6M. ts-jest loading time for the smaller js file is about 20~30sec but about 80~90sec for the larger one.

3 Likes

What is ts-jest doing exactly ? Maybe you should report an issue on this project ?

Actually I have no idea. I tried disabling tsc type checking the generated js bundle outside of ts-jest, and had good speedup (~20sec to ~1sec for compilation of the test code.) But when I tried to apply the same config to ts-jest, the time save did not transfer.

I would love to investigate more and get this solved, but we do not have enough resource currently to pursuit it. We have already moved the majority of our tests to Tezt and our usual development cycle regarding the jsoo related part does not involves running ts-jest tests anymore.

I think it is indeed some problem with either our way of configuring or an upstream bug in ts-jest. We will look into it again when we get time or this become a priority again.

I finally got some time to try brr.0.0.4 in this project. It turns out that in separation mode, the js bundle still crashes at loading time. With some investigation it turns out it has something to do with WebGLRenderingContext, which nodejs clear does not have.

I could make a minimal reproducible example when I get some time this weekend but my general feeling is that it would be nice to be able to split the FFI only part and the browser API dependent part into different opam packages (even better, one package per group of browser API) as I have the (impolite but honest) feeling that after the webgl initialization is made lazy there might be some other initialization code depending on browser APIs that would prevent our codebase from working. I am willing to volunteer doing this splitting but I am not familiar with the build system used by the current brr’s repository.

1 Like

It’s these enum lookups.

Let’s deal with facts rather than feelings. Now that the problem of toplevel initialisation bits has been highlighted by WebWorkers, attention is paid to that in brr. If there are still problems with that, file issues.

Splitting between different opam packages is unlikely to happen. We are not trimming any host installation needs by doing this so there’s absolutely nothing to be gained from that except more bureaucracy for everyone and a degraded package user experience.

Some splitting along the library dimension is however possible. In particular revising the library structure to make it easier to delineate functionality for a bare ECMAScript environment could happen if a proper issue is filed. No need to do this yourself, I will do it, I’ll have to consider a few things to minimize breakage (but some will have to happen if this is done).

2 Likes

In fact such a sanitization has already be done for the webgl2 context which is not available in every browser, so it’s just a matter of extending this to the webgl1 context.

The “feeling” I mentioned is rather a heuristic for guiding allocation of limited resource (of my side) than an emotional one. I wrote that to support my preference about splitting the library over fixing sanitation as a matter of priority. But I agree with you to deal with as many facts as possible here and make the best choice in the long term.

I see and it’s great that this can be fixed easily. I will be sharing a test bench and file an issue regarding this later (hopefully this weekend).

Sorry what I really meant is suggesting to split brr into findlib libraries but not opam packages (hope these are the correct terminology). I agree that splitting into different opam packages in this case is not materially beneficial and only creates maintenance burdens. Do you think it is possible to do it this way? What I am imagining is that the user can specify (sorry for dune syntax) (libraries brr) for the whole brr offering and specify (libraries brr.core brr.webworker brr.webgl) to selectively use only parts of brr (granularity is of course to be discussed).

By leaving the brr library the same as now should minimize the impact to current library users.

Sure thing to do. I think there are several things discussed in a row here, do you prefer a general issue for discussion or separated actable issues each addressing potential changes to the library?

  • sanitize webgl enum lookups etc. so that brr will load in non-browser environments
  • reorganizing (splitting) the library into sub-packages of FFI only parts and browser API dependent part
  • (maybe not to your liking) possibility to seal modules that brr provides under a single toplevel module
1 Like

You can save your time for that it has already been fixed.

I don’t see the point of micromanaging this. Especially now that @hhugo mentioned that js_of_ocaml is finally reconciling its separate compilation linking model with OCaml’s library archive linking model (i.e. unused module are not linked in separate compilation, unless -linkall is specified). What exactly do you get out of this except more names and build specification cruft ?

I’m now even starting to think whether separating the FFI bits is needed at all. For people who are adept of honest dependencies ((implicit_transitive_deps false)) dependency tracking this will mean breakage, and specifying two packages when they want to use brr. I’m not sure exactly what there is to be gained here, especially since I expect most people to use brr in browser environments.

Precise issues, always.

This is done. Please confirm that it works for you.

This could be done. Though I’m getting colder on the idea. File an issue, we’ll see if it gathers interest, but in the end with js_of_ocaml improving the status quo may be good for everyone (browser and node.js users) aswell.

You can save that one. Ask upstream for proper library linking and namespacing support.

Btw. I just want to mention that one of the reason why this will not be done is that signatures would become less readable.

The only toplevel names you get are Jstr, Jv and Fut and it’s good to have them that way in signatures and documentation.

I am sorry for being unresponsive for a while (and I might be so for another week or so) and thank you for your replies and helps. Just to give a quick update. I will try to find some time to give a proper reply.

I just checked with the new commit and it worked perfectly with our codebase without problem! Thank you very much for the fix.

Well, Jv gave us a good amount of trouble in our code base… And I could also recall that Fut also collided once with a toy project of mine. I personally think a library should not directly expose secondary modules (esp those with generic names) except prefixed with the library name to avoid name clashing (and I think at least people around dune is also promoting this culture.) In OCaml, it’s easy to alias a nested module, but quite a hassle to solve a collision.

If I’m understanding it correctly (from my quite limited knowledge), a build manager agnostic solution to this would need changes to how ocamlc resolve modules. I guess it might be possible for e.g. dune to do some code generation trick and perform the repackaging, but it seems unnatural to me. Do you mind to share how you imagine this could be done?

FYI, js_of_ocaml 5.1 has now been released