Js_of_ocaml and JavaScript modules

Ecosystem
1

Hi all, I’m writing to ask for advice re how to bind a library that uses JavaScript modules.

Full context:

  • I’m working on writing JSOO bindings for CodeMirror 6, which is distributed using EcmaScript modules. In the JS world, you’d use a “bundler” to link all the modules into a single file.
  • I’m using Daniel Bünzli’s brr library.
  • My main concerns are twofold: (1) I’d like users to only have to pay for the modules they use, in other words, if you don’t use the @codemirror/closebrackets module, it shouldn’t appear in the JSOO output. (2) Ease of use – I’d like the library to just work for my users.

What’s the nicest way to do this? Does anyone have examples of doing the same for other JavaScript module-based libraries?

3 Likes
2

CodeMirror is certainly worth binding to as it handles a lot of browser mess – in fact if I had time for hacks I’d use it for brr's OCaml console.

I’m not fully aware of all the JavaScript module bundling/packaging opportunities and mecanisms but there’s a lack of conventions in the js_of_ocaml world to enable good interop with the JavaScript ecosystemS. Had we good conventions and a bit of work in opam we could certainly get a nice depext system like we have for C. In any case with js_of_ocaml your bundler (that rollup, webpack, whatever thing) is js_of_ocaml link.

I never did that so that won’t be precise and there may be better ways. But basically what I would do in your case is bundle CodeMirror in the project (automate that step) then map JavaScript modules on OCaml libraries.

In each library directory have the corresponding JavaScript module .js file and the JavaScript glue code to include it (if needed, not exactly up-to-date with how JavaScript modules work), and have these Javascript files be linked whenever the library is.

For the last step I can’t find any documentation about it in js_of_ocaml :–( but this happens via ocamlfind META files. In each library you should add a line:

linkopts(javascript) = "+$(LIBRARY_DIR)/module_glue.js +$(LIBRARY_DIR)/module.js`

where $(LIBRARY_DIR) is the relative directory in which the library resides w.r.t. to opam var lib.

I think that with this both concerns 1) and 2) would be solved.

3 Likes
3

I wanted to follow up on this. I didn’t make much progress but want to document where I got stuck.

There are a few choices to make. First, to use dune or not. In @dbuenzli’s advice, he doesn’t use dune, but plain js_of_ocaml / META files. I’ve been using dune, like it a lot, and am not so familiar with META files / findlib, so I prefer to use dune if possible (note that where I got stuck applies to both the dune approach and the META approach). Let’s explore what this looks like using dune.

Here’s a simple dune file I’ll try using for the view:

If we try building with (js_of_ocaml (javascript_files codemirror-view.js)), we get:

 js_of_ocaml example/index.bc.runtime.js (exit 1)
(cd _build/default/example && /Users/joel/.opam/4.12.0/bin/js_of_ocaml build-runtime --pretty --source-map-inline -o index.bc.runtime.js ../view/codemirror-view.js)
/Users/joel/.opam/4.12.0/bin/js_of_ocaml: Error: cannot parse file "/Users/joel/code/ocaml/codemirror/_build/default/view/codemirror-view.js" (orig:"/Users/joel/code/ocaml/codemirror/_build/default/view/codemirror-view.js" from l:1, c:7)

codemirror-view.js is the file directly from the CodeMirror source on NPM. I guess it’s understandable that it doesn’t quite work because it uses JS modules. Here’s the first line:

import { MapMode, Text as Text$1, Facet, ChangeSet, Transaction, EditorSelection, CharCategory, EditorState, Prec, StateEffect, combineConfig } from '@codemirror/state';

Fine, let’s use Babel to transform this to ES5 (npx babel codemirror-view.js --out-file codemirror-view-babel.js --presets @babel/preset-env). We also change the js_of_ocaml clause to (js_of_ocaml (javascript_files codemirror-view-babel.js)):

 js_of_ocaml example/index.bc.runtime.js (exit 1)
(cd _build/default/example && /Users/joel/.opam/4.12.0/bin/js_of_ocaml build-runtime --pretty --source-map-inline -o index.bc.runtime.js ../view/codemirror-view-babel.js)
/Users/joel/.opam/4.12.0/bin/js_of_ocaml: Error: cannot parse file "/Users/joel/code/ocaml/codemirror/_build/default/view/codemirror-view-babel.js" (orig:"/Users/joel/code/ocaml/codemirror/_build/default/view/codemirror-view-babel.js" from l:236, c:12)

This time, the js_of_ocaml parser trips on a getter:

    dom.focus(preventScrollSupported == null ? {
        get preventScroll() {
            preventScrollSupported = { preventScroll: true };
            return true;
        }
    } : undefined);

I was a bit surprised to see this in the Babel output, but according to MDN this syntax is in ES2015, so I guess it’s fine. I couldn’t find any Babel preset that doesn’t output a getter. So I guess we’re stuck here.

Aside: Someone else hit the same problem with js_of_ocaml. It also can’t parse trailing commas. The advice on GitHub is “A workaround is to not pass that file to jsoo and bundle it later”. So the js_of_ocaml parser is a bit outdated. I’m curious, why does it parse JS in the first place?

Trying to take @hhugo’s advice from GitHub, I commented out the js_of_ocaml clause (as in the current checked-in dune file). This time the build works (great!), but the CodeMirror sources aren’t linked at all. This requires the user to figure out how to link / bundle them by hand, which doesn’t exactly “just work”.

So it seems like the crux of the problem is the js_of_ocaml parser’s incompatibility with JS modules. Has anyone figured out a way to work around this?

1 Like
4

Note: I noticed that the getter appeared in only one place (everywhere else in codemirror-view-babel.js we see get: function get() { ... }). This seems to be a bug in one of Babel’s plugins? Anyway, since it’s only in one place I decided it’s acceptable to edit it out by hand. This leaves us with the now-checked-in codemirror-view-babel.js. js_of_ocaml can actually parse this one and dune build succeeds.

This is great, however we still see some module stuff in codemirror-view-babel.js: (I guess these are CommonJS modules?)

Object.defineProperty(exports, "__esModule", {
  value: true
});
exports.drawSelection = drawSelection;
exports.highlightActiveLine = highlightActiveLine;
...

var _state = require("@codemirror/state");
var _styleMod = require("style-mod");

This takes us back to the original problem: what’s a sane way to make js_of_ocaml (slash OCaml modules) interact with JS (CommonJS?) modules? The snippet I posted above is still going to require a JavaScript bundler (to define exports and handle the requires). But I don’t think I want to bundle codemirror-view.js, codemirror-text.js, etc all separately (one for each OCaml module) because we’re going to be including the same dependencies N times. I’m not sure there’s any way to do this how @dbuenzli suggested, with separate OCaml libraries.

This is a bit unfortunate because it means I should use a JS bundler to bundle all of the CodeMirror JS together before using that as the input to js_of_ocaml. I don’t expect that js_of_ocaml’s dead code elimination is smart enough to remove any of it. But at least it should work, I think.

1 Like
5

Update (6 months later). I gave up on this approach and decided to create bindings to Microsoft’s Monaco editor instead. This turned out to be quite easy because it comes in bundled form (rather than as JavaScript modules).

6

<offtopic>
Did you decide to use Monaco strictly because it was easier to use from OCaml/jsoo, or were there other considerations that motivated the switch? If the latter, I’d love to learn what you’d learned in since March. :slight_smile:
</offtopic>

7

Honestly, it was almost entirely because Monaco is easy to use from jsoo. There’s not a lot of high-quality discussion comparing the two (the author of CodeMirror has a few comments comparing them on HN, but from three years ago). I don’t have a huge amount of experience, but some tradeoffs as I understand them:

  • Monaco is more rigid, Codemirror is more malleable. If you write a Monaco-based editor, it’s (by default) going to look like VS Code. Codemirror editors aren’t as distinctive / recognizable. You can really make it your own.
  • Monaco feels more like a complete editor you can customize; Codmirror, a set of components to assemble.
  • Monaco comes with a lot of nice stuff (or at least the pieces you need to add these): syntax highlighting, autocomplete, suggestions, go to definition / symbol, etc. Codemirror has support for a lot of this as well.
  • Both are written in typescript.
  • Both have fairly nice API docs. Compare Monaco vs CodeMirror.
  • CodeMirror supposedly works a little better on mobile due to using contenteditable.
  • CodeMirror bundles everything together. Monaco uses an AMD loader / require.js to asynchronously load the bulk of its code.
  • Monaco uses web workers to make things smoother: “Language services create web workers to compute heavy stuff outside of the UI thread”.
  • Monaco (VS Code) is developed by Microsoft and very popular. I have no concerns about its development future. Codemirror has an impressive list of sponsors. I’d bet it has less resources devoted to it but still doing pretty well.

Overall it feels quite close to me. If they were equally easy to work with I’d probably start with CodeMirror.

2 Likes
8

Another web-editor that could be used is the Ace-editor (see example of its use here: https://source-academy.github.io/playground)

I’ve written some basic bindings to it using Brr here: Ace editor Brr bindings ($2194985) · Snippets · Snippets · GitLab

2 Likes