Jsoo, production use and user facing applications questions

Ecosystem
1

after reading A short history of ReScript (BuckleScript) it seems that people are suggesting jsoo for production use and are using it for user facing applications;

which leads me to believe that I had a very different understanding, so I hope people with experience running it on prod can help and clarify these questions:

  • aren’t the bundles huge? doesn’t it need to include a lot of ocaml code translations (runtime) as a minimum before user code is included?
  • is chunk splitting even possible? is it possible to split a big application in smaller asynchronously loaded bundles?
  • how do common ocaml concepts like include-ing a module and functors translate to jsoo, don’t these create copies and increase the bundle sizes further (meaning that commonly used ocaml libraries also increase the bundle size significantly)?
  • is there good tree shaking/dead code elimination? for example would using a single function of Base result in a bundle including all of Base or just this one function and its dependencies?
  • have you had issues with debugging, source maps and etc.?
  • is it possible to use something jsx-like syntax to generate html similar to ReasonReact?
  • how are promises handled, for example is Lwt code automatically translated to promises?
  • what about json, serialisation and yojson, is it easy and efficient (without increasing the bundle size too much) to serialise/deserialise types through yojson?

these were many questions…

8 Likes
2

Been using JSOO in “production” for ≥ 8 years in various projects.

aren’t the bundles huge? doesn’t it need to include a lot of ocaml code translations (runtime) as a minimum before user code is included?

No, JSOO is really good at minimizing the output

is chunk splitting even possible? is it possible to split a big application in smaller asynchronously loaded bundles?

Yes it is possible, separate compilation and all (it’s more of a build-system than JSOO problem I guess).

how do common ocaml concepts like include-ing a module and functors translate to jsoo, don’t these create copies and increase the bundle sizes further (meaning that commonly used ocaml libraries also increase the bundle size significantly)?

Not sure, but I think one of the nice things of starting off of the bytecode is that all those things are all gone by then.

is there good tree shaking/dead code elimination? for example would using a single function of Base result in a bundle including all of Base or just this one function and its dependencies?

Yes it seems to really use the minimum required

This project uses Tyxml, Base, Fmt, Lwd, Lwt, Ezjsonm, Digestif, Base58, ZArith, (json-)data-encoding, and a bunch of Tezos libraries (incl. the whole AST + parser & printer for the Micheline syntax). And it really uses all those things, they are not just randomly dragged dependencies:

 $ curl https://tqtezos.github.io/TZComet/main-client.js | wc -c
890171

have you had issues with debugging, source maps and etc.?

Issues, yes a ton, browsers are an awful mess to deal with. But it is unrelated to JSOO (options like --pretty, --no-inline etc. are far enough to make JSOO the least of your problems).

is it possible to use something jsx-like syntax to generate html similar to ReasonReact?

Funny, 10 years ago, ocsigen had a “tyxml syntax” long before anyone had heard of JSX. At the time, it seems that the consensus was “This is very cool, but it’s a bad idea, writing OCaml values directly is just so much better.” Now, there is an equivalent ppx extension (but I haven’t tried).

how are promises handled, for example is Lwt code automatically translated to promises?

Not sure what you mean. Lwt.t values “are” promises, all the low-level js-of-ocaml APIs just use Lwt.

what about json, serialisation and yojson, is it easy and efficient (without increasing the bundle size too much) to serialise/deserialise types through yojson?

You can use the “native” JSON module from JS, or any ocaml library that does it. The project above is an example of using data-encoding with Ezjsonm (libraries “imposed” by Tezos stuff).

13 Likes
3

thanks @smondet this sounds reassuring that it’s possible to use it in prod

I guess I don’t quite understand how the build of a project would work outside of the standard webpack flow. To be quite honest it’s not clear to me how to output of jsoo can be used - can it be passed to something like webpack? And if not - are there ocaml equivalents of standard commonly used webpack plugins and loaders:

  • dynamic imports - instead of import x from "x" - const x = await import("x") that asynchronously loads the module “x”, which the build system translates to: “I’ll put ‘x’ and its dependencies in a separate file, that will not be loaded initially and instead will be loaded only when requested” (the simplest use case being to create 1 starting point for an application with multiple pages and put each page in a separate bundle, or to put rarely used dependencies in separate js bundles and etc.)
  • handling file imports - when using webpack (or other JS bundlers like parcel) you can just “import” a resource like an image and the loaders automatically generate a hashed url, that stays the same in future builds to preserve browser caching (or changes if the file contents change), allowing you to more easily plug it in other build pipelines (like uploading these resources to remote hosts)
  • css modules - instead of importing whole css files you use small css files “per component” (for example a button), class names are scoped locally to the component and common problems with larger web applications are prevented; (I don’t get the appeal of styling with code if there is a dedicated language, more suitable to the task)
  • internationalisation - how is that handled with the jsoo ecosystem?

I guess - what is the build flow of a jsoo application?
or: what is the equivalent to a webpack build flow, that takes not just code, but also images, css, translations and outputs a bundle?

to clarify - if there are equivalents to these common flows in ocaml - I would consider it a huge advantage, because ocaml libraries tend to be faster and of higher quality than js equivalents, but I don’t have experience with jsoo - I know webpack (hence the appeal of bucklescript). I find it difficult to map my existing knowledge to jsoo concepts, so any resources and examples would be appreciated

fwiw I like being able to copy paste html snippets and have matching code and generated html)

I mean JS Promises - would code be translated to use “JS Promises” or it would use Lwt promises, implemented in JS - would the browser event loop be utilized, or would it use the lwt loop translated to JS

thanks again for all of the information and sorry if the questions don’t make any sense, but this is an entirely unfamiliar ecosystem that looks very alien to me

2 Likes
4

Lwt uses its own promises. How it generally works is that a pair comprising a promise of type 'a Lwt.t and a corresponding resolver of type 'a Lwt.u is constructed by Lwt.task, and a callback function is registered for the dom event in question which resolves the promise by applying Lwt.wakeup or Lwt.wakeup_later to the resolver. Code can then “wait” on the promise until it is resolved by the callback. It is basically the same as JS promises except that Lwt promises are truly monadic. If you want to see more look at the source code: there is a simple to understand sleep function in lwt_js.ml which shows you the basics (see lwt_js.ml at github repository)

2 Likes
5

I’m not sure there is a consistent answer to your webpack questions, but it’s more of a build-system question. The jsoo workflow is often going to be “build a js file, and use it”, and that’s about it. You can expose Js modules using jsoo, and you can do dynamic imports, but it’ll be by hand. It would be nice if dune supported more of these use-cases.

Tyxml provides a ppx which supports

  1. True HTML literals, available in both OCaml and Reason syntax. It uses a spec compliant HTML parser, so you can directly copy paste pieces of HTML (as long as they are valid).
  2. JSX Literals, available in the Reason syntax. It works mostly the same way as reason-react (but with tyxml).

Lwt uses its own implementation of promises, mostly because it came first and the semantic differs. There are bindings for FFI.

5 Likes
6

I am really curious about this. What css organization approaches have you used in each project? Were there any libraries that you needed that you had to invent from scratch, if so, which libraries? How many routes / how many api endpoints used in each project?

7

summarizing findings:

  • js_of_ocaml deals with just that - generating js of ocaml; javascript imports and interop is possible through creating bindings to js code
  • you can still pass the output of jsoo to the standard javascript build pipelines (including webpack), for example: Create bindings for JS library · Issue #718 · ocsigen/js_of_ocaml · GitHub
  • I couldn’t find any more complete and more complex examples like using css modules or internationalization, that require “coordination” between both the css files and the js output (I’ll assume there’s nothing existing)

if you want reactive apps:

(the above was disappointing since in my experience building a vdom reactive library takes a lot more effort and energy than people realize. It is not enough to build html diffing logic - there’s all kinds of weird browsers, with quirks, accessibility, touch devices, event handling differences that need to be standardized. These are not fun problems to solve and are ones that reactjs has solved, but less widely used libraries have not)

at first glance, as someone without too much ocaml experience - building a modern reactive browser application with the existing jsoo ecosystem is not that straightforward. especially compared to bucklescript, which was a smoother experience to integrate (at least in terms of tutorials, examples and etc. - these seem to be lacking in jsoo land, so simple things take a good chunk of time to experiment), but also one that is no longer available

1 Like
8

For me the forte of js_of_ocaml is not to build impressive web-native applications in the browser, but rather to easily and quickly convert OCaml code to deploy and run over the web. I can take a command-line program and put up a “web demo” version of it online in one day. Some people do this “in production” and the results can be very effective.
I think that a common practice for lazy people like me is to keep the “business logic” in OCaml, but use Javascript for the “UI” part of the code. Typically the interface boundary between the two can be kept reasonably small, and dealing with whatever best-practices du jour is done purely on the Js side.

Some people try much harder to write pure-OCaml applications to run in the browser, and then they have to cope with all the questions you mention at the OCaml level. I would expect ocaml-vdom / bonsai to be in this league. My understanding is that this part of the field is still in its infancy, so I would not expect the ecosystem to be as mature as what Javascript offers, or compared to where Bucklescript/Rescript wants to go.

(Another thing that js_of_ocaml lets you do is to release OCaml libraries as npm packages. Esy intentionally makes it very easy to write projects with a hybrid of OCaml and Javascript components (not necessarily to run in a browser), and I find this pretty neat.)

9 Likes
9

wow… trip down memory-lane :slight_smile:
(partial list)

TZComet:

  • uses Bootstrap 4, trying to also import the Javascript from it
    • which is fine a long as it is used in “closed loop”, once you want to hack something with it you have to deal with jQuery and its weird event system which is annoying. I know there are jquery JSOO bindings but I just wanted to stay as far away as possible from that thing.
    • it’s all accessed through a Bootstrap sub-module that could be made into a library some day (I have seen other such libraries around but not using Lwd)
  • The first draft was made with the familiar Tyxml_js + react/reactiveData but then I switched everything the newer Lwd by @let-def
  • Interesting to write from scratch the HTML-fragment parsing/generation/updating. I guess there could be a library making that easier (but it would have to be so customizable that I don’t know how the API would have to look like).
  • it’s client-only, but talks to the API exposed by public Tezos nodes: it uses maybe 6 or 7 endpoints (?)

ni3.dance:

  • for a personal / music side-project (private repo, sorry)
  • CSS: voluntarily “old-school cyberpunk” so it loads some bootstrap theme as a base but then is mostly all custom incl. the “loading” animations (for images, youtube videos, …) made by just drawing on the canvas
  • the build-system also generates a no-JS version of the website
  • Uses the Tyxml_js + react + reactiveData combo

github.com/hammerlab/ketrew:

  • Used Bootstrap 3 but the CSS part only; all the interactions were written in OCaml.
  • Also Tyxml_js + react + reactiveData combo (first time I used it I think, took inspiration from Thomas’ blog-post on CueKeeper).
  • It has an actual server side, with an API that is also used by the command-line / terminal interface.

Genspio’s web-demo:

  • also a bootstrap theme’s CSS
  • the interesting part is the OCaml REPL running in a webworker and the communication with the main thread (that could also be turned into a higher-level library — maybe somebody already has).
  • binding to the codemirror text editor was hacky but relatively easy (but the semantics of HTML’s textareas are just crazy).

Intranet-WebUI at NYU-Biology:

  • first big webapp I worked on in OCaml
  • using all the power of Ocsigen around 2011/2013 (camlp4 multi-tier, services, scopes, “server-functions”)
  • some historical pain points I remember:
    • the backend libraries were all using JaneSt Core (before core-kernel or base existed) and at the time JSOO code had to be really pure OCaml
    • the build system was a terrifying mix of make and ocamlbuild
  • it had a completely custom CSS

Should mention a bit smartpy.io/ide that I presented this summer which is a mix of JSOO / custom JS / FB-React on the frontend and the project also uses JSOO to to make a node.js CLI application (easier distribution).

10 Likes
10

Are there good resources for how to do this? I’ve seen the separate compilation support, but my understanding is that it requires jsoo_link to produce a single JS file. Is it also possible to load modules asynchronously?

11

I have been using jsoo for some time now. It is remarkably stable. But I have never been able to produce a js-file with less than 1.7 MByte. How did you achieve a jsoo generated javascript file with less than 1 MByte?

12

I don’t know the details of what you are building, but just for reference I did a quick experiment: an executable that just prints “Hello, World!” and exits is compiled to a 13K JS file.

Cheers,
Nicolas

13

For “hello world” I can get the same. However as soon as I use libraries and build a web application the size increases significantly. Even if not much of the functionality of the libraries is used. Therefore I have the suspicion that unused library code is not removed.

Even if I add functionality, the codesize does not increase a lot. Therefore the suspicion that the library code (whether used or unused) is the size driver.

In my projects I use module and module functors a lot. I don’t know, if this might cause code duplication.

14

I haven’t touched JSOO in a while but I do remember having had some reductions with processing its output in rollup (with the node-resolve and terser plugins). I think Rollup performed some further tree shaking eliminations, but my test wasn’t very elaborate so take it with a grain of salt. :wink:

If you’re not familiar with Rollup, you could do this quick test to see how it behaves, if for example your JSOO output bundle is called bundle.js:

// .rollup.config.js
import resolve    from 'rollup-plugin-node-resolve';
import { terser } from 'rollup-plugin-terser';
export default {
  input: 'bundle.js',
  output: {
    file: 'bundle.min.js',
    format: 'umd',
    sourcemap: true
  },
  plugins: [ resolve(), terser() ]
};

$ npm install rollup rollup-plugin-node-resolve rollup-plugin-terser
$ rollup -c .rollup.config.js
15

How are you building your JS?

I get the minified/clean-up output with dune build --profile release and

(executable
 (name main)
 (modes js)
 (js_of_ocaml
  (flags --no-inline))
 (preprocess
  (pps js_of_ocaml-ppx))
 (libraries
  zarith_stubs_js
  fmt
  base
  js_of_ocaml-lwt
  re.posix
  tezos-contract-metadata
  digestif.ocaml
  ;; tezos-crypto → does not work in JS
  base58
  lwd
  tyxml-lwd))

(I actually don’t remember why I added --no-inline :slight_smile: )

1 Like
16

That is practically the same dune file which I use. As libraries I use js_of_ocaml, the ocaml standard library and an own library which handles a virtual dom on top of js_of_ocaml. In this setting a simple counter application (2 buttons, one for increment and one for decrement) makes my javascript file more than 2 MByte (clearly with --profile release).

17

What is the size of the javascript file without the flag ? If you grow to more than >20Mb maybe the problem is elsewhere:

  • using a ppx dependency in the libraries dune section instead of the preprocess section
  • having a dependency to a library which cannot be optimized well (compilerLib by example make the javascript code grow to 1Mb due to toplevel references)
18

Without the flag --profile release I get 14MByte.

19

Can you share your dune file ? or a minimal example which produce overweight output ?

20

Certainly:

(executable
    (name counter)
    (modules counter)
    (libraries alba.fmlib alba.fmlib_js)
    (modes byte js))

The libraries alba.fmlib and alba.fmllib_js provide the functionality for a virtual dom (i.e. diffing and dom patching).

And the source code:

    open Fmlib

    module Browser = Fmlib_js.Browser
    module Appl = Web_application.Make (Browser) 
    module Vdom = Appl.Dom
    module Attribute = Appl.Attribute

    type message =
        | Increment
        | Decrement

    let model: int = 0

    let view (model: int): message Vdom.t =
        let open Vdom in
        let open Attribute in
        div
            []
            [ button [onClick Decrement] [text "-"]
            ; div [] [text (string_of_int model)]
            ; button [onClick Increment] [text "+"]
            ]

    let update (message: message) (model: int): int =
        match message with
        | Decrement -> model - 1
        | Increment -> model + 1

    let _ =
        let module Program = Browser.Make (Appl) in
        Program.sandbox model view update
1 Like