OCaml in Jupyter-based Environments

[patricoferris]

As @CraigFe said, this conversation might be useful for other people looking to run OCaml for teaching or data-science in a Jupyter-based environments. Continuing the discussion from Suggestions from the OCaml Survey result where binder was mentioned, this looks like a neat way to lower the barrier to entry for programming in OCaml.

To add to what @mseri asked:

@mseri

Do you know if and how one can use ocaml on colab? In principle there is support for the jupyter protocol…

You can but it is a bit hacky. You first need to generate a blank ocaml.ipynb locally and upload this to colabs, this will enable you to select Runtime > Change Runtime Type > Runtime Selection to OCaml. Then you need to install OCaml, jupyter and add the kernel. This code block did the trick for me:

!add-apt-repository ppa:avsm/ppa && apt-get update && apt-get install opam m4 libgmp-dev
!opam init --disable-sandboxing -y 
!opam install jupyter
!opam exec -- ocaml-jupyter-opam-genspec
!jupyter kernelspec install --user --name ocaml-jupyter "$(opam var share)/jupyter"

It takes quite a while and I’m not really going to recommend it, but it does work.

I also put together a few dockerfiles with the purpose of being used in jupyter environments for teaching. Also with an example of using nbgrader although I think this still needs some work (it’s based on @kayceesrk’s blog post). Are people using other setups for teaching OCaml in a classroom/university setting?

---

As a small aside, I started doing the survey analysis in OCaml, but found two issues:

1. CSV loading with OWL was struggling (I think it struggled with the extra , in questions and also didn’t like multiple headers with the same name – other libraries I think just add the column number to make it unique), I ended up using the csv library and then making the dataframe by hand.
2. The plotting library is good, but lacks the customisability that something like Matplotlib offers. I did start using the very good ocaml-matplotlib bindings but it all became a little too much effort…

[XVilka]

Another project that is probably a better inspiration for OCaml-based workflow - Pluto.jl that is written in Julia itself.

See Pluto presentation (20 min) at Juliacon 2020.

[avsm]

@patricoferris

Are people using other setups for teaching OCaml in a classroom/university setting?

We use Jupyter and nbgrader in Cambridge for the first year Foundations of Programming course: https://www.cl.cam.ac.uk/teaching/1920/FoundsCS/materials.html
(the pdf is generated from the Jupyter export, and the content is written in Markdown and converted to notebook format using a modified mdx).

It’s on my todo list to resurrect https://github.com/andrewray/iocamljs so that we can do this with a pure-client-side experience. We have internal hosted servers so that each student has their own container running their Jupyter/OCaml kernel, but I think it would be more robust to switch entirely client side for everything except exercise grading.

[mseri]

Thanks both of you for the very interesting thread.

Until now, I had used binder and more recently colab, but always using python. Looks like using colab with ocaml is still too cumbersome, but I’d be very interested in seeing also how to do it with binder for example.

The few times I used OCaml, I was relying on hand-crafted scripts in js_of_ocaml to preserve interactivity but hiding the coding, as in the ICFP2019 demo. I was also considering something static, using a new feature of scirep now that @pveber has also added an example using owl.

Having something interactive where we can use OCaml, directly in browser or (maybe even better) leveraging third-party hardware – free colab, I am thinking of you – could be extremely interesting and valuable. A long time ago, when one of the first jupyter versions was released, I had looked into iocaml and thought that it would have been hard to port without a nearly complete rewrite. If that is correct, do you think one should look into doing something new, maybe taking inspiration also from newer ideas like Pluto, instead of just trying to port iocaml to jupyter?

[pveber]

Does this modified version of mdx enables generating pictures in the notebook? As @mseri mentioned I developed a modest hack called scirep to generate html from a markdown file that contains executable blocks. This tool shows the interpreter outputs but can also render graphics in the page. Do you have plans to include a similar feature in mdx? (I’m asking because if this is the case, I’d be happy to lend a hand).

I have to say though, writing ocaml code in a markdown document is not very convenient since you don’t have merlin+syntax highlighting helping you. Ideally we’d need an environment that knows both markdown and ocaml very well…

[avsm]

@pveber

This tool shows the interpreter outputs but can also render graphics in the page. Do you have plans to include a similar feature in mdx? (I’m asking because if this is the case, I’d be happy to lend a hand).

Doesn’t support it, and that sounds awesome. @jonludlam did something involving registering OCaml toplevel printers that can output HTML that renders in a notebook (so you can pipe a type tree = ... through dot to render it graphically). Never upstreamed that work I think. Feel free to create an issue on the mdx repo – I think the utility of it will increase dramatically if it can create notebooks!

@pveber

I have to say though, writing ocaml code in a markdown document is not very convenient since you don’t have merlin+syntax highlighting helping you. Ideally we’d need an environment that knows both markdown and ocaml very well…

Yes indeed, mdx supports this mode too. In Real World OCaml, we have all our examples as separate files that can have dune build run on them, and then mdx supports external references as well. See for example the JSON chapter in RWO: you just add a file= block into the Markdown, and the dune promotion rules will either run the toplevel or include the external ML file and update the Markdown content automatically.

[CraigFe]

@pveber

I have to say though, writing ocaml code in a markdown document is not very convenient since you don’t have merlin+syntax highlighting helping you. Ideally we’d need an environment that knows both markdown and ocaml very well…

I played around a little bit with doing this in Emacs’ org-mode – which is close to working out of the box – and got some OK results w.r.t. syntax highlighting, inline evaluation and integration with Merlin:

There’s no reason this couldn’t be applied to Mdx + Markdown too, with its own major mode / VS Code plugin etc., that would just require more upfront work on the editor side to get a suitable integration.

I’m happy that the “multiple file + expect-test inclusions” approach has worked for Mdx users like Real World OCaml, but personally I find it falls a bit short of the potential of an “integrated” literate programming solution in a single buffer. I know that there are efforts in various places to achieve something like that. In the meantime, we’re fortunate to have existing OCaml platform tools that are versatile enough to support a bit of hackery on the behalf of the buffer. For instance, I’d like to see what is possible by defining a Dune dialect for literate OCaml and using that to support building + formatting these files in a first-class way.

[CraigFe]

@mseri

Until now, I had used binder and more recently colab, but always using python. Looks like using colab with ocaml is still too cumbersome, but I’d be very interested in seeing also how to do it with binder for example.

Here’s an example of the Binder + OCaml workflow – a set of notebooks containing take-home work for Computer Science students, which I managed with nb_grader. The Docker image being used was built with this simple Dockerfile:

FROM ocurrent/opam:ubuntu-20.04-ocaml-4.11

RUN sudo apt-get update && sudo apt-get install -y python3-pip libgmp-dev perl pkg-config libcairo2-dev libzmq3-dev m4
RUN pip3 install jupyter

ENV PATH $PATH:/home/opam/.local/bin
RUN opam user-setup install
RUN opam install -y jupyter
RUN opam exec -- ocaml-jupyter-opam-genspec
RUN jupyter kernelspec install --name ocaml-jupyter "$(opam config var share)/jupyter" --user

which is kept in a repo here after being adapted from a very similar one by @patricoferris.

From the feedback my students gave me, it seems to be a reasonably good workflow that I would use again. The first person to load the notebook has to wait a few minutes while the Binder build cache is populated, but after that it works well.

[jonludlam]

Registering ‘rich’ printers with Jupyter is a standard feature of the wonderful ocaml-jupyter (thanks, @akabe!) - you just register a printer as normal that ends up calling Jupyter_notebook.display. I have a couple of examples here for displaying trees – the one you mentioned – and ppm files. It would be very neat to be able to do something similar for mdx, and probably not too tricky too. Perhaps a good starter project?

[pveber]

@avsm

Feel free to create an issue on the mdx repo – I think the utility of it will increase dramatically if it can create notebooks!

Done!

[pveber]

@CraigFe

I played around a little bit with doing this in Emacs’ org-mode – which is close to working out of the box – and got some OK results w.r.t. syntax highlighting, inline evaluation and integration with Merlin

This is fairly impressive, I’d sure love to have this in emacs! BTW I noticed that markdown-mode offers useful commands to perform language-aware syntax highlighting and code block edition with tuareg (and ocp-indent!).

[Naereen]

I’m very interested in this! Have you made any progress @avsm?