Tooling history for a rank beginner

Learning
1

Hello OCAML people!

I am wondering, is there a resource describing the history of Ocaml tooling, or would somebody mind describing that history?

My background is mostly self taught Python and data engineering, but I am really enjoying playing around with OCAML. I am trying to both learn the language and also learn some fundamentals of functional programming. I have played around with some Ruby, elisp, Go, and Zig, but OCAML has kind of clicked for me in a great way, though I must admit I am perhaps not super productive with it. I generally feel like I don’t have a great understanding of typing, async, compilers, preprocessors, algorithms, or other functional concepts.

I am definitely in tutorial hell, and many of the books, videos, and blogposts which I’ve encountered have a lot of different tooling chains, and I’m doing my best to learn the ecosystem. I apologize for an incredibly broad and open ended question, but I’m interested to ask if anybody can help me summarize some of the tooling, or check my understanding here.

Build systems and packages - OPAM, Dune, lower level compiler libraries - My understanding is that OPAM is the de facto package manager and Dune is a project management dependency and build tool. I don’t really understand how package management worked before these tools existed. It seems like pretty much everybody is using Dune.

Database - PG’Ocaml seems like a lower level library than I’ve used before. Caqti seems closer to an ORM, or tooling which I’ve used in Python, particularly when used in tandem with ppx-wrapper. I don’t think Caqti is considered a proper ORM, but I’m not sure then what exactly it is, or what would qualify something as a fully fledged ORM. Generally speaking, I’ve had a very hard time interacting with databases, but I continue chugging along. I don’t mind using raw SQL at all. What are the advantages of lower level database systems?

Utop - runtime environment and interpreter. I don’t really understand how this works “behind the scenes". Is it compiling commands as it runs?

Merlin, ocaml-lsp-server, tree-sitter-ocaml - I generally understand LSP architecture and how treesitter functions. I don’t understand the history of these toolings and what stack people are using in their editors.

Web Frameworks - I have primarily used Dream. I know that Dream is based on Opium, but I do not know what Dream offers that Opium does not. I am pretty amazed by what I’ve seen from the Ocsigen ecosystem, although I found Ocsigen-start difficult to set up. Coming from Python, I naturally have a deep rooted fear of JS frameworks and tooling, and I love the architecture of htmx, and generally prefer server side rendering for its simplicity.

Async - It seems like most projects are still using lwt for async, but many are interested in or switching to eio. I do not really understand what the distinction is.

PPX - I don’t have much experience with metaprogramming other than a rudimentary understanding of Lisp macros. It took me a minute to understand metaprogramming in terms of preprocessors, but I think I understand at a basic level how these work, and I’m amazed at what people have developed particularly as it relates to generating JavaScript. Melange and js_of_ocaml are very interesting, but I don’t understand the advantages and disadvantages of each library or when they are used. I don’t know what other languages use ppx-type systems, and I don’t understand what is unique about OCAML preprocessing that has allowed such an amazing ecosystem to thrive here.

Match statements - I understand them as a sort of high powered if - elif syntax, but I have seen allusions to suggest they are much more powerful than this, and I do not understand how.

Sorry for dumping a wall of text, and for not really asking any clear questions. Please let me know if my understanding is off, or if you have suggestions for ways that I can further explore topics. I am continuing to try and build out little apps, and understand the lambda calculus. Thank you!

2 Likes
2

You’ve asked about quite a lot but I thought I’d provide my 2 cents on some topics.

Lwt and Async both expect you to write code in monadic style. As a lot of async programming is about non-blocking I/O, you effectively wish to defer a computation until its data becomes available. If you don’t have a way of doing that in the language runtime, you can achieve it by functionalisation. To this end, monadic style is a useful design pattern as it amounts to a functionalisation of the program at its sequencing points (much like Continuation Passing Style (CPS), of which monadic style is a generalisation). For example, a computation wishing to “consume” some 'a Lwt.t value will be expected to provide a continuation function using the monadic bind function/operator:

val bind : 'a Lwt.t -> ('a -> 'b Lwt.t) -> 'b Lwt.t

So, the “rest of the computation” lives inside the runtime closure represented by any instantiation of ('a -> 'b Lwt.t). Hence, with a good amount of runtime shenanigans, you can defer and then invoke these continuations at later points (Lwt.t maintains a scheduler invoked by its main loop). That’s a rough description (I’m not an expert), the key idea is that you’re forced to write code in monadic style, which can be quite turgid (although, a lot nicer with let-operators!) and distinct from the usual, direct style, code (this juxtaposition of styles is sometimes referred to as the “function colouring” problem).

The utility of delimited control operators, as provided by OCaml 5’s effects and handlers is that you can basically reify continuations dynamically at runtime (by slicing up the call stack). Instead of writing in a style that manually sequentialises a computation through continuations, you write your program in direct style and rely on runtime mechanisms to capture (and re-enter) execution contexts dynamically. There’s a good few simple examples of programs written in CPS that have natural analogues using control operators. E.g. with some handwaving, you can squint and say that trampolining a program in CPS is a bit like the trivial schedulers people write with effect handlers.

They’re a lot less error-prone than manually destructuring values using switch constructs and if-else statements. They’re also likely to be more efficient than what humans may naively write. Most match compilers emit decision trees (or matching automata) that discriminate over several patterns at the same time.

For example, code like:

type abc = A | B | C

match e with
| (A, B, _) -> 0
| (A, C, _) -> 1
| (B, A, _) -> 2
| (B, C, _) -> 3

may be lowered into a decision tree (DAG) that looks like:

image
image409Ă—404 11 KB

As you can see, the first component of e (e.0 - the 0th component of the tuple) is tested only once. The matching of A and the matching of B covers the first two (and last two) cases where they overlap at that position.

For fun, I often like to note that Andrew Appel’s book “Modern Compiler Implementation in ML” has a C edition where he has a figure that shows you how to do manual pattern matching over tagged unions in C, but the following figure just uses ML-esque pseudocode for patterns:

image
image744Ă—205 39.3 KB

image
image847Ă—358 88.4 KB

I know which one I’d rather write and maintain!

1 Like
3

I wrote something that may be helpful for you: Practical OCaml - DEV Community

And see also the follow-up: Practical OCaml, Multicore Edition - DEV Community

2 Likes
4

The idea of an ORM is to avoid typing SQL queries. An imaginary and simplified example would be: Orm.insert(my_object) instead of INSERT INTO my_object VALUES (my_object.a, my_object.b).

If you want something closer to an ORM, just have a look to Petrol.

5

This is all super helpful. Most of my experience with async is in JS and pythonl. It took me a minute to understand, but I think I’m with you. This gave me a lot of understanding into monads and function coloring. I will dig in a little more into what exactly effects are and how they work. What is meant by trampolining?

Match as a DAG makes sense to me. Thank you

Great resources, I really appreciate this. The first one clarified a lot for me, and the second one definitely helped me understand eio. Still working through these. I would like to get to a point where OCAML feels as natural to me for small tools as writing Python, and this helped me realize that might not be too far off.

Thank you very much, I appreciate this - I had not heard of Petrol. I come from an ORM background, mostly django-orm, but I’ve come to appreciate and much prefer the simplicity of using raw SQL. I’m not sure the ergonomics of an ORM really offer that much when it comes to basic CRUD. With that said, I have spent some time trying to programmatically generate CRUD, mainly in Ruby and Go. I tend to find it a lot easier with ORM type objects. I got halfway through trying to generate endpoints and realized I was basically building an ORM. I was surprised and delighted to find that Ocsigen has done some of this work, I believe they use PPX to automatically generate REST APIs. PPX seems more like the preferred style of accomplishing this, and this generally seems like a better idea to me than what I was trying to accomplish.

1 Like
6

It may help to understand how await could work in JavaScript. It’s not like an await can block the program entirely: the JS event loop can execute other things in the meantime. You can imagine that the JavaScript runtime must have way to reify a form of “continuation” (the context in which the await occurred) which it can defer and restore later. These monadic concurrency libraries in OCaml require you to write in monadic style: this style explicitly sequentialises the computation through continuation functions - these functions can be deferred and executed at a later time (by the scheduler these libraries maintain). As for colouring, JavaScript has a similar “function colouring” juxtaposition in that await can only be used in functions marked async; although this is somewhat less problematic than monadic style (whereby combining monads can lead to all kinds of monad transformer type tetris, or by the popular adage: “monads don’t easily compose”).

In OCaml 5, the situation is different because the runtime can reify (delimited) continuations for you. You can install effect handlers and then execute code that escapes to these handlers. This is similar to exception handlers, except effect handlers are provided with a continuation (means to return to where the effect was performed, with a value). You can imagine the utility in implementing asynchronous concurrency libraries with these primitives: you get to write code in direct style (no explicit monadic style) which can then be deferred and restored later. You’ll find that this area of computer science is riddled with overloaded terminology which effectively imply the same kind of idea (“(stackful) coroutine”, “generator”, “delimited continuation”, “effect handler”, …).

I wouldn’t worry about this, it’s not important, really, it’s just a needless allusion I’ve made.

But: programs written in CPS can be transformed in such a way that invocations of continuations can be rewritten to return thunks (e.g. as an opaque closure value or explicit closure-like object comprising a closure and its intended arguments). Then, the entire program can be “trampolined” (stepped, at the level of functions) by a loop which executes consecutive thunks. The analogy is fairly weak: a program written in CPS which is trampolined by a loop is similar to a direct-style program which raises a “yield” effect at synonymous points, with a scheduler that decides to always invoke the continuation of the code that yielded next (with the related continuation reified dynamically by the runtime). It’s an analogy between a static translation and a dynamic implementation, which is really what the move from monadic concurrency to direct style concurrency (with effects) is all about.

A lot of papers about delimited control take this route for illustrative purposes: e.g. “here’s a function in CPS, now look at the version in direct style which uses delimited control to reify continuations dynamically”.

Your background in JavaScript may be very useful as JS is a good language for learning asynchronous programming and its dynamic nature (e.g. ability to dynamically invoke functions) makes implementing examples of callback-heavy code fairly convenient.

Anyway, don’t worry too much about things I’ve discussed here: verbosity and needless allusions are bad habits of mine.