What are the biggest reasons newcomers give up on OCaml?

Fantastic summary work @patricoferris, thanks a lot! (cc @davesnx : if you were thinking of working on this yourself, maybe don’t.)

With my “OCaml Software Foundation” hat, I can say that in general “developer experience” is one thing we want to help improve. If someone that we could fund would be interested in spending time turning more of the feedback points here into actionable issues – and ideally in helping fixing these issues – please get in touch. (Out of the three clear feedback point you mentioned, “documentation”, “compiler errors” and “Windows support”, we have funded actions on documentation and Windows support, and are open to do more in all three areas if we find the right thing to fund.)

With my “compiler developer” hat and my “contributor to the tooling ecosystem” hat, I also want to say that many of the issues listed actually require a lot more work than people might imagine upfront, and find themselves competing for contributors which are a scarce resource. My general feeling is that most people already contributing are trying to improve things, but progress is slow because solving complex issues takes time.

7 Likes

And yet, it is precisely how monads were discovered and applied to PL semantics and implementation. We can go to (nearly) the source: Wadler’s original paper where he proposes monads for Haskell: “The Essence of Functional Programming.” There, in section “5.2 The Past” he writes:

Finally, something should be said about the origin of these ideas.
The notion of monad comes from category theory [Mac71, LS86]. It first arose in
the area of homological algebra, but later was recognised (due to the work of Kleisli
and of Eilenberg and Moore) to have much wider applications. Its importance emerged
slowly: in early days, it was not even given a proper name, but called simply a “standard
construction” or a “triple”. The formulation used here is due to Kleisli.
Eugenio Moggi proposed that monads provide a useful structuring tool for denotational
semantics [Mog89a, Mog89b]. He showed how lambda calculus could be given call-by-value
and call-by-name semantics in an arbitrary monad, and how monads could encapsulate
a wide variety of programming language features such as state, exception handling, and
continuations

In that paper, Wadler shows how you can rederive the call-by-value CPS interpreter, from a generic call-by-value CPS monadic interpreter, by instantiating the CPS monad. He mentions that this is a generic trick, and you can get other interpreters (e.g. for lambda+Errors) by instantiating other monads. Given that denotational interpreters for languages like lambda+Error, lambda+state, lambda+continuations had existed for well over ten years by the time Moggi came to write his paper, it is probably reasonable to assume that he came to his monad ideas by observing the similarities and differences among various denotational interpreters. The similarities became a monadic interpreter (the “call-by-value semantics in an arbitrary monad”) and the differences became various monads.

All of this is pretty anodyne, and very obvious from Wadler’s paper. Now let me illustrate the problem that a beginner has with monads.

1 the beginner wants to write a program that will raise and catch exceptions.
2. the beginner’s PL wizard friend tells them “use the result monad instead, it’s great!”
3. and sure, you can write lambda, variables, constants, application, and even raise with the result monad.
4. but nowhere is it written down how to do try...catch.

Is our beginner to just figure it out for themself? I mean, I learned this stuff in 1986: so going on 37 years ago. But our beginner has to figure it out without the benefit of a graduate course in programming languages from Prakash Panangaden: it’s not so easy for them.

For the record, it’s straightforward (in the language of the result monad we can find in OCaml) to implement try-catch:

let trycatch _M _FN = match _M with Ok _ as v -> v | Error e -> _FN e ;;
val trycatch :
  ('a, 'b) result ->
  ('b -> ('a, 'c) result) -> ('a, 'c) result = <fun>

The type tells us that this probably does what we expect. And we use it thus:

let readit f =
  let open Fpath in
  let f = v f in
  let* contents = OS.File.read f in
  Ok contents ;;

# trycatch (readit "/etc/motd") (function `Msg s -> Ok s);;
- : (string, 'a) Rresult.result =
Ok
 ".... contents of /etc/motd ...."
# trycatch (readit "/etc/nonexistent") (function `Msg s -> Ok s);;
- : (string, 'a) Rresult.result =
Ok "/etc/nonexistent: No such file or directory"
# 

I’m not claiming that this is the best way to implement a try-catch combinator. But some try-catch combinator should be supplied. It shouldn’t be up to the user to just dream one up. Notice also that in this example, we don’t leave the result monad in order to do a try-catch: we stay in the little language. That’s important, too: it means that your beginner doesn’t have to jump back-and-forth between the “language above the monad” and the “language below the monad” (or in semantics terms, between the left-hand-side and the right-hand-side of the sematic interpreter).

Alternatively, the beginner’s grug brain dev friend tells them to use alerts to make exceptions safer :slight_smile:

1 Like

Your proposal has at least one signal advantage of “use the result monad”: it’s meant for large-scale programming, where stuff like the result monad is clearly only meant for small-scale programming.

With that said, it would go a long way to convincing me to use the result monad for more of my projects (and eschew exceptions) if there were a way to support the “?” and “return” syntax in OCaml.

[info: the “?” suffix operator when appliied to an expression means to match on the expression’s value, and if it’s an Error, to transfer that Error to the scope of the enclosing function, as a return-value. Similarly, “return” anywhere in a function returns from that function with the argument value.]

I haven’t thought about it in great detail, but ISTR that when I thought about it, it seemed to need code post-typechecker, in order to properly implement “?”. I could be wrong about that.

But with those two things, it becomes feasible to write nontrivial amounts of code with the result monad.

1 Like

For Walder and Moggi, they did what they have to do: it’s the reasonable way when you want to study the semantic of a particular monad, or show how a feature can be encoded monadicaly. What I wanted to point is that a monad is a natural generalisation of (|>) and let binding (with its own semantic, specific to each monad). Consider the following code without any monad:

let double x = x + x
and square x = x * x

let foo i j =
  let i' = double i in
  let j' = square j in
  i' + j'

We can give the foo code a shape that is less idiomatic using |>:

let foo_bis i j =
  double i |> fun i' ->
  square j |> fun j' ->
  i' + j'

and that’s it: |> is the bind or >>= of the specific identity monad (type 'a t = 'a), or conversely >>= and let-operator are a natural generalisation of |> and let binding with a semantic specific to each monad.

Isn’t’t that what you got with this:

let (let*?) = Result.bind
let return x = Ok x

let-operator are still present after type checking. I quote what @octachron told me one day:

1 Like

FWIW, from my perspective OCaml really made strides in usability from where things stood, say, 15 years ago. Camlp4 was a pain to get into for me, and the know-how wouldn’t stick (was still painful to use it on a second project). PPX is such a breeze in comparison.

But in the meantime I got spoiled by interactive debuggers for Python and TypeScript integrated into VSCode. The manner of debugging where you can watch human-readable representations of values of identifiers in scope as you step through the code execution line-by-line in the editor.

6 Likes

My own experience with Monads has been:

  1. they’re usually poorly defined because the OCaml philosophy seems to be that reading types is enough to understand how a monad is to be used.
  2. you can’t use them properly (via the let+ let* or let ppx) without understanding exactly what the code gets translated to (because otherwise you might introduce bugs, for example when mixing monadic code with imperative code and the monad mutates some global state). And it’s always a nightmare to remember exactly how all of these let%...in get composed into a succession of let _ in (fun x -> _).

So to me, as a non-theorist and mostly applied guy, it’s often used with this let syntax sugar that’s a very leaky abstraction that’s not pleasant to use (even if it seems to make code more readable).

4 Likes

That’s interesting, it seems that in the JavaScript world most devs have adapted to their equivalent async/await syntax? I won’t claim that they adapted perfectly without any bumps, but it seems to be the mainstream async programming style now. I’d argue that OCaml’s basic let-operators (let* and let+) are only slightly more complex than async/await and conceptually the same.

I don’t disagree at all with let* and let+ being only slightly more complex than async/await (and have been able to successfully use them as a newcomer with some Rust/Haskell/JavaScript/Python background). I think that, by virtue of their naming, however, they are less “discoverable” in the sense that transfer of intuition from other languages does not happen on its own (unless you have successfully used Haskell before, or have an intuition around using Monads from other sources). Monads are a more abstract concept than async/await and different people have different capabilities and experience regarding abstract reasoning.

Async and await are terms used throughout various programming languages and they refer to a specific kind of “non-blocking chainable computation”.

So when you have this basic understanding of what async/await means and how to use it in JavaScript, you have the ability to quickly figure out how to write code that uses async await in Rust or any other language that implements the same intuition around the async/await concept. Even if there are subtleties that you may not grasp yet, you have a good chance of figuring out quickly what you need to do.

The terms “async”, “await”, “Promise” lend themselves well for internet search via search engines or QA sites: you get reasonable search results relating to the specific type of “chaining computation” you want to do. Anything with operators and symbols is harder to search and find, and then you end up at a Monad tutorial (which may be exciting to some percentage of users and frustrating or overwhelming to the rest).

So while there is beauty in knowing that async/await can be represented (or implemented) as a Monad, a language having an “discoverable surface area that facilitates knowledge-transfer” can make onboarding practical-minded people who just want to get some work done or build a cool thing so much easier.

8 Likes

As always, the biggest issue with let+/let* compared to something native
to the language like async/await in Rust, JS, etc. is that they mesh
poorly with existing control flow structures. Async/await typically
works well with loops, exceptions, early return, conditionals, argument position,
etc. whereas let+/let* is more rigid.

6 Likes

Well they’re also more general than async. :slight_smile: Which is why I miss them in ReScript. :frowning:

1 Like

I’m currently searching for a simpler ML-like language.
Ocaml is a nice language and I’m frustrated because starting in my simple project requires a lot of learning and fixings.
I used C in a simple project I don’t remember about a similar pain.

1 Like

How much of that frustration is due to the language, and how much is due to the tooling? More specifically, how much is due to the difficulty of understanding how OCaml builds work?

I ask because my pet theory is that much of the problem for newcomers is the sheer opacity and complexity of the toolchain. It is similar to C (headers, sources) but a lot more complex. A common response to problems is “just use Dune”, but I personally think that is an anti-answer, not far from “you don’t need to know”. Some of us at least want to know exactly how our sources get transformed into running code; to me at least that’s an essential part of understanding “OCaml” - meaning not just the language in the abstract but all the mechanisms that make it work in practice. And that is woefully underdocumented.

4 Likes

The initial build and installation of opam, ocaml, etc. has not always been pain-free for me, and when there’s a problem, it’s pretty confusing at first. It’s possible that that’s part of what @Drito has in mind.

I strongly agree with this.

I started with a TypeScript + Rust project that I’m now converting large portions to OCaml. However, I ran into many dune, jsoo, js/dom interaction issues (as evidenced by my question history). If any of the issues were unresolved, I would have probably gone with ReScrip or stuck with TypeScript. It was only through the helpful responses of this forum that I got to a position where I feel productive in ocaml / dune / jsoo.

Without this forum, limited to only googling, I’d probably have quit quite early on.

In contrast, Rust/cargo & Scala/sbt were easy to get started in.

4 Likes

I think that this somewhat depends on what you mean by the “toolchain” and what you are using OCaml for. Casting my mind back, I don’t think I had any particular difficulties in building simple beginner-like projects using dune (or Makefiles with ocamlfind for that matter, save that as I recall it how dependencies work when linking up your project was not well documented). The problem, such as I had one, was in the main with the OCaml language itself, including its syntax and its type and module systems.

I don’t use OCaml for writing for the browser, and if that is your interest then using jsoo and interacting with the DOM adds an additional layer of complexity because the jsoo wrappers only take you so far and you end up writing code in a frankenstein-like hybrid mixing of OCaml and Javascript and this complicates amongst other things use of the LSP; but that issue seems to me to be about interacting with javascript as a backend rather than than the OCaml build chain itself.

I have not counted them up, but I think that this is reflected in your posts. I am also of the impression that you have adapted to OCaml unusually quickly. For the general case I suspect it is the documentation of OCaml and its libraries which needs to be improved the most and this seems to have been recognized and is being acted on.

I think, but I am not sure, that you are basically saying that ocamlc/ocamlopt are “woefully underdocumented”. I find this surprising / I don’t agree, they are covered in the manual, see the chapter on ocamlc for example.

Your different interpretation might come from the fact that the tools were written with a build model in mind, back in the nineties, that is inherited from C, pretty simple, but also very different from your own expectations. In that build model, compilation units are pairs of a .ml and .mli file in the same directory, and build artifacts are produced in the same directory. Compiled artifacts for compilation units are then linked together, in command-line order, into complete binaries or library archives.

5 Likes

No, I was specifically talking about the build part. So we disagree; I do not think those sections are good documentation, esp. for newcomers. Just as an example, what is a newcomer to make of -no-alias-deps? I’m not a newcomer and I’m still not entirely confident that I know what it means. (Please do not explain it, I use it as an example of underdocumentation.) Even more obvious, look at the titles of those sections. Native-code compilation is not batched?

I think the OCaml build protocols are about as complicated as it gets. It’s a gross oversimplification to boil it down to "

Even if you know what that means (“compilation units are pairs”? wtf?) it is of no help at all when you run into problems. I think I’ll try putting my implementations and my interfaces in different directories; kaboom! It should work why doesn’t it? If my module A depends on module B, why isn’t b.cmi always sufficient? Why on earth do I need b.mli? Do I always need to put my b.cmx/cmo in the -I path? Well, no. it depends.

Now, I’ll admit I’m a little biased. I don’t even want to think about the amount of time I spend figuring this stuff out in order to write Bazel rules for OCaml. I can’t count the number of times I thought I had it figured out only to find another use case that blew everything to smithereens. All of which could have been avoided if the documentation did not suck. Or maybe if I were a little less dim, but let’s not go there.

In any case I asked my original question because I’m not sure if I’m an outlier or not. I like to know what my toolchain is doing, which is one reason I’m not very fond of Dune. Others are quite content to ignore that kind of stuff. I just would like to know if others found this kind of stuff (as opposed to “how does the language work”) a barrier to using OCaml.

2 Likes

As an aside: in many cases the only way I could figure out how the toolchain was supposed to work was by examining the log file of a Dune build. Not because I could not understand Dune but because I could not understand the OCaml build discipline.

I’d like to push back on this being a typical newcomer use case. I am happy to believe that newcomers struggle with setting up a dune project since it is the recommended build tool and its documentation has a reputation for being opaque, but less so that they are going to want to delve into the depths of the OCaml compilation model to start off with.

1 Like