Musings on OCaml tooling

Ecosystem
1

Hello amazing people,

I am a relatively new user. I absolutely adore OCaml, but I am constantly running into missing or broken tools, especially when used in combination, which makes both learning and using the language very frustrating.

I am already at a point where I can resolve most of the issues myself, but I am still miles away from having a setup that’s enjoyable to use, especially as I move between other people’s projects.

I’d like to have an editor plugin that rebuilds the project automatically, provides great type information and error feedback, and lets me load any module into a REPL.

I’ve been using VS Code with Reason Language Server and while it’s fine, it only works for Esy and BuckleScript projects and doesn’t support REPL and there’s currently essentially no good way to use REPL with Reason (I spent a little bit of time of extending dune utop to work with reason, but I still need to return to that and finish it).

So I tried Emacs with Tuareg, which gives me REPL but this is not usable with either Esy or Reason (or I wasn’t able to get it to work anyway).

Now, look, I don’t have any problem switching from Opam + Makefile + Ocaml syntax to Esy + Dune + Reason syntax. But every projects uses its own combination of these tools and there is no editor/plugin that works in every setting and I don’t want to keep setting up these projects, changing editors, relearning all the tricks and keybindings, etc.

So this finally brings me to my questions.

Question 1: How do people handle this? What are your setups and how do handle switching between the different projects?

Question 2: Wouldn’t it make sense to have one ultimate language server that detects your environment (opam switch, esy, bucklescript, whatever), lets you enable Reason syntax on a whim, uses the correct build system, lets you pick your backend (native, jsoo, bs), etc.? This might be naive, but I don’t see why it shouldn’t be possible. It’s all just software, we should use it to alleviate suffering, not multiply it.

Again, I can handle every particular scenario manually, but it’s still painful and not fully what I want, and I’m 100% certain it loses newcomers. I had been trying to start using OCaml for years and the state of the ecosystem always put me off, I was never able to even get started. It finally got better recently and I applaud the heroic efforts of the community, but it’s still very far away from ‘works out-of-box’. Compare this with e.g. Haskell tooling, where you already had the amazing stuff I mention (automatic rebuilds, great type feedback, repl) years ago (I haven’t used it much recently, but I’m fairly certain it’s even better today).

Finally: these are meant to be very constructive questions. I have some spare time and will gladly spend it to resolve some of the issues. I’m just not sure which project, editor, plugin to focus my energy on.

8 Likes
2

I’m using dune + emacs with tuareg and merlin, I suppose this is quite standard.
Switching all my projects to dune has really simplified my life. I guess dune is going in the direction you want.
But the REPL in emacs, while being convenient (I use C-c C-e a lot!) is not perfect. For instance, why don’t we have type info and completion on the ocaml buffer (while it works on any aaa.ml buffer)?
Maybe that’s just a bad config from my side. Moreover the REPL buffer is very slow when it becomes large.

2 Likes
3

Thank you for your input.

I’m indeed using Dune whenever I can, but not all projects do this, especially the older ones. Case in point: Dune itself is not built with Dune, for obvious reasons, so I don’t benefit from any standard tools when working on it (I guess it is a subquestion for Dune devs as to what setups they use).

So, while picking the right tools for my own projects partially resolve the question, it doesn’t help me fully, as I often also want to work on projects that are not mine to help others.

Also, as you mention, the REPL experience is subpar even when it works. https://github.com/dbuenzli/down provides completion but, while it’s a great project, it means you can’t use utop, introducing yet another choice of tooling.

I guess this is as good place as any to reiterate what I think the main problem is: for every piece of the project setup, you have at least two incompatible choices:

  • Packages: opam/esy/npm (for bucklescript)
  • Syntax: ocaml/reason
  • Builds: dune/make/custom setup
  • Repl: utop/rtop/plain ocaml top
  • Backend: native/jsoo/bucklescript
  • Language server: reason/ocam/merlin
  • Editors: vim/emacs/code/atom/…

This leads to combinatorial explosion in complexity and means there is no coherent set of these tools that work well together (some of these parts combine well, but never all of them).

1 Like
4

dune is built with dune. There’s a bootstrap system so it uses a mini-dune to build itself, but once it’s built we have merlin support, watch mode, dune runtest, etc available when developing dune itself.

6 Likes
5

I use Vim which loads the Merlin installation in case it is installed in the switch. Given many/most projects use dune now, as long as it builds I can get type information since dune helpfully generates merlin files automatically.

I don’t use any language server, nor reason, bucklescript etc. I think a lot of your issues seem to be with Reason and Bucklescript which for all intents and purposes are a different language/ecosystem.

So just set up one thing the one way, the other thing another way, don’t mix and you should be good to go.

2 Likes
6

Thanks for the answer.

I don’t use Bucklescript at the moment, but it’s true that many of my issues stem from using Reason. I already considered switching to OCaml syntax, but a) it’s painful, b) it’s just syntax, it should work out-of-box (and mostly does); when something is broken, the answer shouldn’t be to avoid using it, we should fix it.

Also, I cannot really settle for one setup just yet, both because I’m still exploring the OCaml ecosystem and would like to pick the best tools available and also because I’d like to work on other people’s projects, which use their own arbitrary setups that I have to conform to.

7

I think you might be missing the sociology of it all. Reason and all related tooling were created relatively recently in a largely independent fashion to the established OCaml tooling. The developers of each set of tools hardly overlap, and the user communities are quite distinct.

The intent in creating Reason was clearly to create a completely different ecosystem. You say that “it’s just syntax”, but it’s not, precisely because the tooling is different. If Reason people had just cared about their syntax and had wanted to maintain high compatibility with the OCaml ecosystem they wouldn’t have written their own build system, formatter, etc etc. They would have shipped a parser (the OCaml compiler actually supports plugging in a parser) and called it a day.

And given this intention of creating a new ecosystem on the Reason side, I don’t think it’s fair to say that OCaml tooling is “broken” because support for the Reason workflow is not perfect. These tools are meant to serve the OCaml ecosystem, it is already nice that they actually partly support Reason.

In few words, you talk as if we have one ecosystem here but really we have two and it’s not surprising that mixing the tooling won’t work very well.

3 Likes
8

Everything is possible. It just needs someone to do it :slight_smile:

Haskell certainly had its share of issues. From Cabal to Stack, editor integration tooling, REPL ability to (re)load modules, and the fact that for compiling to JS PureScript has always been a more viable option than GHCJS, so you need to adopt that toolchain (Bower, Pursuit, etc.) as well.

The ecosystem is going through a period of growth and expansion. That comes with a growth spurt of new tooling as people try to figure out what works and what doesn’t. I’m sure things will settle down in time.

I think you’re hitting this pain point especially hard because you’re still in the project-hopping exploratory phase. Most people, once they settle down and start working on a project day-to-day, they work out a stable subset of the toolchain and they don’t bump into these issues as much. That’s not to say things become perfect–especially for multi-stack projects it is still somewhat painful. But like I said, I’m hopeful that things will smoothen out in time.

9

Hey, thanks for chipping in!

I think you might be missing the sociology of it all.

I understand these points and while I tentatively accept them for Bucklescript, there’s no reason (heh) to accept them for Reason-as-syntax since that is also being used for amazing fully native projects, such as Revery, which you can then use as libraries in classic OCaml projects. It’s the same ecosystem and the same community unless you insist it be divided.

In few words, you talk as if we have one ecosystem here but really we have two and it’s not surprising that mixing the tooling won’t work very well.

If I understand you correctly, you mean standard vs. Bucklescript ecosystems. If so, this is not what I had in mind at all (or rather, a tiny part of it). Instead, what I meant is that there are several choices for every part of a project, from packager, to build systems, to repls, to language servers, etc., etc., and that these tools don’t always combine very nicely, while there’s no good reason why they shouldn’t.

In any case, my intention was to understand which project(s) to contribute to, rather than discuss philosophy of language ecosystems. And as I said, I am relatively new here, I don’t have an expert understanding of the technology and the community. Nevertheless, I don’t think my experience as a new user should be discarded; I think I perceive real problems (which long-time users might be oblivious to) and I don’t think I am alone in this.

3 Likes
10

Haskell certainly had its share of issues.

Oh, definitely, Haskell was an absolute mess just 5 years ago, and many parts still are today. But it improved considerably and it was trivial to get setups (e.g. emacs + intero or vim + hie) that offered features that I still cannot get in OCaml today, or I just don’t know how.

The ecosystem is going through a period of growth and expansion.

Definitely, and I’d like to help this growth :slight_smile: But since I’m new here, I was wondering which project(s) I should focus my efforts on, hence my questions.

But like I said, I’m hopeful that things will smoothen out in time.

Unlike you, I lack the faith that things will get better on their own, without somebody voicing their concern and others fixing those concerns. Also, I’d like to have a pleasant development experience today, not in time :slight_smile:

2 Likes
11

I suppose this is a difference in philosophy. IMHO the people voicing concerns will often need to step up and help fix those concerns :slight_smile: In other words the ecosystem is simply not mature enough that there will be enough people with time or money budget to step in and fix whatever is asked. And I doubt it ever will be–many mature ecosystems go through the exact same issues.

It’s true that often big(-ish) shops with enough budget will contribute and we’ll all benefit. But relying on them exclusively probably won’t get us exactly the results we hope for, because often they tailor the tooling to suit their own needs at least to a certain extent.

2 Likes
12

In case this is useful, these are the de-facto defaults for the OCaml community these days:

  • Packages: opam/esy/npm (for bucklescript)
  • Syntax: ocaml/reason
  • Builds: dune/make/custom setup

The choice of backend naturally depends on your use case, so there will never be a default there.
For editors, different people naturally have different preferences.
For repls and language servers, you may be right that no default has evolved yet.

Like people have eluded to before, the Reason community seems to be doing their own thing and is quite separate from the OCaml community, for better or worse.

2 Likes
13

oh, actually this one is easy:

(add-hook 'tuareg-interactive-mode-hook #'merlin-mode)
(add-hook 'tuareg-interactive-mode-hook #'company-mode)
2 Likes
14

In such case since you want to use ReasonML, I am confused why you’re posting the issues here. The language semantics are the same, but the parser is different and so is the tooling. Yes, there are a few people using ReasonML here as well but you will probably find a lot more in the ReasonML Forums and they will be more capable of helping you with the Reason tooling.

1 Like
15

Thanks, I’ll try to use that forum also, I wasn’t aware of it!

That said, the issues I mention are not at all specific to ReasonML, that’s just what I’m struggling with currently. But those issues arise anytime you try to combine several of the standard tools, especially if your editor is less supported.

Luckily, I’ve already found a couple of people who understand what the problem is and I’ll be working with them to make OCaml tooling experience as smooth as possible. I understand that old-timers are not bothered, but as I mentioned, the status quo is a nightmare for newcomers.

16

Much of the problem you’re describing is a result of new, slightly-incompatible tooling created by different groups. So saying that one group that created some fairly well-established and working toolchain, is not bothered with beginners because it doesn’t work with a newer, less-established toolchain, is somewhat unfair. The old-timers didn’t create that incompatibility in the first place :slight_smile:

I think a much better approach is what you have already decided to do–find people who agree and are willing to work with you.

6 Likes
17

Definitely. I’m not blaming anyone, and apologies if I came across that way. OCaml tools are world class and people behind them are wonderful. Nevertheless, the situation (as I perceive it), due to no fault of any particular person, is sub-optimal and needs improving.

If anyone feels the same, please join me. If not, no harm done, we’ll go our separate ways :slight_smile:

1 Like
18

FWIW, ~all of my OCaml work has been done using VS Code, GitHub - jaredly/reason-language-server: A language server for reason, in reason, and opam. I use dune utop ___ for an interactive REPL, and I have a separate dune alias and executable set up off to the side that I use with an invocation like dune build @aliasname -w --force to provide a ghcid-like workflow when I want that. No esy or bucklescript anywhere.

If you’re looking for an OCaml REPL that will _re_load modules (maybe from a buffer rather than from disk), then yeah, that’s lacking, and I often miss it as well. Alas, OCaml isn’t a suitable lisp replacement.

1 Like
19

Thanks very much for starting this thread, and I can assure you that no offence is taken from the perspective of the tooling maintainers. I completely agree with you that the editor setup process is of variable quality, although I am focussed on the OCaml experience – ReasonML has a perfectly good plugin available that uses that community’s tooling of choice.

I’ve discussed this over the past few months with several of the Platform maintainers (dune, ocamlformat, odoc, merlin and so on), and we’re in consensus that now is the right time for us to maintain a core language server as part of the OCaml tooling release process. We need to track OCaml compiler trunk more closely, since every iteration results in breaking changes (unsurprisingly, as the AST evolves). We’re also heading towards a stable release of odoc early next year, which will open up exciting possibilities for further LSP integration.

@rgrinberg has kindly offered to lead this effort and communicate with anyone interested in assisting. Our goal is to have the VSCode plugin entirely bootstrap the toolchain from the installation on Windows/macOS/Linux, without any host configuration being required. He’s only just begun pulling all the pieces together, but please do reach out to him and we can join efforts if appropriate.

15 Likes
21

the Ocaml mode in Emacs is so weird → “Tuareg”.

Tuaregs are camel riders… Tuareg camel riders | These Tuareg tribesmen stopped their p… | Flickr

That I have to figure the order of “u” “a” “r” “e” every time.

I do not understand why you have to type the name, the mode detects OCaml files and activate itself. And opam search emacs outputs it.

Why don’t it name itself OCaml-mode directly

There is already a caml-mode, that would be too confusing.

3 Likes