Streamlining the OCaml build ecosystem

I think with opam there actually isn’t a need to centralize around a build tool. When I do a change to a repo in opam I usually don’t care about how it’s built, I just check out the code and do opam pin add foo ./ and every time I want to check my change I do opam reinstall foo and it syncs the latest changes and builds.

That being said, if your needs are simple, maybe check out pds which I have posted about in this category earlier this week.

My standard workflow for a repo is to clone the following repo: https://bitbucket.org/mimirops/ocaml-template/src

Then there is a script in it called genesis (see the README) which takes some parameters and turns that generic repo into the specialized one with a default pds config. After that, just do make and make test. pds is meant to make the simple case simple, hopefully it’s good at that :slight_smile:

Exactly; the problem is documentation, not changing what existing projects use.

The community needs to document the various prominent build systems so that developers can make informed decisions and also learn quickly how to utilise a tool to achieve their goal.

1 Like

As you observe, the OCaml community historically has been plagued with many build systems. I am seeing a lot of momentum behind jbuilder and would use it for new projects. The reason that I would recommend it is that (unlike all previous build systems) it is quite comprehensive, is not an assemblage of tools (modular but hard to understand), designed around Opam, and extremely fast. I also like Ocamlbuild but its limitations triggered the creation of Oasis and you end up with a quite tall stack of tools that are hard to understand and not portable to Windows. I see a lot of potential with jbuilder to converge on a build system.

3 Likes

jbuilder looks promising, but before it can completely displace oasis+ocamlbuild I think it needs to provide a super set of the features (in particular generic rules and plugins, for cppo, ocamlify and the likes).

3 Likes

The jbuilder story really has been extraordinary. The initial goal of jbuilder was to facilitate our public release process, but given the speed and enthusiasm with which it’s been adopted, it now seems destined now to become the default OCaml build system.

While Simon is right that jbuilder isn’t yet fully general, it’s a good bet that it’s flexible enough for your next project. It’s already used for all Jane Street packages, many Mirage and Docker packages, and js_of_ocaml and atdgen, all of which attests to its range. I’d guess jbuilder can handle the needs of 95% of packages.

In the meantime, the functionality is quite good. It builds things 5x faster than ocamlbuild, it supports Reason, compiles to Javascript via js_of_ocaml, emits .merlin files, supports test targets, and will soon have support for doc generation. There’s even a manual.

http://jbuilder.readthedocs.io/en/latest/

You can follow along with the adoption by looking at the revdeps in opam. It seems to be growing at a pretty good rate.

https://opam.ocaml.org/packages/jbuilder/

My hope is that jbuilder can make a real difference to the OCaml ecosystem, making it way easier for people to get into OCaml.

y

9 Likes

The Jbuilder migration is going very well indeed – most of our big repositories like DataKit (see dev reports) have been ported, and see here for notes on the Mirage/jbuilder efforts.

2 Likes

I also find jbuilder interesting but haven’t had the time to play with it yet. Currently it seems to lack, as @c-cube pointed out, a mechanism for sharing build recipes among users (as opam or ocamlfind packages). So, say, Menhir and ocamlyacc are well supported (because their support is hardcoded in the implementation), but if someone wants to build a new parser generator they are out of luck – they will not be able to provide the same user experience to users. I think that, long term, it’s important for the tools we use at the base to support an open world, by accommodating new tools coming up in the future with as little an entrance barrier as possible.

Of course this is also in tension with the simplicity and robustness of the tool (which explains tools like obuild trying to cut down on extensibility), and an alternative is to have the developers of a centralized tool just be very reactive in adding built-in support for new things.

3 Likes

Agreed. I expect jbuilder will eventually move in that direction, but it’s getting surprisingly far by just being light on its feet in terms of supporting things of key importance to the OCaml community.

1 Like

Thank you to Jane Street for solving this problem. Kinda funny how simple the solution is, at least on the front end. I’d personally have preferred fewer parentheses aka something like JSON or YAML, but I’m happy with anything that ‘just works’.

JSON is worthy of consideration but YAML is a disaster (see spec).

@bobzhang mentioned that a JSON frontend allows for easy integration with tooling if you specify your schema with something like http://json-schema.org/

I think for that reason an alternative for jbuilder is something that should still be considered. Keeping the existing sexp frontend of course.

Aww, come on people! Can’t we just all get down with our functional
programming roots? S-expressions have a long and storied past. And unlike
Json, they have a comment syntax…

y

4 Likes

There are two reasons for this:

  1. Tooling: vscode can provide validation and semantics based auto-completion, as a 12 years Emacs users who wrote hundreds of thousands of lines Lisp code, the user experience can not be beaten by an Emacs mode, S expression will be never popular and it looks weird to new users …
  2. Standardize: so that third party tools don’t need dig into jbuilder details
3 Likes

Here is a link of how awesome it is to edit json file in vscode: https://code.visualstudio.com/docs/languages/json#_intellisense-validation
it works as advertised

But, but… we’ve standardized on s-expressions! What’s non-standard about that? It’s a standard that goes back to the 50’s!

I’m feeling a rant about the virtues of ed coming on.

(for those who are not detecting the sarcasm, I don’t actually object to someone adding a JSON frontend, though I still cling to my parens.)

y

1 Like

Reasons not to use JSON:

  1. No comments
  2. Harder to read and to write than s-expressions
  3. Jbuilder will be gaining light meta-programming abilities – these will be awkward to represent as JSON exacerbating point 2

Compared to that having completion for an editor that most OCaml programmers don’t use does not seem worth the cost.

JSON really wasn’t designed to be used as a configuration language, and should not be used as such. This is XML in the 2000s all over again.

2 Likes

I like s expression more than JSON too, and Emacs is my editor.
When I made BuckleScript build system, the first front-end language is S expression as well, later I decided to move it into JSON, I think it is a wise choice.
If we do want to make jbuilder or OCaml more mainstream, we should not use S expression, people do care about those superfcial issues.
For Item 1, most editors support comments, item 2 is subjective, item 3: I wish jbuilder won’t add computation in configuration …

3 Likes
  1. jbuilder’s own sexp format has its own multiline comment syntax. No reason why something similar can’t be added to json.

  2. Agreed but the familiarity of the majority of programmers towards json is so overwhelming that in practice it doesn’t matter.

  3. Why is that? Is it because it’s harder to write valid json as strings?

Compared to that having completion for an editor that most OCaml programmers don’t use does not seem worth the cost.

My experience does not confirm this. Almost every single younger programmer that I’ve seen is using vscode or atom. It’s imperative to support these editors properly. Note that they already have OCaml support on par with Emacs. So it shows you that many people do use and care about them.

JSON really wasn’t designed to be used as a configuration language, and should not be used as such. This is XML in the 2000s all over again.

While that’s the case, this position does not benefit the wider adoption of OCaml. If using sexp as a configuration language is indeed a bottleneck for people to get started with OCaml, this should be fixed regardless of how superior sexp is to json.

6 Likes

Ah yes “let’s just add comment to json”, nobody though about that one, except the 300 projects that all did it differently. If you really want a non-lispy configuration format that supports comments, please follow the Rust example and use toml instead. :slight_smile:

That being said, I think jbuilder’s syntax is fine, and this is all a huge pile of bikeshedding.

5 Likes

That’s not really the case. A lot of projects are using http://json5.org/

The download numbers speak for themselves

Rust’s cargo uses toml. It looks pretty good. Is it well supported?

BTW one of the things I’m grateful for with ReasonML is that we seem to have a serious influx of young programmers into the language. Finally I feel like we’re releasing OCaml from the grips of the ‘get off my lawn’ crowd and we can actually talk about introducing modern changes without getting the inevitable ‘I was perfectly happy with my PDP-10, ergo we don’t need this new thing’ response. (BTW I don’t mean you, Yaron :slight_smile: )

3 Likes