Why is building Ocaml projects still so hard?

Ecosystem
1

Hi,
I think I use Ocaml software since more then 15 years now (hello OcamlMakefile, OMake, ocamlbuild, …) and despite things have improved with opam and dune its still is so hard to build Ocaml software.

In the last decade most of the languages that put some work in a package manager have found a way to build software comfortably. Not just scripting languages also compiled languages like Rust spit something out, even if the build times are long and scary looking. With Ocaml it always certain it doesn’t work. And when it fails I have to brace for a complex search between the blurry lines of what dune and opam define. Why is that? It is so frustrating.

The last candiate: GitHub - jchavarri/jsoo-react-realworld-example-app: Exemplary real world application built with Js_of_ocaml and React

Written by someone that I count to the top tier of Ocaml developers.

But of course, like 99.99% of the time there is a problem with a ppx and because I’m just an avarage Ocamler I’m screwed.

I know you all doing this voluntarily and I’m grateful for all your work. But sometimes it’s just aaaah…

thanks

2

Hey @koala, can you explain the issue?

This repo was created by @jchavarri and was mostly to test an assumption on bundle-size for a blog post (Js_of_ocaml: a bundle size study | Javier Chávarri), so I wouldn’t consider this a stable product/library.

In this exact case is using a pinned version (jsoo-react-realworld-example-app/jsoo-react-realworld-example.opam.template at main · jchavarri/jsoo-react-realworld-example-app · GitHub) of jsoo-react, if there’s an issue with the jsoo-react.ppx the blind suggestion is to update this pin to latest.

Hope was helpful, and don’t get the wrong impression from the quality of the tools where some of them are flagged as " experimental"

3 Likes
3

Dave, to be fair, that project is not marked as experimental. The wording is ‘exemplary’, ‘real world demo’. Nothing there would give the impression that it’s a proof of concept only.

1 Like
4

Good point, but jsoo-react is tagged as experimental and not published to opam for that particular reason

1 Like
5

I’m preparing an answer with more technical details, but I’d like to give a quick answer regarding the nature of this project and the errors I’m getting: The errors are coming from the build tools, originating from a build error of a external ppx dependency. So even a project is experimental, the build/package system should provide a solid foundation, which unfortunately is not the case often times.

6

And here is a more detailed report of what fails. Like always its a ppx dependency. Can you really not living without those? :pleading_face:
In my personal opinion, the continuous troubles with PPX dependencies at least in its current form, stops Ocaml from getting the usage it deserves.

but here are all infos and steps involved, beginning with my env:

gcc --version
Apple clang version 14.0.0 (clang-1400.0.29.202)
Target: x86_64-apple-darwin22.2.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

ocaml setup:

cd ~
rm -rf .opam

opam --version
2.1.4

opam init 
eval (opam init)
ocamlc --version 
5.0.0

according to the readme:

git clone https://github.com/jchavarri/jsoo-react-realworld-example-app.git
cd jsoo-react-realworld-example-app
yarn install # install JavaScript dependencies
make create-switch # create Opam local switch

eval (opam init)
ocamlc -version
4.13.1

make init # install OCaml dependencies
# this is opam install . --deps-only --with-test
# see error message below

error message from make init which translates to opam install . --deps-only --with-test

[ERROR] The compilation of ppx_jsobject_conv.dev failed at "dune build -p ppx_jsobject_conv -j 15".
∗ installed js_of_ocaml-ppx.3.10.0
∗ installed jsoo-react.dev
∗ installed promise_jsoo.0.3.1

#=== ERROR while compiling ppx_jsobject_conv.dev ==============================#
# context     2.1.4 | macos/x86_64 | ocaml-base-compiler.4.13.1 | pinned(git+https://github.com/little-arhat/ppx_jsobject_conv.git#b1c43be#b1c43be2)
# path        jsoo-react-realworld-example-app/_opam/.opam-switch/build/ppx_jsobject_conv.dev
# command     ~/.opam/opam-init/hooks/sandbox.sh build dune build -p ppx_jsobject_conv -j 15
# exit-code   1
# env-file    ~/.opam/log/ppx_jsobject_conv-88423-af26c1.env
# output-file ~/.opam/log/ppx_jsobject_conv-88423-af26c1.out
### output ###
#       ocamlc src/.ppx_jsobject_conv.objs/byte/ppx_jsobject_conv.{cmi,cmo,cmt} (exit 2)
# (cd _build/default && jsoo-react-realworld-example-app/_opam/bin/ocamlc.opt -w -40 -safe-string -g -bin-annot -I src/.ppx_jsobject_conv.objs/byte -I /ocaml-joo-s[...]
# File "src/ppx_jsobject_conv.ml", line 573, characters 12-39:
# 573 |           | Pext_decl (cons_args, ctyp) ->
#                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^
# Error: The constructor Pext_decl expects 3 argument(s),
#        but is applied here to 2 argument(s)



<><> Error report <><><><><><><><><><><><><><><><><><><><><><><><><><><><><>  🐫
┌─ The following actions failed
│ λ build ppx_jsobject_conv dev

So it seems like a problem with this version:

opam search ppx_jsobject_conv
# Packages matching: match(*ppx_jsobject_conv*)
# Name            # Installed # Synopsis
ppx_jsobject_conv --          pinned to version dev at git+https://github.com/little-arhat/ppx_jsobject_conv.git#b1c43be

Because with with the latest ocaml version it works

ocamlc -version
5.0.0
opam install ppx_jsobject_conv

So like always I tried to solve it by fiddling with dune-project and jsoo-react-realworld-example.opam which I find extremely annoying. Sorry.

7

In my experience it’s usually a result of not locking down dependency versions properly. IMHO it’s a big problem in the opam ecosystem. And it seems likely here too.

The pinned commit version of ppx_jsobject_conv used by the example project: jsoo-react-realworld-example-app/jsoo-react-realworld-example.opam at main · jchavarri/jsoo-react-realworld-example-app · GitHub

pin-depends: [
  [ "jsoo-react.dev" "git+https://github.com/ml-in-barcelona/jsoo-react.git#9e86c9f" ]
  [ "ppx_jsobject_conv.dev" "git+https://github.com/little-arhat/ppx_jsobject_conv.git#b1c43be" ]
]

The unbounded version range of ppxlib (and other dependencies) used by ppx_jsobject_conv: ppx_jsobject_conv/ppx_jsobject_conv.opam at b1c43be251d8c7d39ee32ddc9968fdc3d0042c0b · little-arhat/ppx_jsobject_conv · GitHub

depends: [
  "ocaml" {>= "4.10.0"}
  "dune" {>= "2.7"}
  "js_of_ocaml" {>= "3.8.0"}
  "ppxlib" {>= "0.22.0"}
  "webtest" {with-test}
  "webtest-js" {with-test}
]

We can see that the Pext_decl constructor arity changes between different versions of the AST: Sign in to GitHub · GitHub

My bet is that some time between ppxlib.0.22.0 and ppxlib latest version, the default constructor arity changed from 2 to 3. And you can guess the rest.

This is why I insist on bounding my dependency version ranges, although in the case of 0.x.y i.e. projects which have not even reached their first major version, they can effectively make backwards incompatible changes even on a bugfix patch release.

1 Like
8

Thank you very much for your elaborate answer.

Do you see a simple way to fix this particular problem?

What you mention here with unbound dependency version ranges, seems to common problem, is this somehow addressed or discussed in the community?

9

If I understand the problem correctly:

If a project pins exact versions, you run into the problem of: I have two dependencies that require different versions.

If a project requires >= x.y.z; you run into the problem that a future change may break a type signature.

This seems like an issue independent of OCaml’s build system and is fundamental to all of software engineering, regardless of language.

2 Likes
10

It is addressed by extensive CI testing whenever you submit packages to the official opam-repository. This testing ensures that upper bounds are only added when needed which minimizes the problems @zeroexcuses mentions which are of course intrisic to any package manager.

Now in this case the package does not live in the opam-repository, this means that it does not benefit from this testing. Too bad. But don’t jump to conclusions:

On the contrary, if you use packages from the opam-repository it works exceptionally well thanks to the tireless (and unglamorous) work of the maintainers that patiently curate our additions to it.

My experience with other programming language’s package managers (python, ruby) is that they invariably explode in my hands when I need to use them. This rarely happens to me with opam (which I use much more often).

10 Likes
11

Thanks for sharing a detailed explanation. One possible solution is to install manually ppxlib "ppxlib" {>= "0.25.1"} Release 0.25.1 · ocaml-ppx/ppxlib · GitHub since it’s the one that supports OCaml 5.0 and update the pin from jsoo-react to afccc69a2e864649b6cbdc80690a1887019f466a.

Regarding your question about “need a ppx”:

There are cases where a ppx is preferred since it can reduce the amount of boilerplate you would generate by hand.

For each deriving with ppx_jsobject_conv you would need to implement by hand (and in case of nested, it will rapidly become a pain)

val jsobject_of : t -> 'a Js.t
val of_jsobject : 'a Js.t -> (t, string) result

In your case, could strip this ppx and create those by hand and experience it yourself.

12

What makes Rust different from OCaml is that with Rust, the cargo executable acts as both a package manager and a build system. For example, you just need to add a dependency to Cargo.toml and the next time you run cargo build, Cargo will download the dependency before building your program. In contrast, OCaml’s opam is just a package manager, and you need a separate program for the build system, usually Dune nowadays. One consequence is that you need to list your dependencies in the opam file for the package manager and in the dune file for the build system. What’s more confusing is that these two files use different types of library names, which dbuenzli points out is an obstacle for newcomers. Dune’s feature to generate the opam file automatically doesn’t add much improvement because you still need to list the dependencies twice, since the two places use different types of names.

6 Likes
13

If there were an index that mapped between opam and findlib package-names, that would make this problem solvable, wouldn’t it? Then dune could consult that index while building the opam file ?

And that might not even be a hard problem to solve: a “findlib-packages” field in the opam file ? And then during package-build, that field could be compared against the findlib packages that are added by the package-build/install ?

1 Like
14

There already is. This is how opam-dune-lint works (or “worked”, since dune 3 removed the external-lib-deps subcommand that it needed - see https://github.com/ocurrent/opam-dune-lint/pull/46 for work to restore it).

1 Like
15

I read that page (your link to opam-dune-lint) and it doesn’t seem that the index goes in the right direction: a .changes file is named by the opam package, and contains a list of files, from which can be deduced which ocamlfind package. But it doesn’t allow going in the opposite direction, yes? That’s what’s needed, so that you can write your build-file in terms of findlib packages, and have the opam deps automatically inferred, yes?

An opam package can install several findlib packages, after all.

16

This kind of problem (compatibility of ppx-es with different OCaml compiler versions) is meant to be solved by GitHub - ocaml-ppx/ocaml-migrate-parsetree: DEPRECATED. See https://ocaml.org/changelog/2023-10-23-omp-deprecation. Convert OCaml parsetrees between different major versions, which ppx_jsobject_conv doesn’t appear to use.

Having said that I do try to minimize (but not completely eliminate) the amount of ppx-es used in a project, because trying to get all ppx-es used in a project to agree on a non-empty intersection for ppxlib version bounds and such can be challenging, and is often the main reason why upgrading to a newer version of a compiler can take a longer time (you need to upgrade the package that has an upper bound on ppxlib, and then that triggers the need to upgrade more, and so on and sometimes it requires jumps from one major version of another, and sometimes you get stuck for a while with an unsolvable dependency tree while some of the dependencies get updated).

1 Like
17

I don’t understand the premise that GitHub - jihchi/rescript-react-realworld-example-app: Exemplary real world application built with ReScript and React did something wrong. It is sample code for an EXPERIMENTAL library that was purposely not added to opam.

It seems awfully weird that instead of appreciating the 99.9% of the code that does work in the repo, we are focusing on the 0.1% that worked in the past, but does not work now due some library being updated.

This seems to fall into the category of: if you want things to work perfectly, don’t play with experimental code.

1 Like
18

As far as I understand, ocaml-migrate-parsetree is the ‘old style’ and all the PPXs were encouraged to move to the ‘new style’ which was ppxlib: An Update on the State of the PPX Ecosystem and `ppxlib`'s Transition

Based on what I’ve read, ppxlib was supposed to be a better solution to the incompatible AST problem. But to be honest, I don’t understand PPX all that well.

20

I don’t think there’s a need to employ such a tone here. Please keep the conversation nice for everyone.

13 Likes
21

Would you mind to be a little more specific how this testing would work in this case?