[ANN] Compatibility packages for 4.12 (either, semaphore-compat)

[CraigFe]

I’m pleased to announce the release of two small compatibility libraries for modules added in OCaml 4.12.0:

• either – compatibility shims for the Either module, which provides a canonical sum type in the standard library + various utility functions for it. To use it: simply add either as a library dependency and refer to the Either module, which will either alias the Either provided by the standard library or re-implement it as appropriate.

• semaphore-compat – compatibility shims for the Semaphore module in systhreads, which provides binary and counting semaphores for use in multi-threaded programs. To use it: add semaphore-compat as a library dependency and open Semaphore_compat in any module that requires semaphores.

  Note on OCaml concurrency primitives. Users of OCaml’s Mutex module should beware that OCaml 4.12 features a changes to mutex semantics on certain platforms, since the mutexes are now “error checking” (as noted in the changelog). One consequence of this is that unlocking a mutex from a non-owning thread will now always fail, whereas previously this might succeed depending on the platform’s mutex implementation – notably, glibc's mutexes allow this. If your code relies on this property of pre-4.12 mutexes, you may wish to add a dependency on semaphore-compat and switch to using binary semaphores instead (as these provide the right flavour of concurrency primitive in a POSIX-compatible manner).

Opam library authors making use of these compatibility libraries in their public API are encouraged to conditionally depend on them in order to ensure that downstream users of your library don’t pull in unnecessary compatibility shims. This can be done as follows:

depends: [
  ("ocaml" {>= "4.12.0"} | "either")
]

I hope you find these libraries useful!

[hannes]

Thanks for your work on this.

@CraigFe

are encouraged to conditionally depend

While I understand at the opam level how this could work, what about _tags or dune – there’s need to repeat that condition (since in OCaml < 4.12 you’ll need (libraries either) in your dune file / *: package(either) in your _tags file (+in your META file) – or is there something obvious I misunderstand?

I’m asking since I’d be interested in a general strategy / howto of such compatibility libraries. I’m also curious why there are different approaches in respect to package name and strategy:

• stdlib-shims (seems to (attempt to) cover all stdlib changes?)
• bigarray-compat (there’s a -compat suffix)
• seq (there’s a seq.base for ocaml >= 4.07.0)

Are there plans to unify the approaches? I’m in favour of a “conditional dependency if the OCaml distribution does not yet provide the module” and “no dependency if the OCaml distribution provides that module” – but as expressed above, I think this is needed both at the opam and the ocamlfind / library level.

[CraigFe]

To begin, I’ll fill out my reasons for recommending conditional dependencies somewhat.

Compatibility libraries and unconditional dependencies

The issue with unconditionally depending on compatibility libraries is that it forces users to take on that implicit dependency – even if they don’t need it – and ties that decision to the current minimal version supported by one of their dependencies. This prevents use of (implicit_transitive_deps false) in those downstream libraries, since the authors cannot know whether they actually depend on the alias transitively. For instance:

• lwt currently depends on result unconditionally (to retain support for 4.02.0);

• as an author of irmin – which goes back to 4.08.0 and so always has access to Stdlib.Result – I must be able to resolve the Lwt_result.t type, but I don’t in general know which dependencies are necessary to do that since it’s a property of Lwt's code and not mine.

• I must either add an explicit dependency on result – because that’s what’s currently necessary given Lwt's own dependencies – or give up on using (implicit_transitive_deps false) entirely. Either way, if I myself expose a result type in the API of Irmin, this problem cascades to users of Irmin too.

For Lwt specifically, I need to find the time to upstream a patch to fix the issue: https://github.com/ocsigen/lwt/issues/794.

Conditional dependencies and Dune

It’s possible to conditionally depend on a library on the Dune side via either a (select ... from ...) dependency or by generating Dune files programatically (either w/ (include dune.inc) or Jbuild_plugin). None of these solutions are particularly compelling to me, but I’ve been bitten by this problem enough to be willing to consider ugly solutions – at least for very widely-used libraries such as Alcotest.

It’s possible that conditional dependencies are the wrong approach altogether and we should be using something like (re_exports) instead. I’m not familiar with that feature, but it looks like it might work to avoid the above issues. Perhaps a Dune maintainer could weigh in here, and I’ll happily remove my suggestion re. conditional dependencies for something better

Different approaches to compatibility libraries

@hannes

I’m asking since I’d be interested in a general strategy / howto of such compatibility libraries. I’m also curious why there are different approaches in respect to package name and strategy:

• stdlib-shims (seems to (attempt to) cover all stdlib changes?)
• bigarray-compat (there’s a -compat suffix)
• seq (there’s a seq.base for ocaml >= 4.07.0)

Indeed, there are quite a few different mechanisms used for this. Even just for this release, either and semaphore-compat use different techniques! Partly this is because it’s easier to shim modules in Stdlib since it is the default-opened library. In particular, Either can exist without a namespace collision, and the implementation of either can refer to Stdlib.Either to avoid Dune getting confused about a self-referential module. This is the approach that was taken with the result compatability package too.

Unfortunately, the above factors don’t apply to Semaphore, which is why I grudgingly settled on semaphore-compat instead. I’m not sure why the Bigarray compat module didn’t take the more “direct” approach, but I was happy to be consistent with it.

One final point: the stdlib-shims library exists solely to provide the alias Stdlib ↦ Pervasives in order to compensate for the name change. It doesn’t ever provide re-implementations of the modules and functions that were added since (and so really is a -shim and not a -compat). A user of stdlib-shims is still restricted to the set of stdlib functions provided by their minimal supported OCaml version.