Which OCaml compiler versions should we run against in CI?

Ecosystem
,
1

Which ocaml compiler versions should a responsible project aiming to maintain backwards compatibility include in its CI build matrix?

I have seen that testing against 4.04.2 is the standard minimum version, but I’m not sure which subsequent versions we ought to test against. All of them? A special subset? What would be a good source of truth for tracking this in the future?

1 Like
2

I try to lower the version as much as possible, typically to 4.02.3, 4.03.0, or 4.04.2, depending on the project. The more “fundamental” a project, the lower the lower bound should be. For example, Lwt is tested (or was?) against 4.02.3. Bisect_ppx, because it is an optional dev tool, is now tested against 4.04.2 — although it supports 4.02.3, and was tested against it until some of its test dependencies became difficult to use with 4.02.3. It’s fine to test Bisect against less versions, because a Bisect user can usually themselves choose a higher OCaml version build row to run Bisect in, in their own CI.

Then, for most projects, I just test against the latest version of OCaml, to “span” the version axis as quickly as possible. This is true in Luv and Dream. However, for projects that have conditional compilation that is triggered at specific version changes, such as Lwt and Bisect, we test every major OCaml version in between, to make sure that if we edit conditional code, we don’t get the version boundaries wrong.

When using Travis in the past, I would add intermediate versions between oldest and latest to allow_failures. Travis would then prioritize running the oldest and latest version, giving us a fast approximate answer.

I would then add one macOS row on latest OCaml, spanning the Linux-macOS axis. Some projects that don’t feasibly have differences on macOS, like Lambda Soup, don’t have macOS rows.

Projects that are packaged for multiple build systems or targets (JS), like Bisect and Promise, also have one extra row per (build system * OS), on latest OCaml for each build system.

5 Likes
3

Thank you, @antron! This exactly the guidance I needed and is super helpful.

1 Like