What are the best practices for setting up continuous integration for OCaml projects?

ci

#1

I’d like to gather feedback and perhaps help curate documentation on what are the best practices for setting up CI for OCaml projects. Pardon my naivety in this area, but I figured that asking the experts and following a good path would save me (and hopefully others) a lot of work.

I don’t have as many OCaml projects as many of the other developers on this forum, but even with one project I find my current setup and constraints stifling. I’m currently using ocaml-ci-scripts’s travis-opam.sh, but with a long delay and 10 minute+ wait times on several tests it is cumbersome. On top of that, recently, the failures are entirely unrelated to my code; failures like opam not being available. I know that travis supports containerized builds, but I don’t understand how to use travis-docker.sh. My specific questions are:

  1. Is there a way to test on a container with a fixed opam environment (hopefully, with the relevant packages installed) such that when I push, Travis only tests based on the newest code? It is wasteful to repeatedly test the installation of opam and my projects dependencies during development. I’d like to fix my projects dependencies and upgrade them as I individually review newer versions as opposed to riding the wave of latest packages.
  2. Barring this (my personal) ideal setup, what is the next best way?
  3. Is Travis a good CI platform and are there better ones ? I’m unfortunately a bit wedded to Github for this project as I’ve heard good things about Gitlab. I do find Travis’s matrix functionality useful for dividing the testing.

Thank you in advance.


#2

Sadly I don’t have an answer to offer, but I’m also very interested in this. Travis is now unusable (systematic errors) and was always very slow. I think it’d be super nice if someone™ could write a blogpost/documentation page/readme that explains clearly how the travis-docker script works, and how to tinker with it. :innocent:


#3

I utilize Docker to run tests for my (unpublished) ezpostgresql lib. I don’t use travis-docker.sh (must have missed that by then). My CI runs in 2,5 minutes on average with few tests, but all those time are used to build the image and install dependencies. The actual test run is subsecond, so I imagine it would scale to number of tests nicely. Certainly better than the 10+ minutes I also encountered when trying to use travis-opam.sh.

If you want to try it, here are the interesting files that enabled it:

  1. Dockerfile.test
  2. docker-compose.yml (my tests use postgres so I use a container for the sake of simplicity)
  3. .travis.yml

This works great for me. I only tested on one base alpine image so far but I think you would be able to utilize Travis’ matrix to achieve it (perhaps multiple Dockerfiles?). Feel free to ask anything or suggest improvements!


#4

Sadly I don’t have an answer to offer

I actually tried to mirror your nice setup for sequence. But ran into an opam stack overflow error, presumably because my repo doesn’t have your magical cache and there’s no external solver (aspucd) installed from the PPA for an older version of opam that overflows.


#5

My setup comes from @zozozo, he’s the expert :slight_smile:
The cache is nice, but now we should really strive to use docker rather than opam switching all the time…


#6

Indeed, I also saw stack overflow failures in travis recently when using avsm’s opam package. My current solution has been to instead download the opam2 beta binaries, which work quite nicely (all the more since the solver is now bundled in opam2 directly). If you want, you can take a look a my current travis script.

I’m not yet sure whether using the cache feature is really interesting currently, since I’m testing on quite a lot of different versions of ocaml and I fear that the cache may just be too big to really speed up builds. Another reason not to use the cache is that in my latest script I test installation of the package with, and without its test dependencies (to check that no deps are missing), and in that case, using the cache may silently hide errors if, for instance, a dependency was flagged as test only but was actually needed to install the package (let’s say because of a mistake in build system configuration).


#7

For the XenServer project, we maintain a set of upstream libraries and our own components in one Opam repository xs-opam. This repository we build with Travis. The setup could use some polish but it achieves the following:

  • We control exactly the versions of the upstream libraries we are using. These packages live in packages/upstream.
  • We control exactly the versions of our own libraries that we are using by referring to tags in their repositories: packages/xs.
  • We build our own component packages by referring to the master branch of their repositories: packages/xs-extra.

The repository also contains a Dockerfile to build everything locally. All in all this is an example for the continuous integration of a project that is composed of executable components and libraries and might be bigger than what you were looking for. Internally we have another CI system that also builds the rest of the XenServer distribution which is not implemented in OCaml. We use Travis mostly to run it on pull requests and not as the ultimate test.


#8

I wanted to thank everyone for responding. I had a couple of quick thoughts:

I’m impressed by @bobbypriambodo setup, but I’ve noticed that the containerized-Trusty environment does startup faster so I wanted to use it. It seems super odd that only in a non-containerized environment you can use docker to create custom containers. This reasons for this requirement are not-obvious to me.

I’m also impressed the XenServer’s project, but yes that is beyond my requirements.

I ended up using @zozozo’s setup with wget’ing opam2 bin and using Travis’s cache to store ~/.opam. I fix the specific dependencies of my project and cutting out the repeated compilation of big projects like ocamlnet and core_kernel cut my total test time by 50%+.

Thanks again.


#9

If you’re using the .travis.yml file I linked, you should take a look at this recent commit, which ensures that the script indeed tries to install the pinned version (assuming it is dev), and not some other version available on the public repository.


#10

I recently set up builds for the Reason project on CircleCI 2.0 and it’s working wonderfully. Their builds run across multiple versions of OCaml and finish in about 2 minutes if a cache is present.

You can follow along with their config here: https://github.com/facebook/reason/blob/master/.circleci/config.yml

Their build is fairly complicated. A simpler OCaml build would probably look something like this: https://gist.github.com/dwwoelfel/8d0a4f4d92219dd9b798830ec2437ee8

You can find more help in the Circle docs: https://circleci.com/docs/2.0/sample-config/


#11

I also use CircleCI for several OCaml projects and it’s been working well for me:

https://github.com/brendanlong/ocaml-ooxml/blob/master/.circleci/config.yml
https://github.com/arenadotio/pgx/blob/master/.circleci/config.yml
https://github.com/brendanlong/ocaml-trie/blob/master/.circleci/config.yml

Without caching, builds using core take about 3 minutes (with most of the time involving installing dependencies):

https://circleci.com/gh/arenadotio/pgx

With Docker builds you can save a base image to build off of to speed up future builds. I do that in this (non-OCaml) project that needs massive dependencies like WebKit: https://github.com/jangernert/FeedReader/tree/master/docker

The ~/.opam caching @dww mentioned now is probably almost as good though, and significantly less work (I’m going to set it up for my projects now).

EDIT: It turns out CircleCI’s cache is fairly slow, caching ~./opam was almost as slow as doing a full opam install. I ended up taking the cache back out since it’s not quite as good of a test and a 30-second speedup wasn’t worth it.


#12

I just saw a blog blog post where someone describes their steps to get CI working on Gitlab – but for julia. Does anyone have experience doing that with an OCaml project?


#13

Here you have an example:

https://gitlab.com/smondet/vecosek/blob/master/.gitlab-ci.yml
builds everything on a few ocaml versions + builds the documentation and puts it there: http://smondet.gitlab.io/vecosek/