# Basic GitLab CI configuration

**URL:** <https://discuss.ocaml.org/t/basic-gitlab-ci-configuration/3327>\
**Category:** Ecosystem\
**Tags:** ci\
**Created:** [February 12, 2019, 12:08am UTC](https://discuss.ocaml.org/t/basic-gitlab-ci-configuration/3327 "2019-02-12T00:08:41Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![gasche](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/gasche/32/4_2.png) [@gasche](https://discuss.ocaml.org/u/gasche)\
**Post date:** [January 22, 2021, 2:32pm UTC](https://discuss.ocaml.org/t/basic-gitlab-ci-configuration/3327/21 "2021-01-22T14:32:45Z")

</div>

I wish that some of the heavy-ocaml-gitlab-users would team up to write a proper base setup for Gitlab CI for OCaml projects, that people could easily reuse and extend when they do their own project.

---

<div class="post-metadata">

**Author:** ![gasche](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/gasche/32/4_2.png) [@gasche](https://discuss.ocaml.org/u/gasche)\
**Post date:** [January 22, 2021, 2:58pm UTC](https://discuss.ocaml.org/t/basic-gitlab-ci-configuration/3327/22 "2021-01-22T14:58:47Z")

</div>

I think I initially misunderstood the caching used by @cemerick in [basic OCaml gitlab CI config · GitHub](https://gist.github.com/cemerick/76664c4ebfdca20d936de117024d6ed8) : the cache is keyed on $CI\_COMMIT\_REF\_SLUG; if I understand correctly this means that the cache is only maintained for several actions running on the same commit (possibly across several branches), and not from one commit to the next as I had in mind.

---

<div class="post-metadata">

**Author:** ![gasche](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/gasche/32/4_2.png) [@gasche](https://discuss.ocaml.org/u/gasche)\
**Post date:** [January 22, 2021, 5:23pm UTC](https://discuss.ocaml.org/t/basic-gitlab-ci-configuration/3327/23 "2021-01-22T17:23:40Z")

</div>

Gitlab maintains a list of [CI template files](https://docs.gitlab.com/ee/ci/examples/README.html#cicd-templates) for many programming languages, we could contribute one for OCaml.

---

<div class="post-metadata">

**Author:** ![cemerick](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/cemerick/32/1383_2.png) [@cemerick](https://discuss.ocaml.org/u/cemerick)\
**Post date:** [January 22, 2021, 7:44pm UTC](https://discuss.ocaml.org/t/basic-gitlab-ci-configuration/3327/24 "2021-01-22T19:44:16Z")

</div>

> [@gasche](#):
>
> I think I initially misunderstood the caching used by @cemerick in [basic OCaml gitlab CI config · GitHub](https://gist.github.com/cemerick/76664c4ebfdca20d936de117024d6ed8) : the cache is keyed on $CI\_COMMIT\_REF\_SLUG; if I understand correctly this means that the cache is only maintained for several actions running on the same commit (possibly across several branches), and not from one commit to the next as I had in mind.

That’s fortunately not the case; `$CI_COMMIT_REF_SLUG` is the current git branch/ref being built, normalized according to certain rules. You can see examples of all of the variables present in gitlab CI runs (including this one) here: [GitLab CI/CD variables | GitLab](https://docs.gitlab.com/ee/ci/variables/)

> [@gasche](#):
>
> If I understand correctly, the best choice in this thread is the [second version](https://discuss.ocaml.org/u/cemerick) of @cemerick (the gist is outdated and should not be used).

I suppose it’s lagged w.r.t. OCaml version, but otherwise should work well. FWIW, this is representative of what I’ve been using for ~a year now:

```yaml
test:
  stage: test
  only:
    - branches
  cache:
    key: "$CI_COMMIT_REF_SLUG"
    paths:
      - _opam
      - node_modules
  script:
    - '[! -d _opam] || (rm -rf ~/.opam/4.08 && mv _opam ~/.opam/4.08)'
    - opam switch 4.08
    - eval $(opam env)
    - opam switch
    - if ! opam install --deps-only --locked -t -y . ; then opam repository add remote https://opam.ocaml.org && opam update && opam install --deps-only --locked -t -y .; fi
    - npm install
    - dune runtest
    - mv ~/.opam/4.08 _opam

```

(The opam install/repository dance is there so that newer libraries that aren’t included in my base docker image can be picked up.)

I’ll be bumping the OCaml version across my key projects at some point in the next month or so once I finish the current milestone. I’ll see about either posting any updates here, or maybe polishing something for inclusion in the CI template file repo.

---

<div class="post-metadata">

**Author:** ![gasche](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/gasche/32/4_2.png) [@gasche](https://discuss.ocaml.org/u/gasche)\
**Post date:** [January 24, 2021, 1:12am UTC](https://discuss.ocaml.org/t/basic-gitlab-ci-configuration/3327/25 "2021-01-24T01:12:13Z")

</div>

After a long ci-golfing adventure (83 tests), I got a `.gitlab-ci.yml` file that I think is reusable and useful for small projects / libraries:

- project: [https://gitlab.com/gasche/gitlab-ocaml-ci-example](https://gitlab.com/gasche/gitlab-ocaml-ci-example)
- configuration file: [https://gitlab.com/gasche/gitlab-ocaml-ci-example/-/blob/main/.gitlab-ci.yml](https://gitlab.com/gasche/gitlab-ocaml-ci-example/-/blob/main/.gitlab-ci.yml)

Features:

- It is project-agnostic, so it should work unchanged for your own (simple) projects.
- It caches the opam dependencies.
- It builds the project, runs the tests and builds the documentation.
- Several compiler versions can be tested in parallel.
- It provides an easy way to upload the documentation as “Gitlab project Pages”.

CI times are satisfying: on very small libraries I observe a 11mn job time on the first run (or when cleaning the opam cache), and 2mn job time on following runs.

The expected usage-mode of this CI configuration is that you copy it in your own project. If you find that you need/want additional features, ideally you would try to write them in a project-agonistic way and contribute them back to the example repository.

This configuration does not use @smondet’s trick of generating a docker image on the fly. I think this would be an excellent idea to get more reliable caching, but it is too complex for me and I don’t see how to do it in a maintainable and project-agnostic way.

### Current status

I wrote this CI configuration over the week-end, and have not used it much. I expect it to keep evolving somewhat before it stabilizes. Feedback from other people trying to use the configuration would be warmly welcome.

### Aside on `_build` caching

I also implemented caching of dune’s `_build` data, inspired by the [data-encoding](https://gitlab.com/nomadic-labs/data-encoding/-/blob/master/.gitlab-ci.yml) example of @raphael-proust. I don’t need it for my small projects (dune build is 3s, compared to 1m setting up the Docker image), but I thought it would make the CI configuration scale better to larger projects.

When I tested this CI configuration, I discovered that caching the dune `_build` data does not work as well as I had expected. (Tracking issue: [dune#4150](https://github.com/ocaml/dune/issues/4150)).

I can tell because I am asking dune to tell me about what it is rebuilding (`dune build --display short`). I suspect that projects that cache the `_build` data _without_ logging what dune (re)builds are also not caching as much as they think they are.

(But then maybe the use of a fixed-compiler OPAM image, as data-encoding is using, solves the issue.)

### official CI template?

I considered submitting this CI configuration as an “OCaml Gitlab CI template” to go with the official list of “blessed” CI templates in [the documentation](https://docs.gitlab.com/ee/ci/examples/README.html#cicd-templates). But reading the [Development guide for Gitlab CI/CD templates](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/cicd/templates.md) convinced me that my CI configuration is nowhere ready to serve this role.

Gitlab developers apparently expect that users will be able to “include” those CI templates by pointing to their URL, and then tune it for their own use-case (without modifying it) by performing some (unreasonable?) inheritance tricks using whatever those configurations offers as abstraction/inheritance/extension/overriding mechanism. Let’s just say that this is next-level CI configuration writing, and that my script is not ready for this.

---

<div class="post-metadata">

**Author:** ![raphael-proust](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/raphael-proust/32/415_2.png) [@raphael-proust](https://discuss.ocaml.org/u/raphael-proust)\
**Post date:** [January 24, 2021, 9:41am UTC](https://discuss.ocaml.org/t/basic-gitlab-ci-configuration/3327/26 "2021-01-24T09:41:06Z")

</div>

That config file looks great. I’ll probably be porting some of the libraries under the Nomadic Labs umbrella to use that.

I’d recommend:

- Making doc building optional (by detecting `with-doc` dependency in opam file or through the use of a variable).
- Adding a step to check formatting (using `dune build @fmt`) (optional step, controlled through a variable, or through a `{with-test}` dependency to `ocamlformat` in the opam file).
- Add a URL to the artifacts (using the `PIPELINE` variable to generate the correct URL) in the message about checking the artifacts.

I’ll try to add those soon if it hasn’t been done when I get around to it.

---

<div class="post-metadata">

**Author:** ![gasche](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/gasche/32/4_2.png) [@gasche](https://discuss.ocaml.org/u/gasche)\
**Post date:** [January 24, 2021, 11:11am UTC](https://discuss.ocaml.org/t/basic-gitlab-ci-configuration/3327/27 "2021-01-24T11:11:43Z")

</div>

Thanks!

I’m not sure how to make steps “optional” by checking opam files: there may be several opam packages developed in the same git repository, and I would expect all of them to receive CI build/testing together. I just tried and `dune runtest` is a no-op if no tests are around, so there is no worry about failures in this case. Do you have in mind a scenario when we don’t want to run all tests as part of the CI, but only some tests (eg. “fast” tests?). Ideally the CI configuration would have a variable that let us describe what to test (everything by default, nothing if the variable says so, but also a subset if there is a corresponding way to tell dune about that). I never needed to specify only a subset of tests to run in Dune, so I would need to go check the documentation about that.

(I also have no experience with `dune build @fmt`.)

The PIPELINE suggestion is excellent, thanks.

---

<div class="post-metadata">

**Author:** ![raphael-proust](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/raphael-proust/32/415_2.png) [@raphael-proust](https://discuss.ocaml.org/u/raphael-proust)\
**Post date:** [January 24, 2021, 12:10pm UTC](https://discuss.ocaml.org/t/basic-gitlab-ci-configuration/3327/28 "2021-01-24T12:10:04Z")

</div>

The `@fmt` target for `dune` is available if the correct stanza is present in your `dune-project` file: `(using fmt <version>)`.

The `dune build @fmt` command will return a non-zero exit code if some files are detected to not be formatted correctly. In interactive use you then issue `dune promote` command to get the formatting fixes on your working files. But in CI you can just rely on the exit code. (It also prints the diff of formatting on its output so it’s fairly obvious what went wrong.)

The main issues are:

- How to automatically detect whether we can run this? (That’d probably be from the `dune-project`.)
- How to automatically detect what version of `ocamlformat` is needed? (That’d probably be from the `.ocamlformat` config file.)
- How to allow users to disable it if they don’t want it running on the CI?
- How to deal with multiple-project single-repo?

---

<div class="post-metadata">

**Author:** ![gasche](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/gasche/32/4_2.png) [@gasche](https://discuss.ocaml.org/u/gasche)\
**Post date:** [January 24, 2021, 2:24pm UTC](https://discuss.ocaml.org/t/basic-gitlab-ci-configuration/3327/29 "2021-01-24T14:24:25Z")

</div>

According to [Dune’s documentation on formatting](https://dune.readthedocs.io/en/stable/formatting.html), starting with `(dune-lang 2.0)` formatting is enabled, but ocamlformat requires an `.ocamlformat` file. I would thus:

- skip the `dune run @fmt` step if no `.ocamlformat` file is present at the root
- provide a CI variable to let users define the formatting taget (default `@fmt`), which lets them  
for example use weird stuff like `@subdir/fmt`, with the convention that an empty value means that formatting is disabled in the CI. This approach, “target alias or empty”, also works for `@runtest` and `@doc`.

I don’t think we need to do any work to guess the version of `ocamlformat` to run: this should be a `{dev}` depedency of the `opam` file, so it will have been built during the build-dependencies stage.

(Note: this discussion could also have happened on the [gitlab-ocaml-ci-example](https://gitlab.com/gasche/gitlab-ocaml-ci-example/) issue tracker.)

---

<div class="post-metadata">

**Author:** ![gasche](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/gasche/32/4_2.png) [@gasche](https://discuss.ocaml.org/u/gasche)\
**Post date:** [January 25, 2021, 3:36pm UTC](https://discuss.ocaml.org/t/basic-gitlab-ci-configuration/3327/30 "2021-01-25T15:36:32Z")

</div>

@raphael-proust I implemented the following of your suggestions:

- show artifact URLs directly in the logs
- configurable targets for the build, test, doc stages (in particular, an empty variable skips the corresponding step)

I made some progress the `_build` caching issue, thanks to a tip from (my informer) @ejgallego who had the exact same issue for Coq. The permissions of some files are changed when restored from cache. Unfortunately the only way I can reproduce the issue is by using `dune` to build a problematic directory, any attempt at a simpler reproduction has failed so far. But this does seem to be a Gitlab issue.

---

<div class="post-metadata">

**Author:** ![gasche](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/gasche/32/4_2.png) [@gasche](https://discuss.ocaml.org/u/gasche)\
**Post date:** [January 26, 2021, 11:12am UTC](https://discuss.ocaml.org/t/basic-gitlab-ci-configuration/3327/31 "2021-01-26T11:12:48Z")

</div>

The issue with dune `_build` caching is caused by a bug in gitlab-runner, which does not preserve file permissions for the targets of symlinks; I reported a bug upstream at [https://gitlab.com/gitlab-org/gitlab-runner/-/issues/27496](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/27496). In the meantime we will have to work around this bug by caching not the `_build` directory itself, but a tar. I may not have managed to locate the issue without a helpful tip from @ejgallego who ran into the same issue with the Coq CI.

(It is also possible that this bug would be problematic for `_opam` caching, if we had packages within opam that installed (non-executable) files using symlinks. I don’t know of any package doing that, so I think the issue can be ignored for now, but please comment if you know of such a case.)

Edit: a Gitlab developer suggested a workaround (opt-in to an alternative `zip` implementation), so the problem is now solved (but the limitations of dune that made debugging too difficult remain). In case anyone is curious, the root cause of the bug is the frugality of Go’s `archive/zip` standard library package, which encodurages consumers to implement their own zip-file-extraction logic independently, resulting in buggy and often-insecure code.

---

<div class="post-metadata">

**Author:** ![lefessan](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/lefessan/32/2293_2.png) [@lefessan](https://discuss.ocaml.org/u/lefessan)\
**Post date:** [February 1, 2021, 3:32pm UTC](https://discuss.ocaml.org/t/basic-gitlab-ci-configuration/3327/32 "2021-02-01T15:32:37Z")

</div>

Hi @gasche  
I just discovered your project. Why not try to include it in `drom` ? Currently, `drom` only targets Github, which is an issue as many projects prefer Gitlab. So, if you feel like contributing it, we can help 🙂

---

<div class="post-metadata">

**Author:** ![gasche](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/gasche/32/4_2.png) [@gasche](https://discuss.ocaml.org/u/gasche)\
**Post date:** [February 1, 2021, 3:48pm UTC](https://discuss.ocaml.org/t/basic-gitlab-ci-configuration/3327/33 "2021-02-01T15:48:32Z")

</div>

After using it in “the wild”, I’m not satisfied with the “copy this large script and then hack on it randomly” model of the first version I released. My short-to-medium-term plan is to make the script “include-able by URL”, so that people can keep a short `.gitlab-ci.yml` script that points to the main script and tweaks a few variables of interest. See the tracking issue [gitlab-ocaml-ci-example#2](https://gitlab.com/gasche/gitlab-ocaml-ci-example/-/issues/2).

(I timed out on gitlab-CI hacking for a week or two, so this will not get done immediately.)

I think that it would be much easier to add Gitlab-CI support in project-starter tools such as `drom` after the script is reworked to be included by URL. For now, I am mostly hoping for feedback from established/experienced Gitlab users, that could try to adopt the script on their project (as a form of beta-testing) and report bugs or important missing features.

---

<div class="post-metadata">

**Author:** ![AltGr](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/altgr/32/4220_2.png) [@AltGr](https://discuss.ocaml.org/u/AltGr)\
**Post date:** [February 1, 2021, 3:57pm UTC](https://discuss.ocaml.org/t/basic-gitlab-ci-configuration/3327/34 "2021-02-01T15:57:02Z")

</div>

Ha, good work @gasche ! I too have been defining my own gitlab CI scripts some time ago, so here are my 2 cents.  
This is very light-weight and quite bare-bones, but doesn’t rely on dune and may be fine for small projects.

```auto
build:
  image: ocamlpro/ocaml:4.10
  before_script:
    - "[-d _opam] || opam switch create . ocaml-system --deps-only --locked"
  cache:
    paths:
      - _opam
    key:
      files:
        - foo.opam.locked
  script:
    - opam exec -- make all
    - opam exec -- make test

```

This uses our [light-weight alpine images](https://hub.docker.com/r/ocamlpro/ocaml) with the beta of opam 2.1, which makes e.g. external dependencies handling much easier to handle. The image provides a system compiler at the right version, which avoids recompilations, and the dependencies are cached as long as you don’t change your `.opam.locked` file.

Of course, this is much more limited in many ways, in particular it’s just for one version of OCaml at a time.

Also I agree on the `include` workflow for Gitlab CI, it’s pretty nice.  
EDIT: forgot to add, I like the `include` workflow better combined with `extends` though.

---

<div class="post-metadata">

**Author:** ![gasche](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/gasche/32/4_2.png) [@gasche](https://discuss.ocaml.org/u/gasche)\
**Post date:** [February 1, 2021, 5:57pm UTC](https://discuss.ocaml.org/t/basic-gitlab-ci-configuration/3327/35 "2021-02-01T17:57:46Z")

</div>

> doesn’t rely on dune

Ah, you are relying on this shiny build system called `make`, but I don’t have a Makefile in my small projects of interest. (Maybe it’s a good practice to keep them for friendliness to non-OCaml-users?)

The script is nicely minimalistic, and indeed opam 2.1 makes depext handling much nicer. I prefer to not use 2.1 yet, to keep an environment that is close to what the users have at home, if they want to debug the script.

In my tests there is a large startup-time difference betwen the `debian-testing-opam` image I use for the build and the `alpine:latest` image I use for Pages deployment (about 1m20s vs. 20s). I guess I should to test alpine-opam images to see if I can reduce the load time a bit (assuming `opam depext` handles the alpine package manager correctly).

---

<div class="post-metadata">

**Author:** ![AltGr](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/altgr/32/4220_2.png) [@AltGr](https://discuss.ocaml.org/u/AltGr)\
**Post date:** [February 2, 2021, 10:16am UTC](https://discuss.ocaml.org/t/basic-gitlab-ci-configuration/3327/36 "2021-02-02T10:16:09Z")

</div>

> Ah, you are relying on this shiny build system called `make`

Haha, that’s just an example, the point is that the script is so simple it should be trivial to customise for a given use-case; a more generic command could have been `opam install .`, but I wanted to hilight that you can use your own commands without going through opam. You’ll also need to specify your opam files for proper caching, unfortunately (`foo.opam` / `foo.opam.locked` …)

Alpine has been used for CI for a while, so it should be in the top-tier of supported distros for depext handling.

---

<div class="post-metadata">

**Author:** ![raphael-proust](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/raphael-proust/32/415_2.png) [@raphael-proust](https://discuss.ocaml.org/u/raphael-proust)\
**Post date:** [February 4, 2021, 5:24pm UTC](https://discuss.ocaml.org/t/basic-gitlab-ci-configuration/3327/37 "2021-02-04T17:24:13Z")

</div>

I’ve just used the yaml via copy-pasting on two projects and it just works. I’ll use the open MRs as opportunities to tweak things out, but it’s already much better than the homebrew stuff I had cooked up and reheated whenever I needed a new feature/test.

---

<div class="post-metadata">

**Author:** ![Kakadu](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/kakadu/32/1204_2.png) [@Kakadu](https://discuss.ocaml.org/u/Kakadu)\
**Post date:** [February 13, 2021, 9:57am UTC](https://discuss.ocaml.org/t/basic-gitlab-ci-configuration/3327/38 "2021-02-13T09:57:12Z")

</div>

Sorry for offtopic, but do you have a link to docker-based instructions for GitHub?

---

<div class="post-metadata">

**Author:** ![raphael-proust](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/raphael-proust/32/415_2.png) [@raphael-proust](https://discuss.ocaml.org/u/raphael-proust)\
**Post date:** [February 13, 2021, 2:54pm UTC](https://discuss.ocaml.org/t/basic-gitlab-ci-configuration/3327/39 "2021-02-13T14:54:31Z")

</div>

Not sure what you mean by Docker-based instructions.

You can have a look at the Tezos project which does interesting things with Docker for its CI:

- [Tezos / opam-repository · GitLab](https://gitlab.com/tezos/opam-repository/) generates a Docker image with already installed opam dependencies. It actually generates a collection of them: one with only the runtime dependencies for packaging the binaries, one with runtime+build deps for building the binaries, one with runtime+build+test dependencies for running the tests.
- [Tezos / tezos · GitLab](https://gitlab.com/tezos/tezos) pulls the different images for running the different parts of the CI.

It’s definitely over-engineered for small projects, but for bigger projects like Tezos it works quite well. Notably:

- it avoids rebuilding all the opam dependencies,
- it allows fine control over what is included in the released images,
- it allows fine control over the exact versions of all the opam packages that are used as dependencies,
- it makes it relatively easy to include non-opam/non-OCaml dependencies (e.g., python dependencies for tests and doc).

Do you have a more specific query? What kind/size of project are you looking at?

---

<div class="post-metadata">

**Author:** ![Kakadu](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/kakadu/32/1204_2.png) [@Kakadu](https://discuss.ocaml.org/u/Kakadu)\
**Post date:** [February 13, 2021, 3:00pm UTC](https://discuss.ocaml.org/t/basic-gitlab-ci-configuration/3327/40 "2021-02-13T15:00:06Z")

</div>

@raphael-proust I meant Github, not Gitlab, so I marked question as off topic.  
At the moment github action for OCaml made by avsm is terrible because it can’t do caching. And I’m looking for the ways to use GitHub actions + Docker image. Maybe somebody already did that. …

[Previous page](https://discuss.ocaml.org/t/basic-gitlab-ci-configuration/3327.md?page=1)