Basic GitLab CI configuration

Ecosystem
1

I was looking for a basic gitlab CI configuration (e.g. suitable for internal libraries and end-user executables), but didn’t turn anything up.

This works well enough as a foundation, perhaps others will find it useful:

4 Likes
2

Just wondering about your use of caching there – you don’t seem to restore the _opam directory in the test phase, so does that work?

3

The same cache “stanza” appears in both jobs, so yes, the _opam directory is restored. If it weren’t, then my dune call would fail. :slight_smile:

(Looking now, I think I can actually just lift that out of the job specs since it’s the same across them all.)

4

(Oh, do make sure to click through to the full gist, discourse only inlines a portion.)

5

but what copies from the _opam directory back to the main /home/opam/opam-repository/.opam file? The copy only seems to go in one direction into the cache

6

My understanding of gitlab caches is that you only specify the included paths, which are restored prior to the job running, and stored after it finishes. There’s no notions of to or from afaik.

7

I might be missing something obvious, but what’s the point of saving the contents of the .opam directory into the _opam directory, but never using it? Doesn’t something need to copy from _opam back to /home/opam/.opam/4.07 in order to restore the cache back into the opam context, so that the subsequent opam install is fast?

1 Like
8

I’m new-ish to OCaml and opam myself, so perhaps I’m doing something foolish. However, here’s what happening:

  1. The build job (part of the build stage, perhaps not the greatest naming scheme :-P) first pulls any retained cache, copying in ./_opam. (Note that . is the project working dir, not ~ or whatever.)
  2. [ -d _opam ] || cp -r ~/.opam/4.07 _opam makes a new local switch from the canonical 4.07 switch, if it doesn’t exist yet. This is only relevant if the cache was dry.
  3. opam install --deps-only -t -y . does its thing, installing into that local switch.
  4. The cache is updated with the contents of ./_opam.
  5. The test job begins by pulling the retained cache, which puts that local switch in place again.
  6. We eval $(opam env) to twiddle our paths appropriately.
  7. I like having the output of opam switch available in the log, ¯\(ツ)/¯
  8. The tests get run.
  9. The caches are again updated, though this should be a no-op given what this test job does.

To answer your question:

No, I don’t want to copy out of _opam. AFAIK, gitlab caches only act on the context of the project working directory, so always operating on the local switch is essential. If I didn’t set up ./_opam, or created a new “global” switch, or otherwise modify things outside of ., that work would never be cached. This is why there’s only one opam install; the results of that process go into ./_opam, are cached at the end of the build job, and are restored at the beginning of the test job.

i.e. the output of that opam switch command in the test job looks like this:

$ opam switch
#   switch                 compiler                    description
->  /builds/org/myproject  ocaml-base-compiler.4.07.1  4.07
    4.02                   ocaml-base-compiler.4.02.3  4.02
    4.03                   ocaml-base-compiler.4.03.0  4.03
    4.04                   ocaml-base-compiler.4.04.2  4.04
    4.05                   ocaml-base-compiler.4.05.0  4.05
    4.06                   ocaml-base-compiler.4.06.1  4.06
    4.07                   ocaml-base-compiler.4.07.1  4.07

I hope that’s clarifying? If you or anyone else have suggestions on how to improve this (outside of dropping the duplicated cache entry), do tell. :slight_smile:

1 Like
9

Ahh, I missed the move from a global opam switch to _opam for a project-local one. Thanks for the detailed explanation!

One thing I’m not sure about is whether the cp -r ~/.opam/4.07 _opam is a sound way of creating the local switch (vs opam switch create . 4.07). Your way is certainly faster, but the compiler isn’t fully relocatable. However this will still work since the old path is still present in ~/.opam/4.07 as well so the stdlib can be found there. It may lead to some odd bugs though, so you might want to just do opam switch create to make the local switch to be totally safe.

10

Ah-ha! I originally tried to use opam switch, but was apparently using it wrong (didn’t include 4.07 or whatever), and got this error, which made no sense to me at the time:

ERROR] Could not resolve set of base packages:
        Your request can't be satisfied:
          - No available version of ocaml-base-compiler satisfies the constraints

It was only after reading a bit about how local switches worked that I thought to try the dumb copy, which got me moving along.

Can you clarify what you mean by “the compiler isn’t fully relocatable”? I’d obviously like to avoid placing landmines for myself, but it seems that the resulting environment is good, i.e. the local switch seems to entirely displace or at least come before any “global” switch:

opam@e8f4ee896928:/app$ opam env
OPAM_SWITCH_PREFIX='/app/_opam'; export OPAM_SWITCH_PREFIX;
CAML_LD_LIBRARY_PATH='/app/_opam/lib/stublibs:/home/opam/.opam/4.07/lib/ocaml/stublibs:/home/opam/.opam/4.07/lib/ocaml'; export CAML_LD_LIBRARY_PATH;
OCAML_TOPLEVEL_PATH='/app/_opam/lib/toplevel'; export OCAML_TOPLEVEL_PATH;
MANPATH=':/app/_opam/man'; export MANPATH;
PATH='/app/_opam/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'; export PATH;

opam@e8f4ee896928:/app$ opam config list

<><> Global opam variables ><><><><><><><><><><><><><><><><><><><><><><><><><><>
arch              x86_64           # Inferred from system
jobs              71               # The number of parallel jobs set up in opam configuration
make              make             # The 'make' command to use
opam-version      2.0.3            # The currently running opam version
os                linux            # Inferred from system
os-distribution   debian           # Inferred from system
os-family         debian           # Inferred from system
os-version        9                # Inferred from system
root              /home/opam/.opam # The current opam root directory
switch            /app             # The identifier of the current switch
sys-ocaml-version                  # OCaml version present on your system independently of opam, if any

<><> Configuration variables from the current switch ><><><><><><><><><><><><><>
prefix   /app/_opam
lib      /app/_opam/lib
bin      /app/_opam/bin
sbin     /app/_opam/sbin
share    /app/_opam/share
doc      /app/_opam/doc
etc      /app/_opam/etc
man      /app/_opam/man
toplevel /app/_opam/lib/toplevel
stublibs /app/_opam/lib/stublibs
user     opam
group    opam

Is your warning along the lines of “don’t rely on that not breaking later”?

11

This is my GitLab CI configuration, if it’s worth anything to anyone: https://gitlab.com/emelle/emelle/blob/master/.gitlab-ci.yml

12

I’ve now hit an “odd bug”, so you’ve been proven right here. :smile:

For example, any library that depends upon ocamlc -where during its build will fail to install properly. Specifically in my experience, num does this, which results in its artifacts being spread between the local and global switches, resulting in a variety of bad outcomes.

As for relocatability, I now see that the switch’s path is hardcoded in a bunch of different places within its built compilers (relevant to the num example, the absolute path to stdlib is defined in lib/ocaml/caml/s.h).

Since I still really don’t want to pay for an opam create as part of my CI build, I’m now always using the global switch, but I do move it in its entirety into my project working directory right at the end of a build stage. This makes it accessible to the caching mechanism. e.g.:

prep:
  stage: prep
  only:
    - branches
  cache:
    key: "$CI_COMMIT_REF_SLUG"
    paths:
      - _opam
      - node_modules
  script:
    - '[ ! -d _opam ] || (rm -rf ~/.opam/4.07 && mv _opam ~/.opam/4.07)'
    - eval $(opam env)
    - opam switch
    - opam install --deps-only -t -y .
    - mv ~/.opam/4.07 _opam
13

Note that this bit in particular you can actually redefine via the OCAMLLIB environment variable. Try e.g.:

OCAMLLIB=bla ocamlc -config
14

Thanks, noted. Though I guess now that I’ve been burned once by my taking shady shortcuts w.r.t. local switches, I’m twice shy to dive into those waters again. :wink:

15

That’s certainly wise, since bytecode programs will break in general if you do this. I think the hard-coding of path to the ocamlrun executable in bytecode executables is one of the last stumbling blocks for relocation. But the issue about this is suspended.

1 Like
16

That’s a shame. I see that the latest activity on that issue is actually entirely parallel to the subject in this thread:

At the moment we need to compile OCaml for each CI run, which takes ~10 minutes in addition to the other stuff.

It appears that my switch-caching gymnastics above are working around OK, but it’d of course be even better if creating local switches were easy and inexpensive.

1 Like
17

I’m interested in getting a base Gitlab CI configuration as well, with the following components:

  • caching support
  • CI, ideally with support for a family of OCaml versions
  • publishing documentation as Gitlab pages

If I understand correctly, the best choice in this thread is the second version of @cemerick (the gist is outdated and should not be used). @TheAspiringHacker has another recipe with no caching of the project dependencies, a weirdly detailed build script, but the documentation deployed to Gitlab pages.

1 Like
18

This project hits 2/3 of your goals: .gitlab-ci.yml · master · Nomadic Labs / data-encoding · GitLab

It doesn’t support multiple ocaml versions. I’d be interested in a solution for that too.

1 Like
19

@raphael-proust thanks! I like the use of the extends feature to share build configuration between actions, and I think it could be adapted to have several build actions for several OCaml versions.

In your project however, if I understand correctly you are only caching the local _build directory populated by the build system. I would like to cache the OCaml environment around it (opam + dependencies, including testing and documentation dependencies), which I suspect is taking more time/energy than actually rebuilding the project. This suggests that @cemerick approach is a better fit – but of course the two could be used together.

20

It’s hacky (and a bit manual because gitlab-ci is not a dependency monad :)) but
this project https://gitlab.com/tezos/flextesa/blob/master/.gitlab-ci.yml#L16
caches the dependencies by generating a “fat” docker image that has them already
installed.

1 Like