How to setup local OPAM mirror

Learning
1

Is it possible to mirror opam repository - both GitHub repository and distribution files somehow? I need it for a local docker builder machine, because invoking installation every time takes too much time during the downloading step. I found the Opam local mirror thread, but it describes a bit different thing. And how to use it inside the Docker container, e.g. the standard ocaml/opam2 Alpine container?

2

One possibility is to clone locally the opam-repository, add this repository as an opam repository and remove the default one.

3

Ok, so the github repo mirror will be like:

RUN opam repository remove default && \
opam repository add mirror https://localgit.some.dns/opam-repository.git

in the building machine Docker files.
But what about creating a file mirror for them?

4

Note that opam repository add|remove by default only affects the selections of the current switch. See the manpage (opam repo --help) and hints for details.
In your case you will probably prefer opam repository set-url default git://...

To create a mirror, see opam admin --help ; you could use it to fill directly ~/.opam/download-cache.

See http://opam.ocaml.org/doc/Manual.html#Repositories for more details on how the cache is handled.

2 Likes
5

And after that, run

opam admin cache

to populate the repository with the distfiles, and then make opam use that cache by adding it to ~/.opam/config in archive-mirrors if you need to.

2 Likes
6

The OCaml Docker images at ocam/opam2 and ocurrent/opam include a clone of opam-repository already.

To cache the archives, you could configure Docker to use BuildKit, which extends the Dockerfile syntax with support for caching.

See: https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/experimental.md

3 Likes
7

Thank you, everyone! All suggestions are useful. I will post an entire solution once will make it work.

8

I think it will be beneficial for OCaml community to have the instruction/page on how to setup the regional mirror, like some of the Linux distributions or package managers. And on how to add them into the global mirror lists:

See example pages on how to setup the mirror:

P.S. I want to setup a regional mirror, since downloading opam packages from China is a pain.

1 Like
9

For people who want a local copy for messing up with packages rather than for a CI process or community benefit: you can add a local git repo as a repository with opam repository add my-crazy-repo git+file://home/jrandomhacker/my-crazy-repo.

10

I had a similar need and my solution to it might help you out. I didn’t create a local git repo within my org, instead I just use the provided repository by the default ocaml docker images, and I tweak them a bit to work for me. All builds I trigger at my job are airtight (ie they don’t reach out to the internet – at least not directly, more below). So I leveraged the following things:

  • An HTTP proxy that caches fetched artifacts (this is the only thing that can reach the internet), pointing only to opam.ocaml.org.
  • The OPAMFETCH environment variable. Which in the man page reads
OPAMFETCH specifies how to download files: either `wget', `curl' or a
custom command where variables %{url}%, %{out}%, %{retry}%,
%{compress}% and %{checksum}% will be replaced. Overrides the
'download-command' value from the main config file.
  • A custom script that uses my HTTP proxy instead of opam.ocaml.org, which I set to OPAMFETCH according to the docs. The checksum is in the form of something like md5=<md5sum>. So for example you can see the sha256 for merlin3.3.3 here:

And you can see that it’s cached in opam.ocaml.org:

 $ curl --head https://opam.ocaml.org/cache/sha256/72/72909ef47eea1f6fca13b4109a34dccf8fe3923a3c026f1ed1db9eb5ee9aae15
HTTP/1.1 200 OK
...

The script I set to OPAMFETCH just replaces opam.ocaml.org with something like cache-proxy.my-big-enterprise.tld.

I noticed that what @avsm mentions is what they do in opam.ocaml.org (or something very similar, I don’t remember, I looked it up long ago), so by doing that you’d get a similar cache locally. I didn’t want to maintain that and I’m happy to just use the upstream opam.ocaml.org cache. I’m not assuming there’s any guarantees around that (nor do I expect that), but it’s a convenient way for me to fetch the sources defined by the default opam repository. Otherwise I’d have to set up all the different proxies to fetch sources and it would be a mess for me.

One thing I’d note however, please keep supporting OPAMFETCH! It was a life saver :smile:

Once that setup is done in my customized docker container, I just prefetch and install stuff like core, utop, etc.

I’m sure there’s a more convenient way of doing this, so if you end up documenting it then I’ll be very keen to read it.

11

Still no straightforward way to do this. I want to setup both opam tree and archives at https://mirrors.tencent.com, having an easy step-by-step guide like for https://wiki.gentoo.org/wiki/Project:Infrastructure/Mirrors/Source and easy way to setup daily mirroring, along with specifying the URL of it in opam easily.

12

It’s very important, because downloading opam packages is very slow (can take hours) behind the Chinese Great Firewall. So the mirroring will solve the problem for the entire country. Thank you.

1 Like
13

We have recently set up an OPAM mirror at https://mirrors.sjtug.sjtu.edu.cn. Users may use this mirror by running either of the following commands:

opam init default https://mirrors.sjtug.sjtu.edu.cn/git/opam-repository.git

opam repo set-url default https://mirrors.sjtug.sjtu.edu.cn/git/opam-repository.git --all --set-default

As I have no past experience in using OCaml and OPAM, I hope someone could help test if our mirror functions correctly.

For technical details, we synchronized the git repo, patched repo file, and provides a reverse proxy to https://opam.ocaml.org/cache. Not sure whether this is the correct way to set up a mirror.

3 Likes
14

I hoped for the solution without patching anything, better a documented properly like described in https://github.com/ocaml/opam/issues/4103

Anyway, great job!

1 Like