Generating .opam file url section with dune

Ecosystem
1

Hello, friends! Was hoping you could help out with an opam repository debacle.

We host our own opam repository at the UChicago Library, which has led to a number of interesting challenges. (And we haven’t really come across anyone else who is doing so, which is what makes the challenges particularly challenging.)

Each package-entry in the opam repository on the server we’re hosting it from contains a directory with a (versioned) name of the package, which contains the opam file from the original source and pretty much nothing else.

So say I’m on my desktop and I want to install amazing-library. I type the following:

$ opam repository add dldc https://dldc.lib.uchicago.edu/opam
$ opam install amazing-library

Now here, if the opam file for amazing-library on the server contains a url section including a GitHub https clone URL, opam successfully downloads the source for the project. Yay. However, if the opam file on the server does not contain a url section, opam install amazing-library on my machine fails with a build error due to not being able to download the source for the project:

#=== ERROR while compiling amazing-library.~dev ========================================#
# context     2.3.0 | linux/x86_64 | ocaml-system.4.14.1 | https://dldc.lib.uchicago.edu/opam#013d76892572+
# path        /usr/app/lib/opam/test/.opam-switch/build/amazing-library.~dev
# command     /usr/app/lib/opam/opam-init/hooks/sandbox.sh build dune build -p amazing-library -j 7 @install
# exit-code   1
# env-file    /usr/app/lib/opam/log/amazing-library-1521043-6b8737.env
# output-file /usr/app/lib/opam/log/amazing-library-1521043-6b8737.out
### output ###
# Error: I don't know about package amazing-library (passed through --only-packages)

One aspect of this workflow is kind of bothersome. We use dune to generate the opam files for all of our projects whose source is hosted on GitHub, and for the life of me I cannot figure out what magic s-expression I need to stick in dune-project to get it to generate the url section of the opam file, when generate_opam_files is set to true. Because of this, we have to have our tooling grovel the dev-repo field out of the opam file, use it to construct a url section, and either stick that url section onto the end of the opam file or put it in a separate url file. Not the end of the world, but ultimately it’s hacky and I’m not crazy about the extra complexity it introduces into our workflows for no good reason.

What I would like is to be able to stick something in my dune-project file and have dune add the url section to the opam file when it generates the opam file. Is that even possible? I hope the answer is yes! Would love any advice you may have to give on this.

2

See GitHub - tarides/dune-release: Streamlining the release of dune packages to opam

EDIT: Eg

$ cd .../amazing-library
$ git tag -a v0.0.1
$ dune-release distrib
$ dune-release opam pkg
[+] Wrote opam package description _build/amazing-library.0.0.1/opam
1 Like
3

IIUC, I think you are looking for the (source ...) field:
https://dune.readthedocs.io/en/stable/reference/dune-project/generate_opam_files.html

1 Like
4

Ooh, very nice; just tried this out. So it creates the tarball in order to compute the checksum, then puts both that and the link to where GitHub is going to put the tarball when it sees the release tag. Thanks; I think we’re going to try this!

1 Like
5

As far as I know, (source ...) in dune-project only affects the dev-repo field in the .opam file.

6

Do you have any guesses as to why they didn’t put this functionality straight into dune? I thought the idea was for the dune-project to supersede the .opam file.

Maybe the way opam is thinking of this, there are two distinct things that both get (confusingly) talked about as the opam file:

  • awesome-library.opam at the root of the project, for opam package-level stuff on your local machine
  • the more release-y opam file that goes into the opam repository serverside, which is like awesome-library.opam but has the additional url file appended to it in braces, with checksums and links to tarballs and stuff

If that’s right, then I guess the dune tool only aims to overtake the functionality of the first type of opam file?

7

Not sure tbh. Best guess is that originally dune was intended for building only and not for package management (publishing etc.) and this was never revisited. Maybe in the future there will be a dune pkg publish command.

1 Like