Dune: how to deal with opam extra-source?

Learning
1

Hi OCaml Dune experts, I’ve been trying to adopt Dune as the build system for a followup to my Orsetto project, and it’s going okay, but I haven’t yet tried to deal with the problem of the extra-source fields in its opam file. I need help, and scouring the fabulous documentation for dune leaves me confused about how to proceed. Help! Please?

Basic summary: the OMakefile system in Orsetto is designed to work offline apart from opam in the usual case, and building it requires downloading some files from a web site (and I don’t want to redistribute them inside the source code for the package). These files are listed with extra-source fields in the opam file, so that when building with opam they just magically land in the root of the source tree after the tarball is unpacked. The OMakefile in Orsetto finds them and uses them if they’re present. But if they’re not present, then it invokes curl and downloads them directly.

These extra-source files include various published databases for generating OCaml source code and the tests of that source code. I’d like to do something functionally equivalent in the dune project for my replacement, but there are some additional complications. I see that dune wants to generate the opam file automatically, which is nice, and I want to do that. Alas, I can’t figure out how to generate the extra-source fields. (I’ve found out about the project.opam.template feature) Moreover, I’m not sure how to do the “if OPAM didn’t download these, then download them yourself” logic.

Can someone with more Dune kung fu than me help sort me out, please? Obviously, I could patch out to a Bourne shell script, but I don’t want to be broken on Windows because of that. Surely, there must be some way to do this natively in Dune?

2

The bad news is that we don’t have a good story for handling things like extra-sources yet. The good news is that it’s something that we’re working on right now.

For now, I would suggest that you keep the parts of your Makefile (or OMakefile) that handle this field and wrap your invocation of dune with it.

1 Like
3

I guess you could use something like the following.

(rule
 (target my-download.lol)
 (action (run bash "curl" …)))

I think the main issue you might have is that it downloads inside your _build/ directory so it’d be erased by dune clean and rm -rf _build/ and such. You could try something with the promote modes maybe?

4

Thanks for confirming my suspicions. I’ve written a workaround along those lines. This issue is probably the most challenging part of making my project usable on non-Unix platforms.

5

You are welcome to use my unreleased dk scripts if you want a Makefile-like build tool that can invoke whatever (dune, opam, etc.) on Windows/Linux/macOS.

Try copy-and-pasting the following on Linux/macOS or in PowerShell on Windows:

git clone https://gitlab.com/diskuv/samples/DkHelloWorld.git
cd DkHelloWorld
./dk dksdk.ci.cmake.link HELP

That is the simplest cross-platform script I have. There are no prerequisites the user has to pre-install. You can make your own build or dev scripts (the above was dksdk.ci.cmake.link defined in cmake/scripts/dksdk/ci/cmake/link.cmake).

It is straightforward to add more OS-es. Basically it is a CMake caching downloader + CMake script invoker designed for a simple user experience.

Since I doubt you know CMake, you would only need to use two commands:

There are examples of both commands in the source code directories of cmake/scripts · 1.0 · Diskuv / Samples / DkHelloWorld · GitLab

I haven’t formally slapped a license on the code, but Apache 2.0 for ./dk, ./dk.cmd, ./cmake/FindDkToolScripts.cmake and the existing ./cmake/scripts/**/*.cmake scripts as of 2023-05-20. Eventually I’ll do a proper announcement once I’ve made a self-upgrade script and added documentation.

1 Like