Should opam use a lockfile by default?

[mseri]

@BikalGurung

Perhapas the notion of a package version could adopt a git commit hash since opam-repository is already in a git repo.

That would not help in some of the breaking changes (package no longer available upstream, with a new tarball provided). And we are trying to avoid adding patches without incrementing the versions now.

[AltGr]

Hello, chiming in to provide some hints, and see where some extensions or ironing is needed

• Archives no longer existing should no longer be a problem with opam 2’s cache: all upstream files are kept, indexed by their hash(es), and the policy is to never clean that cache. In other words, you can still download a package archive if the package has been removed from the repo, but you didn’t run opam update. Even if the upstream is no longer there.

  Limitations:

  • this only applies to the official repo at https://opam.ocaml.org, or any HTTP repo you regularly use opam admin cache on
  • the cache is not used, by default, for VCed repos. You can, however, configure any cache URL arbitrarily in your ~/.opam. See http://opam.ocaml.org/doc/Manual.html#configfield-archive-mirrors

• The opam repository is on git, and you can use the same syntax as ffor pins to use it at a specific commit. Combined with the cache mentionned above, it means you can really re-use the exact same repository for your rebuilds.

  Try

  opam repo add locked-repo git+https://github.com/ocaml/opam-repository#d115d9bb
  

  or even

  opam switch create reproducible --repos=locked-repo=git+https://github.com/ocaml/opam-repository#d115d9bb
  

  Don’t know what hash you are currently on ? It’s part of the output of opam config report.

Hope this helps!

[jordwalke]

As some mentioned here, lock files don’t achieve total reproducibility - if a package is somehow removed from the network, or if your network is disconnected.

esy generates/uses lockfiles by default and people check them into repos by convention, but if you want total reproducibility even in the face of network disconnection, you can pass the flag --cache-tarballs-path=./vendor to esy which will also snapshot tarballs of all your dependencies/compilers from your lockfile in that directory.
If your repo can handle it you can source control it as well so you can build on every commit regardless of what the network does. I don’t recommend that for open source projects, but it’s very nice for internal projects.

[rjbou]

There is the opam-bundle project that could be useful, but it handles more from a system view (it begins by compiling ocaml).
Sources archives are already stored/cached in the opamroot (using their hash), we could easily imagine a command that produces a a sort of lock sources: as the --lock command, but it would copy cached sources (need to handle pinned dependencies also) and aggregate them in an archive. And on install, use this archive sources as a external existing cache.