Rust package manager has a feature called “yank”. To quote the doc :
Occasions may arise where you publish a version of a crate that actually ends up being broken for one reason or another (syntax error, forgot to include a file, etc.). For situations such as this, Cargo supports a “yank” of a version of a crate.
Do you think a similar functionality would be useful in opam ? I think it should prevent installing this version for fresh install, and probably without special flag make opam upgradè downgrade the package to the latest non yanked version until a newer version is release.
To provide some context, dune beta18 was recently release and it broke non-jbuilder buildsystem building projects with base/core dependencies. For background see core_kernel__.cmx is not installed in beta18 · Issue #567 · ocaml/dune · GitHub
12 days after report, the broken version is still live in opam, affecting a lot of users and making beginners experience more painful. I understand unpublishing a version from opam is not easy and releasing a new version might be heavy, so that a yanking mechanism with opt-in force install could be a good way of solving this kind of problems in the future ?
I don’t think we should have let beta18 be live for as long as we did, in retrospect. I had a discussion with @rgrinberg about whether to add a conflict, and our reaction a few days ago was that it only affected a few users. However, it turns out to be quite a few projects, and also on osx due to the fd leak issue (that requires bumping the ulimit due to macOS’s incredibly low default). So this specific version’s bugs probably outweighed the utility of the new features.
I do have some other “local conflicts” for things that shouldn’t go in the mainline opam repository. For example, I have a GitHub - avsm/no-camlp4: an opam remote that does nothing but conflict with camlp4 package to alert me whenever a local dependency has brought camlp4 back in again, which is perhaps another use for a local package override facility.
One thing that I considered is publishing releasing candidates to opam. This could be productive for both users and dune itself. Does anyone else do this?
I’m not sure. I thought I could simply publish and rc tag as usual but have opam not recommend the upgrade by default. I’d still like to get wide user testing for RC’s somehow.
As a hack, a broken package can be disabled using available: [false] stanza.
Or by introducing a dependency on a special package that is never available, but that will provide a user friendly message when somebody is requiring it.
We (BAP) have our own opam repository where the release candidates are published and we wait two weeks before we even try to make a PR to the upstream (and then wait yet another two weeks to get it merged).
Interesting. May I ask how many users does this testing repository have? My main fear with having these “bleeding edge” repos is that I’ll be putting work that will end up in a black hole. From all the OCaml users I know, I know of very few who use custom opam repositories. I can’t remember the last I’ve used one myself personally.
I have no idea, honestly. We are using it internally in our CI system and test manually. I always encourage people to use our repository, though no sure how many are actually following my advice.
Probably, we should have something like testing and stable branches of opam-repository. And updating stable only once a few months after extensive testing. While keeping the rolling release approach for the testing branch.
A development repository would be useful, I think, if it could be disabled without having to remove it completely. Having opam repository disable dune-testing and opam repository enable dune-testing would streamline the process.
interesting idea. The point of “yank” in cargo though is not to beak explicit dependencies. Say you depend on a version explicitly or put a version constraint which makes the yanked version the only installable one, it will be pulled in. You cannot publish new packages with a dependency to a yanked version though.
opam could have support for -rc/-alpha version so that they need to be explicitly upgraded/installed (or have a global preference to opam upgrade them all). Not sure how hard it would be
As in opam recognizing filtering out on version numbers by default? I think it would be nice with more explicit control, like named masks (testing, obsolete, insecure, etc.) in the repo, which users must match with a global or per-package unmasking. I mentioned this January 26 in a PR about cleaning up old packages, which is another use case. I guess tags may be reused for this, with a dedicated namespace to avoid accidental masking.
Some anecdotal evidence re: whether jbuilder-beta18 was used by anyone:
Breaking jbuilder just before the mirage retreat (where everyone updates their system because the network in Morocco is limited) was definitely a source of some frustration, and quite a lot of debugging turned out to have “oh you’re using a broken jbuilder release” as its root cause. But alas that is the way it is with beta software, there will be broken things. It sounds like the building of dependencies / reverse dependencies (recursively/transitively) mentioned above might be a good thing.
I think there are many good points in this thread raised by everyone, and thanks to @Joris for explaining the details of how yank works in the Rust ecosystem.
Just for good measure, since it has gone unadressed, I’d like to make an argument against letting people remove versions from opam. Doing so would:
break pinned dependencies
make it harder to do reproducible builds, checksumming, signing, CI, anything that rests on the assumption that a release has a given hash and is immutable
for example when you later reintroduce a new software release with a different version number (meaning we have to come up with a new coordinate system for xxx version 1.02, version [1,2,3,4, ... yanks+1])
As I see it there is one way to do this that doesn’t introduce any extra complexity: Bumping minor version and re-releasing the old package.
An alternative that comes with some complexity, but yet is still fairly viable, would be to change the OPAM metadata for the broken release to be uninstallable, as suggested in comments above. I think this is the most practical solution.
PS: I’m a fan of the discussion of overlays and debian-testing-like overlays on top of OPAM for the brave and hungry.
I’m completely for partitioning opam better into stable/unstable etc. (we can do it in a debian way, but I actually think their naming convention is confusing). Not only this will give stability to those who need, but it will also save some of us a lot of pinning and rushing out releases because of user demand. But I think the most important thing to make sure is that we have actual presence on the bleeding edge.
One more thing I’d like to mention here, the fact that dune is a {build} dependency ended up being quite harmful here. I think some people were doubly confused after downgrading dune, that the problem wasn’t fixed.
Finally, let’s not design dune’s release process around the internet quality in Marrakech That’s the most important issue being pointed out in this thread, let’s hope that it’s addressed next time.
All the propositions of multiple repositories and stable/testing tags don’t solve the real problem. Original question is how to deal with a broken package once published. The same problem could happen in a stable repository. If it is not possible to unpublish or yank a package in the stable repository, we have no way to deal with a broken package.
A testing repository would reduce the risk of such problems by increasing the number of people who test packages before they are merged to stable. Using tags/masks in main repository would also allow adding a mask:broken tag after the breakage is discovered, so that the problematic package gets downgraded after then next opam update and upgrade (unless one locally unmasks).
Re: Partitioning: I think some sort of partitioning would be good.
People have also proposed having a “known good, works on all mirageos targets” repo where new fancy, incorrectly packaged C code is not allowed.
Re: {build}: Shouldn’t it be OPAM’s job to rebuild if any dependency that was used (be that {build} or {test} or {doc}) dependency gets updated?
Re: Internet in Marrakesh: While it’s completely standard for a North American company to disregard the African continent and its inhabitants, one could argue that by designing dune’s release process around working software you would have had a chance to kill two birds with one stone in this case.
