# Introducing a "yanked" mechanism in opam **URL:** \ **Category:** Ecosystem\ **Tags:** jbuilder, opam\ **Created:** [March 13, 2018, 2:47pm UTC](https://discuss.ocaml.org/t/introducing-a-yanked-mechanism-in-opam/1707 "2018-03-13T14:47:04Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Joris](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/joris/32/1073_2.png) [@Joris](https://discuss.ocaml.org/u/Joris)\ **Post date:** [March 13, 2018, 2:47pm UTC](https://discuss.ocaml.org/t/introducing-a-yanked-mechanism-in-opam/1707/1 "2018-03-13T14:47:05Z")
Rust package manager has a feature called “yank”. To quote the doc : > Occasions may arise where you publish a version of a crate that actually ends up being broken for one reason or another (syntax error, forgot to include a file, etc.). For situations such as this, Cargo supports a “yank” of a version of a crate. Do you think a similar functionality would be useful in opam ? I think it should prevent installing this version for fresh install, and probably without special flag make `opam upgrade`̀ downgrade the package to the latest non yanked version until a newer version is release. To provide some context, dune beta18 was recently release and it broke non-jbuilder buildsystem building projects with base/core dependencies. For background see [core\_kernel\_\_.cmx is not installed in beta18 · Issue #567 · ocaml/dune · GitHub](https://github.com/ocaml/dune/issues/567) 12 days after report, the broken version is still live in opam, affecting a lot of users and making beginners experience more painful. I understand unpublishing a version from opam is not easy and releasing a new version might be heavy, so that a yanking mechanism with opt-in force install could be a good way of solving this kind of problems in the future ? ---
**Author:** ![avsm](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/avsm/32/6_2.png) [@avsm](https://discuss.ocaml.org/u/avsm)\ **Post date:** [March 13, 2018, 3:32pm UTC](https://discuss.ocaml.org/t/introducing-a-yanked-mechanism-in-opam/1707/2 "2018-03-13T15:32:09Z")
> [@Joris](#): > > 12 days after report, the broken version is still live in opam, affecting a lot of users and making beginners experience more painful I don’t think we should have let beta18 be live for as long as we did, in retrospect. I had a discussion with @rgrinberg about whether to add a conflict, and our reaction a few days ago was that it only affected a few users. However, it turns out to be quite a few projects, and also on osx due to the fd leak issue (that requires bumping the ulimit due to macOS’s incredibly low default). So this specific version’s bugs probably outweighed the utility of the new features. I do have some other “local conflicts” for things that shouldn’t go in the mainline opam repository. For example, I have a [GitHub - avsm/no-camlp4: an opam remote that does nothing but conflict with camlp4](https://github.com/avsm/no-camlp4) package to alert me whenever a local dependency has brought camlp4 back in again, which is perhaps another use for a local package override facility. ---
**Author:** ![rgrinberg](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/rgrinberg/32/40_2.png) [@rgrinberg](https://discuss.ocaml.org/u/rgrinberg)\ **Post date:** [March 13, 2018, 3:50pm UTC](https://discuss.ocaml.org/t/introducing-a-yanked-mechanism-in-opam/1707/3 "2018-03-13T15:50:15Z")
One thing that I considered is publishing releasing candidates to opam. This could be productive for both users and dune itself. Does anyone else do this? ---
**Author:** ![avsm](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/avsm/32/6_2.png) [@avsm](https://discuss.ocaml.org/u/avsm)\ **Post date:** [March 13, 2018, 3:51pm UTC](https://discuss.ocaml.org/t/introducing-a-yanked-mechanism-in-opam/1707/4 "2018-03-13T15:51:21Z")
Noone does, but we could certainly arrange for special CI coverage. Are you thinking of putting it into a new package name, or a different remote? ---
**Author:** ![rgrinberg](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/rgrinberg/32/40_2.png) [@rgrinberg](https://discuss.ocaml.org/u/rgrinberg)\ **Post date:** [March 13, 2018, 3:53pm UTC](https://discuss.ocaml.org/t/introducing-a-yanked-mechanism-in-opam/1707/5 "2018-03-13T15:53:57Z")
I’m not sure. I thought I could simply publish and rc tag as usual but have opam not recommend the upgrade by default. I’d still like to get wide user testing for RC’s somehow. ---
**Author:** ![avsm](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/avsm/32/6_2.png) [@avsm](https://discuss.ocaml.org/u/avsm)\ **Post date:** [March 13, 2018, 4:31pm UTC](https://discuss.ocaml.org/t/introducing-a-yanked-mechanism-in-opam/1707/6 "2018-03-13T16:31:16Z")
That’s an interesting idea. I wonder if we could depend on a `base-experimental` package to pull in bleeding edge packages. The best way to do this in opam2 would be to simply have a opam-testing remote that users who wish to test bleeding edge packages could just add. ---
**Author:** ![ivg](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/ivg/32/103_2.png) [@ivg](https://discuss.ocaml.org/u/ivg)\ **Post date:** [March 13, 2018, 4:36pm UTC](https://discuss.ocaml.org/t/introducing-a-yanked-mechanism-in-opam/1707/7 "2018-03-13T16:36:37Z")
As a hack, a broken package can be disabled using `available: [false]` stanza. Or by introducing a dependency on a special package that is never available, but that will provide a user friendly message when somebody is requiring it. ---
**Author:** ![ivg](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/ivg/32/103_2.png) [@ivg](https://discuss.ocaml.org/u/ivg)\ **Post date:** [March 13, 2018, 4:39pm UTC](https://discuss.ocaml.org/t/introducing-a-yanked-mechanism-in-opam/1707/8 "2018-03-13T16:39:23Z")
> [@rgrinberg](#): > > One thing that I considered is publishing releasing candidates to opam. This could be productive for both users and dune itself. Does anyone else do this? We ([BAP](https://github.com/BinaryAnalysisPlatform/opam-repository/tree/bap-1.4)) have our own opam repository where the release candidates are published and we wait two weeks before we even try to make a PR to the upstream (and then wait yet another two weeks to get it merged). ---
**Author:** ![rgrinberg](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/rgrinberg/32/40_2.png) [@rgrinberg](https://discuss.ocaml.org/u/rgrinberg)\ **Post date:** [March 13, 2018, 6:55pm UTC](https://discuss.ocaml.org/t/introducing-a-yanked-mechanism-in-opam/1707/9 "2018-03-13T18:55:02Z")
Interesting. May I ask how many users does this testing repository have? My main fear with having these “bleeding edge” repos is that I’ll be putting work that will end up in a black hole. From all the OCaml users I know, I know of very few who use custom opam repositories. I can’t remember the last I’ve used one myself personally. @avsm Seems like mirage also has experience with custom repositories: [https://github.com/mirage/mirage-dev](https://github.com/mirage/mirage-dev). How useful does mirage-dev end up being? ---
**Author:** ![ivg](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/ivg/32/103_2.png) [@ivg](https://discuss.ocaml.org/u/ivg)\ **Post date:** [March 13, 2018, 7:03pm UTC](https://discuss.ocaml.org/t/introducing-a-yanked-mechanism-in-opam/1707/10 "2018-03-13T19:03:32Z")
> [@rgrinberg](#): > > Interesting. May I ask how many users does this testing repository have? My main fear with having these “bleeding edge” repos is that I’ll be putting work that will end up in a black hole. From all the OCaml users I know, I know of very few who use custom opam repositories. I can’t remember the last I’ve used one myself personally. I have no idea, honestly. We are using it internally in our CI system and test manually. I always encourage people to use our repository, though no sure how many are actually following my advice. Probably, we should have something like `testing` and `stable` branches of `opam-repository`. And updating stable only once a few months after extensive testing. While keeping the rolling release approach for the `testing` branch. ---
**Author:** ![hcarty](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/hcarty/32/1824_2.png) [@hcarty](https://discuss.ocaml.org/u/hcarty)\ **Post date:** [March 13, 2018, 7:03pm UTC](https://discuss.ocaml.org/t/introducing-a-yanked-mechanism-in-opam/1707/11 "2018-03-13T19:03:38Z")
A development repository would be useful, I think, if it could be disabled without having to remove it completely. Having `opam repository disable dune-testing` and `opam repository enable dune-testing` would streamline the process. ---
**Author:** ![Joris](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/joris/32/1073_2.png) [@Joris](https://discuss.ocaml.org/u/Joris)\ **Post date:** [March 14, 2018, 1:23am UTC](https://discuss.ocaml.org/t/introducing-a-yanked-mechanism-in-opam/1707/12 "2018-03-14T01:23:09Z")
interesting idea. The point of “yank” in cargo though is not to beak explicit dependencies. Say you depend on a version explicitly or put a version constraint which makes the yanked version the only installable one, it will be pulled in. You cannot publish new packages with a dependency to a yanked version though. ---
**Author:** ![Joris](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/joris/32/1073_2.png) [@Joris](https://discuss.ocaml.org/u/Joris)\ **Post date:** [March 14, 2018, 2:59am UTC](https://discuss.ocaml.org/t/introducing-a-yanked-mechanism-in-opam/1707/13 "2018-03-14T02:59:09Z")
opam could have support for -rc/-alpha version so that they need to be explicitly upgraded/installed (or have a global preference to `opam upgrade` them all). Not sure how hard it would be ---
**Author:** ![paurkedal](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/paurkedal/32/586_2.png) [@paurkedal](https://discuss.ocaml.org/u/paurkedal)\ **Post date:** [March 14, 2018, 8:28am UTC](https://discuss.ocaml.org/t/introducing-a-yanked-mechanism-in-opam/1707/14 "2018-03-14T08:28:19Z")
> [@hcarty](#): > > A development repository would be useful, I think, if it could be disabled without having to remove it completely. Having opam repository disable dune-testing and opam repository enable dune-testing would streamline the process. Not the same thing, but from the opam 2.0 manual: > opam repository add will now setup the new repository for the current switch only, unless you specify --all which I think is exactly what I’d want if using a separate switch for testing. ---
**Author:** ![paurkedal](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/paurkedal/32/586_2.png) [@paurkedal](https://discuss.ocaml.org/u/paurkedal)\ **Post date:** [March 14, 2018, 8:41am UTC](https://discuss.ocaml.org/t/introducing-a-yanked-mechanism-in-opam/1707/15 "2018-03-14T08:41:32Z")
> [@Joris](#): > > opam could have support for -rc/-alpha version so that they need to be explicitly upgraded/installed (or have a global preference to opam upgrade them all). Not sure how hard it would be As in opam recognizing filtering out on version numbers by default? I think it would be nice with more explicit control, like named masks (`testing`, `obsolete`, `insecure`, etc.) in the repo, which users must match with a global or per-package unmasking. I mentioned this [January 26 in a PR about cleaning up old packages](https://github.com/ocaml/opam-repository/pull/11194), which is another use case. I guess tags may be reused for this, with a dedicated namespace to avoid accidental masking. ---
**Author:** ![cfcs](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/cfcs/32/833_2.png) [@cfcs](https://discuss.ocaml.org/u/cfcs)\ **Post date:** [March 16, 2018, 3:24am UTC](https://discuss.ocaml.org/t/introducing-a-yanked-mechanism-in-opam/1707/16 "2018-03-16T03:24:51Z")
Some anecdotal evidence re: whether `jbuilder-beta18` was used by anyone: Breaking `jbuilder` just before the mirage retreat (where everyone updates their system because the network in Morocco is limited) was definitely a source of some frustration, and quite a lot of debugging turned out to have “oh you’re using a broken jbuilder release” as its root cause. But alas that is the way it is with beta software, there will be broken things. It sounds like the building of dependencies / reverse dependencies (recursively/transitively) mentioned above might be a good thing. 🙂 I think there are many good points in this thread raised by everyone, and thanks to @Joris for explaining the details of how `yank` works in the Rust ecosystem. Just for good measure, since it has gone unadressed, I’d like to make an argument against letting people remove versions from opam. Doing so would: 1. break pinned dependencies 2. make it harder to do reproducible builds, checksumming, signing, CI, anything that rests on the assumption that a release has a given hash and is immutable - for example when you later reintroduce a new software release with a different version number (meaning we have to come up with a new coordinate system for `xxx version 1.02, version [1,2,3,4, ... yanks+1]`) As I see it there is one way to do this that doesn’t introduce _any_ extra complexity: Bumping minor version and re-releasing the old package. An alternative that comes with some complexity, but yet is still fairly viable, would be to change the OPAM metadata for the broken release to be uninstallable, as suggested in comments above. I think this is the most practical solution. PS: I’m a fan of the discussion of overlays and `debian-testing`-like overlays on top of OPAM for the brave and hungry. 🙂 ---
**Author:** ![rgrinberg](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/rgrinberg/32/40_2.png) [@rgrinberg](https://discuss.ocaml.org/u/rgrinberg)\ **Post date:** [March 16, 2018, 8:58am UTC](https://discuss.ocaml.org/t/introducing-a-yanked-mechanism-in-opam/1707/17 "2018-03-16T08:58:06Z")
I’m completely for partitioning opam better into stable/unstable etc. (we can do it in a debian way, but I actually think their naming convention is confusing). Not only this will give stability to those who need, but it will also save some of us a lot of pinning and rushing out releases because of user demand. But I think the most important thing to make sure is that we have actual presence on the bleeding edge. One more thing I’d like to mention here, the fact that dune is a `{build}` dependency ended up being quite harmful here. I think some people were doubly confused after downgrading dune, that the problem wasn’t fixed. Finally, let’s not design dune’s release process around the internet quality in Marrakech 🙂 That’s the most important issue being pointed out in this thread, let’s hope that it’s addressed next time. ---
**Author:** ![Khady](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/khady/32/469_2.png) [@Khady](https://discuss.ocaml.org/u/Khady)\ **Post date:** [March 16, 2018, 2:30pm UTC](https://discuss.ocaml.org/t/introducing-a-yanked-mechanism-in-opam/1707/18 "2018-03-16T14:30:39Z")
All the propositions of multiple repositories and stable/testing tags don’t solve the real problem. Original question is how to deal with a broken package once published. The same problem could happen in a stable repository. If it is not possible to unpublish or yank a package in the stable repository, we have no way to deal with a broken package. ---
**Author:** ![paurkedal](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/paurkedal/32/586_2.png) [@paurkedal](https://discuss.ocaml.org/u/paurkedal)\ **Post date:** [March 16, 2018, 3:01pm UTC](https://discuss.ocaml.org/t/introducing-a-yanked-mechanism-in-opam/1707/19 "2018-03-16T15:01:49Z")
A testing repository would reduce the risk of such problems by increasing the number of people who test packages before they are merged to stable. Using tags/masks in main repository would also allow adding a `mask:broken` tag after the breakage is discovered, so that the problematic package gets downgraded after then next opam update and upgrade (unless one locally unmasks). ---
**Author:** ![cfcs](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/cfcs/32/833_2.png) [@cfcs](https://discuss.ocaml.org/u/cfcs)\ **Post date:** [March 19, 2018, 6:10pm UTC](https://discuss.ocaml.org/t/introducing-a-yanked-mechanism-in-opam/1707/20 "2018-03-19T18:10:35Z")
@rgrinberg 1. Re: Partitioning: I think some sort of partitioning would be good. People have also proposed having a “known good, works on all mirageos targets” repo where new fancy, incorrectly packaged C code is not allowed. 2. Re: `{build}`: Shouldn’t it be OPAM’s job to rebuild if any dependency that was used (be that `{build}` or `{test}` or `{doc}`) dependency gets updated? 3. Re: Internet in Marrakesh: While it’s completely standard for a North American company to disregard the African continent and its inhabitants, one could argue that by designing dune’s release process around _working software_ you would have had a chance to kill two birds with one stone in this case. 😉 @Khady > If it is not possible to unpublish or yank a package in the stable repository, we have no way to deal with a broken package. I proposed two solutions in my [post the day before yours](https://discuss.ocaml.org/t/introducing-a-yanked-mechanism-in-opam/1707/16), perhaps you were viewing an out-of-date tab when you posted your reply? [Next page](https://discuss.ocaml.org/t/introducing-a-yanked-mechanism-in-opam/1707.md?page=2)