Bitbucket stopped supporting mercurial repositories

@khady, who regularly checks our links on ocamlverse, alerted me to the fact that bitbucket mercurial repos are no longer supported at all. A few OCaml repos are linked to on ocamlverse:

These repos are all dead now but are available via opam (temporarily?), and I imagine there are many more with this issue. I’d rather not delete these from ocamlverse but have the authors transition to git repos we can link to instead, as well as fixing the opam sources.

2 Likes

I the tarballs are no longer reachable, in opam we can retrieve them from the opam cache. These could be useful for the authors if they are moving their repository and want to preserve the old releases

2 Likes

An incredibly dumb grepping of the opam-repository repository revealed something like ~300 opam files across 67 projects mentioning tarballs previously hosted by bitbucket.org, which indeed are all gone at this point:

$ grep -RE 'bitbucket.org.+\.tar' . | grep -Po '(?<=./packages/)([^/]+)' | uniq
opencc
planck
trie
conjury
ppx_poly_record
unmagic
ppx_integer
camlon
bitstring
ppx_monadic
nlopt-ocaml
ibx
oth
opencc0
ppx_implicits
pvem_lwt_unix
ppx_overload
ppx_orakuda
levenshtein
meta_conv
nonstd
ppxx
lua_pattern
opencc1
charInfo_width
tidy
pvem
spotlib_js
gnuplot
sibylfs-lem
gadelac
ppx_test
ppx_sexp
treeprint
typpx
ppx_dotbracket
minimal
azure-cosmos-db
orakuda
pa_monad_custom
ocamldap
macaque_lwt
opamfind
libsvm
ocaml-indent
spotlib
tcx
ppx_meta_conv
pds
docout
camlimages
oni
snabela
mmseg
tiny_json
revops
ppx_curried_constr
tiny_json_conv
ppx_pattern_guard
spotinstall
ocamlspot
hll
orsetto
dune_watch
merlin-of-pds
pa_ovisitor
sosa
  1. I wonder if there is a “safe” place that’s been set aside for such orphaned libraries to be relocated to (presuming tarballs matching the published hashes are recoverable somewhere)?
  2. Given the inevitability of broken links like this (and also the security implications of package repositories not retaining released artifacts themselves, immutably), has there been any consideration of having the opam repository capture and retain tarballs permanently? I know this is a bit off topic for this thread, but seeing changes like this (where both upstream tarballs and their hashes are being changed) gives me flashbacks to some bad old days with Maven, npm, and PyPI when those package managers had a similar laissez-faire approach to artifact retention and integrity.

[edit: I posted a more comprehensive question on the topic of opam’s security posture and data integrity policies: Opam-repository: security and data integrity posture]

4 Likes

just worth mentioning that bitbucket has supported git for a long time and that many of those bitbucket.org/* URLs still work.

 $ wget "https://bitbucket.org/smondet/nonstd/get/nonstd.0.0.3.tar.gz"
--2020-09-22 19:43:52--  https://bitbucket.org/smondet/nonstd/get/nonstd.0.0.3.tar.gz
Resolving bitbucket.org (bitbucket.org)... 18.205.93.0, 18.205.93.2, 18.205.93.1, ...
Connecting to bitbucket.org (bitbucket.org)|18.205.93.0|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5036 (4.9K) [application/x-tar-gz]
Saving to: 'nonstd.0.0.3.tar.gz'

nonstd.0.0.3.tar.gz               100%[=============================================================>]   4.92K  --.-KB/s    in 0.002s  

2020-09-22 19:43:52 (2.05 MB/s) - 'nonstd.0.0.3.tar.gz' saved [5036/5036]

1 Like

Oh, very nice then, that’s good to know. I had checked a half-dozen of them that happened to come back 404, and I guess I made a poor assumption on that (unlucky) basis.

I have a few repos in there, I’ll work on getting them up with new links this weekend.

1 Like