[help wanted] crowd-sourcing package build fixes in opam-repository

Hi everyone,

I have collected a list of 50 or so packages on opam-repository whose
build is currently broken on 4.07.1. I would like to crowd-source
fixes for these packages, so that either their build is fixed or they
are marked as unbuildable. I’m posting the list in a second post
below. Please consider contributing by taking one of the packages in
the list with a TODO marker and trying to fix the issue.

Workflow

The post below is a wiki. You can edit it to mark that you are working
on one of the issues.

TODO => WIP(@user) => DONE(@user, link to change/report)

Some issues don’t have a TODO marker, because it’s unclear what to
do. Feel free to add one if you have a suggestion for what to do.

How?

I have marked some hypotheses of mine on what the issue probably is,
and how it could be fixed. Some common issues:

  • The package is missing an upper bound on one of its dependencies;
    this shows up as build failures asking for library functions that
    don’t exist anymore. You have to inspect the dependencies of older
    and newer versions of the package, possibly the changelog of the
    dependency, to guess what the upper bound should be; then test it.

  • The package is broken due to an incompatibility between ‘bytes’ and
    ‘string’, often in an _oasis file; you can just add a dependency on
    the virtual package ‘base-unsafe-string’, which will mark it
    uninstallable on safe-string versions of OCaml.

  • The archive where the package can be found has changed; opam files
    for newer versions will lead you to the new URLs – if the new place
    doesn’t have archives for older versions, you could ask the
    maintainer and/or remove the package.

  • The checksum for an archive has changed. Inspecting the new archive
    to ensure that it is correct helps.

Why?

The failing packages are often older versions of existing, non-failing
packages. While many of us may not care about the buildability of
older versions, it does matter for the good health of the repository:

  • Some users have pinned older versions for stability reasons. If they
    want to, for example, upgrade the OCaml version but preserve the
    version of their dependencies, this should still work – or be
    clearly marked as uninstallable, not a confusing error in the middle
    of the build.

  • Old packages show up in the “revdeps” of new opam-repository
    submissions – when we upload a new version of a package, its
    dependencies are automatically tested. Each build failure in the
    revdeps requires human investigation, to understand if the issue is
    due to the new package or is irrelevant. Irrelevant failures consume
    time and should be reduced as much as possible.

  • In the future we want to keep getting more aggressive on CI testing,
    for example test OCaml packages against the development version of
    OCaml on a regular basis. Again, broken builds slow us down and make
    it harder to spot the true issues.

Also

If you have encountered other packages that fail to build on the opam
CI, feel free to make a post of your own to list them, with build
logs, and encourage us all to contribute in fixing the issue.

3 Likes
  • aws: missing version bound on whoever provides the Util package

    aws.0.0.2
    aws.0.0.3

  • broken: 0.4.2 appears to fail to build, TODO investigate

    log

  • brotli, brozip: missing depexts

    brotli.1.0 (first fail)
    brotli.2.0.2 (last fail)
    brozip.1.0
    brozip.1.1

  • camldiet: reuses standard library’s Map interface contraviariantly; TODO fix upstream

    Camldiets.0.2

  • cmark: missing depexts

    cmark.0.2.0

  • crunch: 1.0.0.0 has out-of-date _oasis (string/bytes conflict); later versions work

    crunch.1.0.0
    crunch.1.4.0 (works)

  • fstar: OCaml 4.07.1 ins unspported for 0.9.4-beta0; TODO fix in opam-repository (unclear which versions are supported)

    fstar.0.9.4-beta0

  • dtoa: download error, the archive name has changed; TODO fix in opam-repository

    dtoa.0.1

  • grib: missing depexts

    grib.0.9.7
    grib.0.11.0

  • gsl: 1.18.5 fails to build due to C API incompatibility on ‘gsl_multifit_linear_svd’; later versions work

    gsl.1.18.5
    gsl.1.19.1 (works)

  • hardcaml-{llvmsim,vpi}: missing depexts

    hardcaml-llvmsim.0.2
    hardcaml-llvmsim.0.3
    hardcaml-vpi.0.1
    hardcaml-vpi.0.2
    hardcaml-vpi.0.3

  • hdfs: missing system library (not defined as a depext, package-specific logic)

    hdfs.0.1

  • hlarp: 0.0.{1,2} fail to build due to deprecation warnings on ‘re’ (Re_Glob => Re.Glob) TODO fix opam-repo; 0.0.3 works

    hlarp.0.0.1
    hlarp.0.0.2
    hlarp.0.0.3 (works)

  • humane-re: 0.0.{1,2} missing ounit dependency? TODO investigate why 0.1.0 builds

    0.0.1
    0.0.2
    0.1.0 (works)

  • imagelib: tarball download error for imagelib_20160413? TODO check

    log

  • io: missing dependency on camlp4; TODO fix opam-repository

    io.0.2

  • javascriptCore: missing system dependency

    log

  • lablqml: Qt dependency / packageconfig issue (it looks like)

  • lambdasoup: archives changed checksum? TODO investigate

    0.5
    0.5.1
    0.6
    0.6.1

  • lilis: bytes/string breakage in oasis; TODO fix in opam-repository

    0.1.3

  • links: appears to be incompatible with OCaml 4.07, due to a fix in the type system which rejects more programs. TODO fix in opam-repository

    0.7.0
    0.7.1

  • llvmgraph: missing or incorrect depext llvm-7-dev; TODO report

    llvmgraph.0.2

  • lwt-zmq.1.0-beta4: incorrect libzmq-dev depext? TODO compare with later versions

    log

  • lwt-zmq.{1,2}.0.0: oasis bytes/string issue; TODO fix opam-repository

    1.0.0
    2.0.0

  • mariadb: 1.0.1 builds correctly, older and newer versions fail; TODO investigate

    0.5.0 (fails)

    1.0.0 (fails)
    1.0.1 (works)
    1.1.0 (fails)
    1.1.1 (fails)

  • maxminddb: system dependency problem (no depext)

    log

  • mirage 2.5.0: bad checksum; TODO investigate (other versions work fine)

    log

  • mirage-qubes.0.2: lib/formats.ml syntax error; TODO investigate, probably a missing OCaml version bound?

    log

  • mirage-qubes.0.3: missing mirage-types-lwt dependency; TODO fix opam-repository

    log

  • mpris.0.1.1: oasis/ocamlbuild interaction bug? TODO investigate

    log

  • msgpck.1.2 bad checksum; TODO investigate

    log

  • nunchaku.0.3.1: missing upper-bound on Containers; WIP(@c-cube) fix opam-repository

    log)

  • obandit: versions <=0.2.1 use stale URLs; TODO fix in opam-repository

    0.1.38 (fails)
    0.1.41 (fails)
    0.1.42 (fails)
    0.2 (fails)
    0.2.1 (fails)
    0.3.4 (works)

  • objsize: download error; TODO investigate

    log

  • ocaml-monadic: missing ocaml version upper bound on 0.1.0 (Asttypes.arg_label is not a string anymore)

    log

  • ocaml-protoc-yojson: looks like missing upper bound on 0.1.0; TODO investigate

    log

  • ocapic.3.3: missing camlp4 dependency; TODO fix in opam-repository

    log

  • ocephes.0.8.1: bytes/string build error, missing version bound on OCaml; TODO fix in opam-repository

    log

  • orocksdb <=0.2.2: the build system gives a non-portable ad-hoc path for system library, fixed in 0.3.0; TODO disable in opam-repository

    0.1.0 (fails)
    0.2.0 (fails)
    0.2.1 (fails)
    0.2.2 (fails)
    0.3.0 (works)

  • osdp 0.5.4: bytes/string error; TODO fix opam-repository

    log

  • owl: “Package `ocaml-compiler-libs’ not found”; TODO investigate

    log

  • ppx_getenv: Parsetree.constant incompatibility, needs OCaml version upper bound and and ocaml-migrate-parsetree support; TODO fix in opam-repository, report upstream

    1.0
    1.1

  • qfs: missing system dependency

    0.4
    0.5
    0.6
    0.7
    0.8

  • qocamlbrowser: checksum error on lablqml?

    0.2.9
    0.2.10

  • rfsm: dead archive link; TODO investigate

    log

  • sodium: 0.4.1<=V<=0.5.0 have incorrect C bindings which fail to build; TODO investigate

    0.2.1 (works)
    0.4.1 (fails)
    0.5.0 (fails)
    0.6.0 (works)

  • sqlite3.3.0.0: missing dependency on pkg-config; DONE(@gasche, PR) fix in opam-repository

    3.0.0 (fails)
    4.0.0 (works)

  • traveka: missing dependency on sosa; TODO fix in opam-repository

    log

  • tsdl, tsdl-image: missing system dependency? DONE (@dbuenzli) investigated

    tsdl.0.9.5 (works)
    tsdl.0.9.6 (fails) (@dbuenzli this is usually due to the fact that depexts don’t allow us to get the right SDL version see this comment)
    tsdl-image (fails)

  • wiringpi: missing system dependency; missing depext or conf-wiringpi package? TODO investigate

    log

  • xen-eventchn: C code in 1.0.6 doesn’t build, 1.0.7 does; TODO disable in opam-repository

    1.0.6 (fails)
    1.0.7 (works)

  • xenctrl 0.9.{29,30,31}: incompatibility in C stubs (xc_domain_create argument number). Probably incompatible with newer C API versions. TODO investigate

    0.9.29
    0.9.30
    0.9.31

  • zlib.0.{4,5}: missing ‘zlib’ depext? DONE(@gasche, PR) fix in opam-repository

    0.4
    0.5

  • zmq <=3.2-2: build error due to C stubs and the ‘uint’ library; zmq 4.0-8 uses ‘stdint’ and works well; TODO report

    3.2-0 (fails)
    3.2-1 (fails)
    3.2-2 (fails)
    4.0-8 (works)

2 Likes

I don’t know if it is the right thing to do but since I’ll have use for a conf-libmaxminddb package in the future I made a PR to add it and it might fix the existing maxminddb package.

qfs and hdfs don’t have packages in e.g. debian default repositories, so I don’t see how that can be fixed on opam side (putting build-from-source instructions for hadoop or libqfs sprinkled with sudo inside opam package is not an option). IMHO proper way to solve this would be along the lines of https://github.com/ocaml/opam/issues/3140 then it would allow to plug in extra apt repositories into CI to satisfy all depexts without hardcoding arbitrary choices into opam packages.

Also, since I am sort-of the maintainer of WiringPi: I actually don’t know how to get the WiringPi.h in place. The upstream of the library that we’re binding to is installing it from a git repo, there is no distribution packages and even if there were some they would only exist on Raspberry Pi, since that’s what upstream is targetting.

No idea on how to get that through Travis successfully.

Thanks to you both for the feedback! We don’t need to get pixel-perfect build, I’ll mark those packages as “investigated”.

I wonder if it would be possible to record that knowledge of untestability somewhere in the opam packages themselves, for future reference. Maybe just a comment in the opam files (but those get rewritten by automatic updates), or as an extra file somewhere in the package hierarchy? @kit-ty-kate (or anyone), would you have a suggestion?

1 Like

@gasche in case you didn’t notice this is also the problem with tsdl:

Regarding:

Maybe it would be worth opening a discussion on the opam issue tracker (if there’s not already one). Meanwhile if you’d like to standardize something asap. Remember that opam v2 supports x-* extension fields.

3 Likes

@dbuenzli: you suggest we could use something like

x-ci-failure-explanation: ["why the build fails on the CI"]

?

(Talking about “the CI” in this way feels a bit ad-hoc, but on the other hand I don’t think that it would appropriate to generalize this to “why the build fails”, which is too unconstrained; hints that are relevant for users or in general non-automated scenarios can already go in the user-visible failure messages.)

Something like that with a bit more machine readable metadata e.g. an optional list of tokens that identifies the failing runs in the test matrix (if that exists).

I’m sure CI freaks like @avsm or @kit-ty-kate have ideas about this.

We could also just decide upon a mark for the ci to prevent a package from being tested as we know the build is expected to fail. The reason, when we really want to add one, can live as a comment

The problem with comments is that they tend to suffer when we use automated tools to rewrite a large number of packages. I’d rather have the information as a text field – but then, the existence of that field could be used by the CI.

1 Like

One suggestion I have is to maintain a branch of opam-repository where the CI adds in a ton of metadata fields to the opam files from its runs (e.g. (x-alpine-failure: """...""")). By doing this in a different branch, it avoids spamming the master branch of the repository, but still makes it easily available with a quick git branch.

The CI could then inspect this other branch in order to get mechanical hints about what to do with a particular opam file.

1 Like

I found more broken revdeps while fixing sqlite3 (traveka, links, broken), so I added them to the list.

I’m happy for mpris.0.1.1 to be removed if it makes things simpler. There’s a version 0.2.0 which is functionally identical but uses dune.