Ocaml-migrate-parsetree update fails

I have just done an opam update/upgrade and it tries and fails to upgrade ocaml-migrate-parsetree. This is reproducible on two machines on which I have ocaml and opam installed. This is the uninformative response I get;

opam upgrade
The following actions will be performed:
  - upgrade   ocaml-migrate-parsetree 1.7.3 to 1.8.0
  - recompile ppxfind                 1.4
          [uses ocaml-migrate-parsetree]
  - recompile ppx_tools_versioned     5.4.0
          [uses ocaml-migrate-parsetree]
  - recompile ppx_deriving            4.5
          [uses ocaml-migrate-parsetree]
  - recompile lwt_ppx                 2.0.1
          [uses ocaml-migrate-parsetree]
===== 4 to recompile | 1 to upgrade =====
Do you want to continue? [Y/n] y

<><> Gathering sources ><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
[lwt_ppx.2.0.1] found in cache
[ppx_deriving.4.5] found in cache
[ppx_tools_versioned.5.4.0] found in cache
[ppxfind.1.4] found in cache
[ocaml-migrate-parsetree.1.8.0] downloaded from cache at https://opam.ocaml.org/cache

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
[ERROR] The compilation of ocaml-migrate-parsetree failed at
        "/home/chris/.opam/opam-init/hooks/sandbox.sh build dune build -p
        ocaml-migrate-parsetree -j 7".

#=== ERROR while compiling ocaml-migrate-parsetree.1.8.0 ======================#
# context     2.0.7 | linux/x86_64 | ocaml-base-compiler.4.11.1 | https://opam.ocaml.org/#fed9d729
# path        ~/.opam/4.11.1/.opam-switch/build/ocaml-migrate-parsetree.1.8.0
# command     ~/.opam/opam-init/hooks/sandbox.sh build dune build -p ocaml-migrate-parsetree -j 7
# exit-code   1
# env-file    ~/.opam/log/ocaml-migrate-parsetree-113133-d6d332.env
# output-file ~/.opam/log/ocaml-migrate-parsetree-113133-d6d332.out

<><> Error report <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
+- The following actions failed
| - build ocaml-migrate-parsetree 1.8.0
- No changes have been performed

I am a neophyte on the inner workings of opam and I have no idea how to track this down. Does anyone have any ideas? ocaml-migrate-parsetree-113133-d6d332.out is empty.

huh, mh, this is weird, nothing showed up wrong in CI when I merged ocaml-migrate-parsetree.1.8.0 in opam-repository.
I just tried myself and it built fine.
Could you do:

$ opam install -b ocaml-migrate-parsetree.1.8.0
$ cd ~/.opam/4.11.1/.opam-switch/build/ocaml-migrate-parsetree.1.8.0

and see what’s in this directory, it should look like this:

$ ls
_build  CHANGES.md  dune  dune-project  dune-workspace.dev  examples
LICENSE.md  Makefile  MANUAL.md  ocaml-migrate-parsetree.install
ocaml-migrate-parsetree.opam  README.md  src  test  tools

if it’s not empty, try to compile it by hand:

$ dune build -p ocaml-migrate-parsetree

If it succeeds then something might be wrong with the sandbox script. Try to call:

bash -ex ~/.opam/opam-init/hooks/sandbox.sh build dune build -p ocaml-migrate-parsetree

hopefully this allows you to find the culprit.

I have gone through these steps.

'ls' in ~/.opam/4.11.1/.opam-switch/build/ocaml-migrate-parsetree.1.8.0 looks OK with:

_build/ dune-workspace.dev  MANUAL.md src/ CHANGES.md  examples/
ocaml-migrate-parsetree.install test/ dune LICENSE.md
ocaml-migrate-parsetree.opam tools/ dune-project Makefile

dune build -p ocaml-migrate-parsetree succeeds with:

Done: 1155/1159 (jobs: 1)

although 4 jobs are missing - I don’t know if that is significant but there is no error report.

bash -ex ~/.opam/opam-init/hooks/sandbox.sh build dune build -p ocaml-migrate-parsetree gives

+ set -ue
+ command -v bwrap
+ ARGS=(--unshare-net --new-session)
+ ARGS=("${ARGS[@]}" --proc /proc --dev /dev)
+ ARGS=("${ARGS[@]}" --bind "${TMPDIR:-/tmp}" /tmp)
+ ARGS=("${ARGS[@]}" --setenv TMPDIR /tmp --setenv TMP /tmp --setenv TEMPDIR /tmp --setenv TEMP /tmp)
+ ARGS=("${ARGS[@]}" --tmpfs /run --tmpfs /var)
+ add_mounts ro /usr /bin /lib /lib32 /lib64 /etc /opt /home
+ case "$1" in
+ B=--ro-bind
+ for dir in "$@"
+ '[' -d ro ']'
+ for dir in "$@"
+ '[' -d /usr ']'
+ ARGS=("${ARGS[@]}" "$B" "$dir" "$dir")
+ for dir in "$@"
+ '[' -d /bin ']'
+ ARGS=("${ARGS[@]}" "$B" "$dir" "$dir")
+ for dir in "$@"
+ '[' -d /lib ']'
+ ARGS=("${ARGS[@]}" "$B" "$dir" "$dir")
+ for dir in "$@"
+ '[' -d /lib32 ']'
+ for dir in "$@"
+ '[' -d /lib64 ']'
+ ARGS=("${ARGS[@]}" "$B" "$dir" "$dir")
+ for dir in "$@"
+ '[' -d /etc ']'
+ ARGS=("${ARGS[@]}" "$B" "$dir" "$dir")
+ for dir in "$@"
+ '[' -d /opt ']'
+ ARGS=("${ARGS[@]}" "$B" "$dir" "$dir")
+ for dir in "$@"
+ '[' -d /home ']'
+ ARGS=("${ARGS[@]}" "$B" "$dir" "$dir")
+ COMMAND=build
+ shift
+ case "$COMMAND" in
+ '[' -n '' ']'
+ add_mounts ro /home/chris/.opam/4.11.1
+ case "$1" in
+ B=--ro-bind
+ for dir in "$@"
+ '[' -d ro ']'
+ for dir in "$@"
+ '[' -d /home/chris/.opam/4.11.1 ']'
+ ARGS=("${ARGS[@]}" "$B" "$dir" "$dir")
+ add_mounts rw /home/chris/.opam/4.11.1/.opam-switch/build/ocaml-migrate-parsetree.1.8.0
+ case "$1" in
+ B=--bind
+ for dir in "$@"
+ '[' -d rw ']'
+ for dir in "$@"
+ '[' -d /home/chris/.opam/4.11.1/.opam-switch/build/ocaml-migrate-parsetree.1.8.0 ']'
+ ARGS=("${ARGS[@]}" "$B" "$dir" "$dir")
+ add_ccache_mount
+ command -v ccache
+ CCACHE_DIR=/home/chris/.ccache
+ ccache_dir_regex='cache_dir = (.*)$'
+ local 'IFS=
++ ccache --print-config
+ add_mounts rw /home/chris/.ccache
+ case "$1" in
+ B=--bind
+ for dir in "$@"
+ '[' -d rw ']'
+ for dir in "$@"
+ '[' -d /home/chris/.ccache ']'

One reference point is that I have three machines with pretty much the same packages installed. With one of the three the upgrade succeeded. The other two were dealt with in a remote console using ssh, and both failed. However, taking physical possession of one of the failing computers, going to its console and running opam upgrade still fails in the way reported. I would wager it is something to do with sandboxing but I have no idea what.

It was the file ~/.opam/opam-init/hooks/sandbox.sh. Copying the version of that file from the succeeding computer to the unsuccessful ones made the upgrades work on the unsuccessful ones.

So where do these sandbox.sh files come from and why are they out of date? Weird. Updates always worked fine in the past.

Ah I see. That’s indeed an issue with the sandbox script in opam. It seems that the trigger of the issue here is ccache. I suspect that if you uninstall it temporarily it should work. Could you report the issue on https://github.com/ocaml/opam/issues/ ?

I am happy to report the issue on github, but I would like to understand what I am reporting. I have diff’ed the non-working version of sandbox.sh against the working one and there is very little difference between the two - just some differences in the functions dealing with filesystem mounting. Those changes must somehow have resolved the issue but I do not see how.

The thing that changed since opam upgrade was previously succesfully applied and then yesterday unsuccessfully applied to the failing version of sandbox.sh is that ccache has been upgraded from version 3.7.12 to version 4.0. One feature of this is that ccache’s default cache directory has moved from ~/.ccache to ~/.cache/ccache, and I notice that the sandbox.sh scripts appear to expect it at ~/.ccache. This has possibly tripped up the failing version of sandbox.sh but for some reason not the succeeding version.

I still also do not understand where sandbox.sh comes from. I tried moving it and running opam init again but that did not regenerate it with the latest version.

Now that I have a better handle at what was going on I have reported an issue at https://github.com/ocaml/opam/issues/4406