How does Dune search libraires during the build process of itself?

Ecosystem
1

Question

Could somebody tell me how libraries (such as pp) are searched during the build process of Dune itself? When I attempt to build Dune as a system-wide binary on Gentoo, I encounter an internal error (details below). I haven’t reported this to Dune’s repository yet since I’m using a non-standard environment; Portage manages system-wide OCaml libraries and binaries while opam manages them in $HOME/.opam.

Here is the error message I received during the build:

Done: 1480/1542 (jobs: 1)cd _boot && /usr/bin/ocamlopt.opt -o ../_boot/dune.exe -g -I +threads unix.cmxa threads.cmxa readdir.o wait4_stubs.o signal_stubs.o platform_stubs.o copyfile_stubs.o xdg_stubs.o sha512_stubs.o sha256_stubs.o sha1_stubs.o custom_opamStubs.o spawn_stubs.o dune_stats_stubs.o dune_flock.o dune_digest_stubs.o csexp_rpc_stubs.o fsevents_stubs.o inotify_stubs.o fswatch_win_stubs.o winsize.o -args compiled_ml_files -alert -unstable
Internal error, please report upstream including the contents of _build/log.
Description:
  ("Unexpected find result", { found = Not_found; lib.name = "pp" })
Raised at Stdune__Code_error.raise in file
  "otherlibs/stdune/src/code_error.ml", line 10, characters 30-62
Called from Fiber__Scheduler.exec in file "vendor/fiber/src/scheduler.ml",
  line 76, characters 8-11
-> required by ("<unnamed>", ())
-> required by ("<unnamed>", ())
-> required by ("<unnamed>", ())
-> required by ("load-dir", In_build_dir "default/test/blackbox-tests/utils")
-> required by ("toplevel", ())

I must not crash.  Uncertainty is the mind-killer. Exceptions are the
little-death that brings total obliteration.  I will fully express my cases. 
Execution will pass over me and through me.  And when it has gone past, I
will unwind the stack along its path.  Where the cases are handled there will
be nothing.  Only I will remain.

Full log

Background

I’m using Gentoo Linux, where Portage, a source-based package manager, compiles software packages from source code. Dune’s internal libraries (e.g., dune-private-libs) are currently built with the vendored csexp and pp in Portage, which contradicts dune-private-libs.opam that instructs removing these packages before the actual build. This leads to the “make inconsistent assumptions” error if some package depends on both csexp and Dune’s internal libraries that also depend on csexp (e.g., dune-private-libs; see my post on Gentoo forum.)

To address this, I modified the ebuilds (the build script for Portage) for Dune’s internal libraries to remove csexp and pp directories in the vendor directory before the build (rm vendor/{csexp,pp}, see my PR on the Gentoo repository). While this modification allowed the internal libraries to build successfully, the Dune build subsequently failed with the internal error mentioned above.

Since my environment is unusual as it has OCaml libraries installed both system-wide (managed by Portage like /usr/lib64/ocaml/stdune/stdune.cmi) and in $HOME/.opam (managed by opam), I haven’t reported this problem to Dune’s repository. Instead, I want to understand the library search mechanism during the build process of Dune, which might help me debug and resolve this problem.

The full log is here. I’m sorry, but I couldn’t get _build/log.

Dune’s ebuild is here. The installation process is as follows where $(makeopts_jobs) is an arbitrary number specified by the user:

./configure \
	--libdir="$(ocamlc -where)" \
	--mandir=/usr/share/man \
	--docdir=/usr/share/doc \
	--etcdir=/etc \
	--datadir=/usr/share \
	--sbindir=/usr/sbin \
	--bindir=/usr/bin
ocaml boot/bootstrap.ml -j $(makeopts_jobs) --verbose
./_boot/dune.exe build @install -p dune --profile dune-bootstrap \
	-j $(makeopts_jobs) --display short

The internal libraries are built with the following command (see the ebuild for dune-private-libs. Note that it simultaneously compiles and installs ordering, dyn, and stdune for some reasons.)

dune build @install --display=short --profile release -j $(makeopts_jobs) --for-release-of-packages=(package names)
2

Hello @toku-sa-n,

I think the issue that you may be seeing comes from the fact that Dune, for bootstrapping reasons, is built using a “mini-Dune” which is not able to look for libraries outside of the local workspace (see dune/boot/libs.ml at main · ocaml/dune · GitHub).

Cheers,
Nicolas

3

Thank you for your response. I still find it strange that Dune is unable to find its vendored pp for the following reasons:

  1. Based on the code you provided, Dune seems to expect its vendored libraries to reside within its root directory. I confirmed that the source directory (vendor/pp/src) and source files were present both before and after building Dune.
  2. I haven’t modified the build process of the Dune package (dev-ml/dune) itself; only the build process of its internal library packages (dev-ml/dune-private-libs, etc.) has changed. Furthermore, before these modifications, building dev-ml/dune completed successfully without any issues.
  3. Although I didn’t mention this in the original post, dev-ml/dune does not depend on the packages for its internal libraries (dev-ml/dune-private-libs, etc.). Therefore, modifying dev-ml/dune-private-libs should not affect the build process of dev-ml/dune.