Linking error loading dune toplevel with `camlpdf`

Ecosystem
1

Hello friends,

I’m trying to use Dune with camlpdf and I’m running into trouble getting a project-aware repl. dune build works just fine, but then I get screwy linking error when I try to run dune utop:

> opam exec -- dune utop
File "_build/default", line 1, characters 0-0:
/usr/bin/ld: cannot find -lcamlpdf_stubs: No such file or directory
/usr/bin/ld: cannot find -lxdg_stubs: No such file or directory
/usr/bin/ld: cannot find -llambda_term_stubs: No such file or directory
/usr/bin/ld: cannot find -llwt_unix_stubs: No such file or directory
collect2: error: ld returned 1 exit status
File "_none_", line 1:
Error: Error while building custom runtime system

Manually loading packages with findlib in the REPL the old-fashioned way works:

> rlwrap ocaml
OCaml version 4.14.2
Enter #help;; for help.

# #use "topfind";;
# ****************
# - : unit = ()
Findlib has been successfully loaded. Additional directives:
  #require "package";;      to load a package
  #list;;                   to list the available packages
  #camlp4o;;                to load camlp4 (standard syntax)
  #camlp4r;;                to load camlp4 (revised syntax)
  #predicates "p,q,...";;   to set these predicates
  Topfind.reset();;         to force that packages will be reloaded
  #thread;;                 to enable threads

- : unit = ()
# #require "camlpdf";;
# ********************
# /home/teichman/.opam/ocaml-basics/lib/camlpdf: added to search path
/home/teichman/.opam/ocaml-basics/lib/camlpdf/camlpdf.cma: loaded
# #show Pdf;;
...etc...

I tried getting dune to tell me the compiler command it ran:

> opam exec -- dune utop --verbose
Shared cache: enabled-except-user-rules
Shared cache location: /home/teichman/.cache/dune/db
Workspace root: /home/teichman/tmp/aaargh-ld-problem
Auto-detected concurrency: 28
Dune context:
 { name = "default"
 ; kind = "default"
 ; profile = Dev
 ; merlin = true
 ; fdo_target_exe = None
 ; build_dir = In_build_dir "default"
 ; instrument_with = []
 }
Running[1]: (cd _build/default && /home/teichman/.opam/ocaml-basics/bin/ocamlc.opt -w @1..3@5..28@31..39@43@46..47@49..57@61..62@67@69-40 -strict-sequence -strict-formats -short-paths -keep-locs -w -24 -g -o .utop/utop.bc /home/teichman/.opam/ocaml-basics/lib/ocaml/compiler-libs/ocamlcommon.cma /home/teichman/.opam/ocaml-basics/lib/ocaml/compiler-libs/ocamlbytecomp.cma /home/teichman/.opam/ocaml-basics/lib/ocaml/compiler-libs/ocamltoplevel.cma /home/teichman/.opam/ocaml-basics/lib/findlib/findlib.cma /home/teichman/.opam/ocaml-basics/lib/findlib/findlib_top.cma /home/teichman/.opam/ocaml-basics/lib/logs/logs.cma /home/teichman/.opam/ocaml-basics/lib/lwt/lwt.cma /home/teichman/.opam/ocaml-basics/lib/logs/lwt/logs_lwt.cma /home/teichman/.opam/ocaml-basics/lib/ocaml/unix.cma /home/teichman/.opam/ocaml-basics/lib/ocaml/bigarray.cma /home/teichman/.opam/ocaml-basics/lib/ocplib-endian/ocplib_endian.cma /home/teichman/.opam/ocaml-basics/lib/ocplib-endian/bigstring/ocplib_endian_bigstring.cma /home/teichman/.opam/ocaml-basics/lib/ocaml/threads/threads.cma /home/teichman/.opam/ocaml-basics/lib/lwt/unix/lwt_unix.cma -I /home/teichman/.opam/ocaml-basics/lib/lwt/../stublibs /home/teichman/.opam/ocaml-basics/lib/react/react.cma /home/teichman/.opam/ocaml-basics/lib/lwt_react/lwt_react.cma /home/teichman/.opam/ocaml-basics/lib/result/result.cma /home/teichman/.opam/ocaml-basics/lib/uutf/uutf.cma /home/teichman/.opam/ocaml-basics/lib/uucp/uucp.cma /home/teichman/.opam/ocaml-basics/lib/uuseg/uuseg.cma /home/teichman/.opam/ocaml-basics/lib/zed/zed.cma /home/teichman/.opam/ocaml-basics/lib/trie/trie.cma /home/teichman/.opam/ocaml-basics/lib/mew/mew.cma /home/teichman/.opam/ocaml-basics/lib/mew_vi/mew_vi.cma /home/teichman/.opam/ocaml-basics/lib/lambda-term/lambda_term.cma -I /home/teichman/.opam/ocaml-basics/lib/lambda-term/../stublibs /home/teichman/.opam/ocaml-basics/lib/xdg/xdg.cma -I /home/teichman/.opam/ocaml-basics/lib/xdg/../stublibs /home/teichman/.opam/ocaml-basics/lib/utop/uTop.cma /home/teichman/.opam/ocaml-basics/lib/camlp-streams/camlp_streams.cma /home/teichman/.opam/ocaml-basics/lib/ocaml/str.cma /home/teichman/.opam/ocaml-basics/lib/prelude/prelude.cma /home/teichman/.opam/ocaml-basics/lib/etude/etude.cma /home/teichman/.opam/ocaml-basics/lib/camlpdf/camlpdf.cma lib/lib.cma .utop/.utop.eobjs/byte/dune__exe__Utop.cmo -linkall -warn-error -31)
File "_build/default", line 1, characters 0-0:
Command [1] exited with code 2:

The errors that command throws when I run it are the same as the errors I get running dune utop. And I found that when I added -a with an absolute path to libcamlpdf_stubs.a to the (cd _build/default && etc. command and ran the command directly in my shell, it succeded! So theoretically, if I could get Dune to do the same thing, dune utop would work?

I created a dummy project on GitHub to demonstrate the problem, in case anyone wants to try reproducing it:

Anyway, I’d really like to be able to use dune utop the way I would in one of my normal OCaml projects. Does anyone know of a way I can get dune utop to run the right command to link with libcamlpdf_stubs.a etc. without blowing up? It would be a drag to have to change my normal build-a-custom-toplevel-with-dune workflow for this project.

Thanks as always!
-Matt

2

I haven’t looked in detail, but note that loading libraries with native code in a toplevel requires to have a version of the C stubs built as a shared library (.so under Linux, eg dllcamlpdf_stubs.so). Sometimes library authors forget to install the shared version of the stubs and only install the static version instead (.a, used for native-code compilation). I would start by checking if the shared version is available.

You can also look at the output of dune ocaml top to get an idea of where it is looking for the shared stubs.

Cheers,
Nicolas

1 Like
3

Aha, yeah, thanks very much for the suggestions! I have looked at this a bit. So all the errant libraries are in the “stublibs” directory of my switch:

> ls ~/.opam/ocaml-basics/lib/stublibs | grep camlpdf
dllcamlpdf_stubs.so
dllcamlpdf_stubs.so.owner

xdg, lwt_unix, lambda-term, etc. are also in there. I also noticed that the command that dune seems to be running to create the toplevel tells the compiler about this directory containing all the *.so files in one of the many arguments to -I.

4

Update! I discovered that if I create a custom toplevel using one of dune’s toplevel stanzas, rather than with dune utop, I do not get these linking errors! Wow. I’m not locked into using utop, and in fact I slightly prefer the regular OCaml toplevel, so this seems ideal as a solution.

In case this is helpful to others in the same situation, here’s my lib/dune file:

(library
 (name lib)
 (libraries prelude etude camlpdf))

(toplevel
 (name repl)
 (libraries prelude etude camlpdf lib))

(env
 (dev
  (flags (:standard -warn-error -A))))

Then to get a REPL:

$ opam exec -- dune build
$ opam exec -- dune exec lib/repl.exe
OCaml version 4.14.2
Enter #help;; for help.

# #show Lib;;
module Lib : sig module Example_module = Lib.Example_module end
# #show Pdf;;
... etc. ...

Documentation for toplevel stanzas:

5

I took a closer look, I believe the issue is that camlpdf is built using the “force custom” flag:

$ opam exec -- ocamlobjinfo ~/.opam/5.3.0/lib/camlpdf/camlpdf.cma | grep custom
Force custom: YES

This implicitly forces the command used by Dune to link the toplevel to be done in “custom mode” (ie by creating a custom runtime with all the C code statically linked). The problems is that Dune is not expecting this and does not pass the necessary -I flags that are required to find the static libraries to begin with.

Camlpdf can probably be built without the “force custom” flag set, probably something to discuss with its author. cc @JohnWhitington

Cheers,
Nicolas

2 Likes
6

Aha! Thank you so much. This explains everything, including (I think!) why a custom toplevel worked fine.

7

Was also not familiar with ocamlobjinfo. Looks quite useful!