Who adds -shared when building ocaml-protoc-yojson?

Ecosystem
1

I’ve done great progress towards building truly statically linked binaries with OCaml. In a nutshell:

  • Switch to 4.10.0+musl+static+flambda;
  • Install Musl on my host, in my case version 1.2.2 which is > 1.1.24 I’ve seen mentioned related to static building;
  • Symlink from /usr/include/linux/ to missing /usr/include/x86_64-linux-musl/linux/;
  • Tell Dune to (ocamlopt_flags (:standard -ccopt -static));

Unfortunately, some packages like ppx_deriving and ocaml-protoc-yojson fail to install in static switches. I’d like to understand why and if it could be worked around, so I dug into ocaml-protoc-yojson's case. Here’s the output from opam:

#=== ERROR while compiling ocaml-protoc-yojson.0.2.0 ==========================#
# context     2.0.7 | linux/x86_64 | ocaml-variants.4.10.0+musl+static+flambda | https://opam.ocaml.org#4a7aed82
# path        ~/.opam/4.10.0+musl+static+flambda/.opam-switch/build/ocaml-protoc-yojson.0.2.0
# command     ~/.opam/opam-init/hooks/sandbox.sh build make lib.native
# exit-code   2
# env-file    ~/.opam/log/ocaml-protoc-yojson-9716-6d07ae.env
# output-file ~/.opam/log/ocaml-protoc-yojson-9716-6d07ae.out
### output ###
# ocamlbuild -use-ocamlfind -pkgs yojson  -I src -I tests pbrt-yojson.cmxa
# ocamlfind ocamlopt -c -g -annot -bin-annot -safe-string -no-alias-deps -keep-locs -w A-4-42-41 -package yojson -I src -I tests -o src/pbrt_yojson.cmx src/pbrt_yojson.ml
# ocamlfind ocamlopt -a -keep-locs -package yojson -I src src/pbrt_yojson.cmx -o src/pbrt-yojson.cmxa
# ocamlbuild -use-ocamlfind -pkgs yojson  -I src -I tests pbrt-yojson.cmxs
# ocamlfind ocamlopt -shared -linkall -keep-locs -package yojson -I src src/pbrt-yojson.cmxa -o src/pbrt-yojson.cmxs
# + ocamlfind ocamlopt -shared -linkall -keep-locs -package yojson -I src src/pbrt-yojson.cmxa -o src/pbrt-yojson.cmxs
# sh: 1: shared-libs-not-available: not found
# File "caml_startup", line 1:
# Error: I/O error: shared-libs-not-available -o 'src/pbrt-yojson.cmxs'  '-Lsrc' '-L/home/lis/.opam/4.10.0+musl+static+flambda/lib/easy-format' '-L/home/lis/.opam/4.10.0+musl+static+flambda/lib/biniou' '-L/home/lis/.opam/4.10.0+musl+static+flambda/lib/yojson' '-L/home/lis/.opam/4.10.0+musl+static+flambda/lib/ocaml'  'src/pbrt-yojson.cmxs.startup.o' 'src/pbrt-yojson.a'
# Command exited with code 2.
# make: *** [Makefile.opamlib:17: lib.native] Error 10

I grepped the repo of ocaml-protoc-yojson and there is no -shared in any file. Did opam (2.0.7) add that option to ocamlopt somehow? If so, why would it do that in a static switch?

If anyone managed to work around this issue, I’d love to know how. :smiley:

1 Like
2

I’m not sure what you’re trying to achieve with -shared… but I’d just like to point out I got success building mixed ocaml/C/C++ on Alpine (from a docker container), where musl is the default libc. There’s nothing special to do other than passing -ccopt -static to the ocaml compiler. In particular:

  • no need to select a special opam switch
  • finding the C++ standard library worked out of the box (unlike on ubuntu, which is where I gave up when I tried)

See for example how it’s done for dune-deps, whose CI build runs on a docker image created with ocaml-layer. You can just run docker run -it mjambon/ocaml:alpine and cat /Dockerfile to see the setup. Demo:

$ docker run -it mjambon/ocaml:alpine
Running "eval $(opam env)", which initializes PATH and other variables.

In scripts and dockerfiles, dont forget to run your commands as
  opam exec -- COMMAND
or equivalently as
  eval $(opam env) && COMMAND

bash-5.0$ opam switch
#   switch   compiler                    description
->  4.10.0   ocaml-base-compiler.4.10.0  4.10.0
    default  ocaml-system.4.08.1         default
bash-5.0$ cat > toto.ml
Printf.printf "%f\n" (Unix.gettimeofday ())

bash-5.0$ ocamlopt -o toto -ccopt -static unix.cmxa toto.ml
bash-5.0$ ldd toto
	/lib/ld-musl-x86_64.so.1 (0x7f12214ae000)
bash-5.0$ file toto
toto: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), statically linked, with debug_info, not stripped
bash-5.0$ ./toto
1612212056.085955

If you don’t want a position-independent executable (PIE) static binary (not sure when this is needed), on Alpine you have to pass -ccopt -no-pie:

bash-5.0$ ocamlopt -o toto-no-pie -ccopt -static -ccopt -no-pie unix.cmxa toto.ml
bash-5.0$ ldd toto-no-pie 
/lib/ld-musl-x86_64.so.1: toto-no-pie: Not a valid dynamic program
bash-5.0$ file toto-no-pie 
toto-no-pie: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, with debug_info, not stripped

There are no issues with installing the packages you mentioned:

bash-5.0$ opam install ppx_deriving ocaml-protoc-yojson
...
<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
-> installed easy-format.1.3.2
-> installed biniou.1.2.1
-> installed ocaml-compiler-libs.v0.12.3
-> installed ocaml-migrate-parsetree.2.1.0
-> installed ppx_derivers.1.2.1
-> installed sexplib0.v0.14.0
-> installed stdlib-shims.0.1.0
-> installed ppxlib.0.21.0
-> installed ppx_deriving.5.2
-> installed yojson.1.7.0
-> installed ocaml-protoc-yojson.0.2.0
Done.
2 Likes
3

Thanks for this detailed response! FYI I’m not the author of ocaml-protoc-yojson, I’m just wondering why there’s a -shared in opam’s error log since it’s not in the project’s Makefile. That said:

I think this means that I was mistaken about the meaning of those +static switches to begin with. Those seem to be about the compiler’s own building and not its output. I’ll try the same with 4.11.1+musl+flambda directly on my host (no Docker) as a test… yup, everything installs fine! Building statically… yup:

$ file basics.exe
basics.exe: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, with debug_info, not stripped
$ ldd basics.exe
        not a dynamic executable

So the lesson here is that one needs a +musl build, not +static, to build statically (in addition to -ccopt -static of course).

1 Like