I have an OCaml/C library that I’m trying to build and package on OPAM. I’m completely lost. For reference, the library is libdash, but I’ll try to keep things abstract.
The library has three parts:
A C library in src built using libtool/autotools
OCaml bindings for that library in ocaml
An OCaml tester in test
The OCaml bindings use ctypes to interface with the C library. Building the C library generates a static library as well as some dynamically loadable shared libraries.
I have a build workflow that functions just fine when run manually, but I have not yet been able to craft a .opam file to manage that workflow. The C library and the OCaml bindings build just fine. The core issue I’ve encountered is that I’m unsure about where/how to install the C libraries so that they are visible when building clients, like the tester.
If I install the C library globally, I have no problem. If I install the files into %{_:lib}%, then my clients can build but they can’t run. If I install the files in %{stublibs}%, then the clients can run but not build can’t even build (EDIT: I can’t reproduce the original behavior here). I could install in one and then setup symlinks, but that doesn’t seem quite right.
@avsm has helpfully provided a reference to ocaml-yaml, which seems to have a pretty similar build structure, but I don’t really know how to read the dune build files there.
What’s the right thing to do here? Am I missing a magic incantation? Should I just link or copy the libraries? Should I give up on writing my own build and try to use dune to magic things up in the background? Is ctypes the wrong approach, and I should just statically link with the C code? (If so, is there a good reference on how to work with C structs and pointers from OCaml?)
(I’d be happy to add more detail here, i.e., show concrete scripts and errors, if that’s helpful.)
I’ve made some progress on this on my own. I install the C libraries in stublibs, generating a META file for the library with the following included:
linkopts="-ccopt -L$(opam var stublibs) -cclib -ldash"
At this point, I can now build everything. But when I try to run a program using the bindings, I get a linker error when running the natively compiled code:
$ ./test.native
./test.native: error while loading shared libraries: libdash.so.0: cannot open shared object file: No such file or directory
If I set LD_LIBRARY_PATH to stublibs, it works. The bytecode compiled version works on its own (presumably because CAML_LD_LIBRARY_PATH includes the stublibs). What am I missing?
I can’t answer your questions, but my general feeling is that you are trying to fix a problem by doing things manually that Dune should do for you automatically.
Maybe there is a simpler way to do this in Dune, but it’s not documented well (I’m assuming you looked at the C-libraries part of the Dune documentation, in the Quickstart and in the dedicated chapter on foreign libraries) and there should be a better documentation and a boilerplate example to point at.
Maybe there isn’t currently a way to do what you need (but from a distance it seems very close to the “sandboxed builds” scenario from the Dune documentation on foreign library), and Dune should be improved to cover it.
With this analysis, my recommendation would be to open an issue on the Dune issue tracker, repeating your basic setup and pointing to the present forum thread. (I could also start cc-ing random Dune contributors in the hope that they would answer, but letting them handle this choice through the issue tracker sounds better.)
Per @gasche, what I didn’t see was where the linking work was being done. I was hoping there’d be some concrete ruling (put the shared libraries in .opam/.../foo or something). But the vibe I’m getting is that I shouldn’t be doing that linking work myself, and I should just try converting my build to dune and seeing what happens. Fair enough! I’ll give it a whack and report back.
No worries. The library is parser and AST bindings for the dash shell, for use in my own POSIX shell (smoosh). I am impossibly old fashioned, but it looks like I need to get with the times.
Maybe you’ve already solved your problem, in which case, please ignore.
If not, and if you have a repo someplace where I could clone+build, and a clear description of the failure, I’d be happy to have a look. I’ve done a bunch of projects where I used OcamlMakefile and (more recently) autoconf, and lots/lots/lots of C/C++ bindings.
I’ve built your project and hopefully repro-ed your problem. For concreteness:
chet@twitter:~/Hack/Ocaml-Hacking/src/libdash/test$ make
ocamlfind ocamlopt -g -package libdash -linkpkg test.ml -o test.native
/usr/bin/ld: cannot find -ldash
/usr/bin/ld: cannot find -ldash
collect2: error: ld returned 1 exit status
File "caml_startup", line 1:
Error: Error during linking
Makefile:17: recipe for target 'test.native' failed
make: *** [test.native] Error 2
So this is a build-time problem, not a run-time problem, yes?
ETA: I recall now that you’ve got a run-time problem, but I clearly cannot repro that. Could you provide more … “explicit” instructions on how to repro? E.g. starting from a git checkout of the “opam” branch (after a “git clean -xf”) ?
I have a version building but not testing using in the dune branch. I’ve adapted my tester to fit in line with dune’s conventions (a singular executable that does all the testing, rather than a driver and a shell script that runs it). My current issue there is a build failure:
I’m a little bit baffled by this, since I just ran dune build and dune install on the rest of the library, after making sure that my .opam area there is fully cleaned out. Any thoughts? Reinstall ctypes and ctypes.foreign?
I’ve had both build and runtime problems. Currently on the opam branch, I have a build error that I haven’t seen before:
$ git clone ...
$ git checkout opam
$ TMPDIR="/" opam install . # TMPDIR is necessary on my mac
[libdash.0.1] no changes from git+file:///Users/mgree/libdash#opam
The following actions will be performed:
∗ install libdash 0.1*
<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><> 🐫
[ERROR] The compilation of libdash failed at "/Users/mgree/.opam/opam-init/hooks/sandbox.sh build make install".
#=== ERROR while compiling libdash.0.1 ========================================#
# context 2.0.5 | macos/x86_64 | ocaml-variants.4.07.1+flambda | pinned(git+file:///Users/mgree/libdash#opam#cf201df3)
# path ~/.opam/4.07.1+flambda/.opam-switch/build/libdash.0.1
# command ~/.opam/opam-init/hooks/sandbox.sh build make install
# exit-code 2
# env-file ~/.opam/log/libdash-4366-b76d46.env
# output-file ~/.opam/log/libdash-4366-b76d46.out
### output ###
# Making install in src
# /Applications/Xcode.app/Contents/Developer/usr/bin/make install-am
# .././install-sh -c -d '/Users/mgree/.opam/4.07.1+flambda/lib/stublibs'
# /bin/sh ../libtool --mode=install /usr/bin/install -c libdash.la dlldash.la '/Users/mgree/.opam/4.07.1+flambda/lib/stublibs'
# libtool: install: /usr/bin/install -c .libs/libdash.0.dylib /Users/mgree/.opam/4.07.1+flambda/lib/stublibs/libdash.0.dylib
# install: /Users/mgree/.opam/4.07.1+flambda/lib/stublibs/libdash.0.dylib: Operation not permitted
# make[3]: *** [install-libLTLIBRARIES] Error 71
# make[2]: *** [install-am] Error 2
# make[1]: *** [install] Error 2
# make: *** [install-recursive] Error 1
<><> Error report <><><><><><><><><><><><><><><><><><><><><><><><><><><><><> 🐫
┌─ The following actions failed
│ λ build libdash 0.1
└─
╶─ No changes have been performed
This is a novel build issue to me, but I guess the OPAM sandbox doesn’t want just anybody copying things in there? The funny thing is that this process has worked before, but not now. Perhaps earlier it was overwriting files I had put there myself… maybe?
(1) it would be good to stick to a single target – a single “known bad” – with as few moving parts as possible. So (if possible) no docker, opam, dune, etc, involved in the build process. Your original thing with just Makefiles was good for this. [I’ll admit that if you were talking with somebody who knew dune intimately, they’d probably for that, and rightly so]
(2) I figured that getting a “known good” would be useful. So I did the following (just to check):
However, given that you’re using ctypes, this might not be enough – it’s been forever since I tried to use ctypes, so I don’t know.
ETA: in case it’s useful: “rpath” is used by the runtime linker to know where to look for shared libraries (when they’re not installed in a standard location, and those directories aren’t listed in one of several standard files). It’s different from -L and -l, which are link-time directives. yeah: “so complicated”. But at least, it’s got nothing to do with Ocaml: this is just the way C shared-libs work.
This is helpful, thanks. I remember playing with -rpath when I was first getting things going, and that looks like it might be the way forward for the pure Makefile solution.
On the other hand, the dune solution seems quite close—I suspect I’m just missing something.
I’m going to abandon the dune-based approach because I have something working and am already several weeks behind on my goals/plans. I suspect that some well-placed incantations in a dune file will reproduce something equivalent to the fix I have here, though. If someone wants to play with what I have at https://github.com/mgree/libdash/tree/dune, please go ahead!.
I messed up when testing this solution. It doesn’t work either. Both native and bytecode executables will build, but they won’t run without an LD_LIBRARY_PATH. I’m unsure what to change here…
I’ll give it a whack and report back.