Packaging foreign archives with bindings using ctypes

mgree · December 20, 2023, 2:02pm

I’m working on libdash, which is offers OCaml and Python bindings to the dash POSIX shell parser. dune is being uncooperative around packaging the library up properly (i.e., with both bindings and code). My work is in the ~~ocaml-static-ctypes branch~~ main branch (since OCaml bindings don’t work right now anyway).

The structure is:

an autotools build constructs libdash.a and dlldash.so from the (lightly modified dash source)
some static ctypes bindings connects the C interface to OCaml
a bunch of OCaml converts the C ASTs into more sensible structures
some executables use the OCaml ASTs to do some work

A critical property here is that the C library is not a common one, and must be distributed with the OCaml source. The produced OCaml native library should include all of the C code.

dune woes

I—after several years of trying off and on—have gotten dune to build the project locally and have tests pass. (I posted here three years ago.) But there’s a problem: the build fails with multiple uses of libdash.a. Here’s the output:

$ dune build
Error: Multiple rules generated for
_build/install/default/lib/libdash/libdash.a:
- ocaml/dune:8
- ocaml/dune:8
-> required by _build/default/libdash.install
-> required by alias all
-> required by alias default

The offending clause:

(library
  (name libdash)
  (public_name libdash)
  (modes native)
  (modules (:standard \ json_to_shell shell_to_json ast_json))
  (libraries ctypes ctypes.foreign)
  (foreign_archives ../dash)
  (ctypes
    (external_library_name dash)
    (build_flags_resolver (vendored (c_flags :standard) (c_library_flags :standard)))
    (deps (glob_files ../src/*.h) ../src/builtins.h ../src/nodes.h ../src/syntax.h ../src/token.h ../src/token_vars.h)
    (headers (preamble
              "\
             \n#include \"../src/shell.h\"\
             \n#include \"../src/memalloc.h\"\
             \n#include \"../src/mystring.h\"\
             \n#include \"../src/init.h\"\
             \n#include \"../src/main.h\"\
             \n#include \"../src/input.h\"\
             \n#include \"../src/var.h\"\
             \n#include \"../src/alias.h\"\
             \n#include \"../src/redir.h\"\
             \n#include \"../src/parser.h\"\
             \n#include \"../src/nodes.h\"\
             \n"))
    (type_description
      (instance Types)
      (functor Type_description))
    (function_description
      (instance Functions)
      (functor Function_description))
    (generated_types Types_generated)
    (generated_entry_point Cdash)))

In particular, the problem is my use of (foreign_archives ../dash). This, however, is the key line that makes the executables work. Without it, the linker fails when building the executables and it can’t find libdash’s symbols.

linking woes

The build works fine if I move (foreign_archives ../dash) to the executables, but then there’s a different problem: others trying to link with libdash won’t get the actual libdash.a and those clients will fail at link time with missing symbols. I’ve tried using an install stanza to make sure libdash.a is in (section lib), but it has no effect on libraries that then try to link with dash.

what should i do?

The library is currently built with custom ocamlfind commands that once worked but no longer work on Linux. They’re fine on macOS; it’s a similar linking issue.

My feeling is that this is a bad interaction between ctypes and dune, and I should simply be able to do this. (I don’t see the issue of the multiple dependency on libdash.a, which should indeed be used more than once. That’s what libraries are for!) But I would not be the first person to have wrongly felt “I should simply be able to do this”! So… what do I do?

I see a few ways forward, in order of preference:.

Figure out the magic incantation that lets dune actually package the C code with the library.
Give up on Ctypes entirely and interface with C more directly, allowing me to use foreign_archives with the library. (This is the approach taken by re2.)
Give up on OCaml bindings (I am the only client; everyone else uses Python—and I could too, at the cost of some communication overhead).
Create a conf-libdash that does the C library installation and have the OCaml bindings depend on that.

I put (4) last because it is a remarkable amount of faffing about for something that should be really very simple… making me more inclined to take a performance hit in order to no longer need to interact with OCaml (which has been incredibly frustrating in its interaction with C… my last post on this was in ~~2021~~ 2020).

lukstafi · December 21, 2023, 1:16pm

Noob opinion / question: if (4) conf-libdash solves problems, what’s remarkably bad about it?

mgree · December 21, 2023, 1:33pm

I have two objections to conf-libdash:

I would need to figure out pkg-config and depexts for the C library, and I don’t want to do that.
I don’t want to manage two OPAM packages for what is conceptually a single thing.

If I were going to do things that make me unhappy, I would simply drop the dependency on OCaml. A bummer—but a much more simplifying one.

emillon · December 21, 2023, 2:05pm

I’ll try to have a better look at this later, but it’s possible that it’s a bug in the ctypes support in dune (see this one I fixed recently).\

This is 404, FYI.

A couple suggestions in the meantime:

(libraries ctypes ctypes.foreign)

You probably don’t need ctypes.foreign if you’re using (ctypes) as this method uses stub generation, not libffi. Actually, ctypes might be superfluous too since it’s implied by (ctypes).

(external_library_name dash)

This one might cause an issue, actually. I’m not completely familiar with that part of dune yet, but it’s possible that when used in conjunction with (build_flags_resolver vendored), this name is actually not used, and would clash with foreign_archives above. You can try dash2 here.

(build_flags_resolver (vendored (c_flags :standard) (c_library_flags :standard)))

You can replace that with just (build_flags_resolver vendored).

(headers (preamble
          "\
         \n#include \"../src/shell.h\"\

Does it work with (headers (include ../src/shell.h ../src/memalloc.h)) ? (make sure to update to (using ctypes 0.3) to ensure that relative paths are handled correctly). if this work you should be able to remove the (deps ...) above.

As for the approach, packaging the library within the ocaml project like this is a valid approach. You can also take inspiration from ocaml-yaml which vendors the C yaml library and uses ctypes to bind to it on the ocaml side. It does not use dune’s support for ctypes, though. So that’s an in-between you can try first.

Also, note that (ctypes) is still a bit experimental and likely to change. As you’ve noted, this is a bit too invasive and we’re likely to add a lighter mechanism where you define a toplevel (ctypes ...) to define how a module is generated, independently of any (library) or (executable), a bit like things like ocamllex or menhir are exposed. Your feedback is very valuable.

mgree · December 21, 2023, 2:16pm

Thank you for the response and detailed analysis!

Fixed the link—I merged to main because the OCaml build is broken anyway. I currently have a build that generates working executables but unusable libraries. (My build canary has been failing for… a long time.)

When I use (include ...) it generates includes with angle brackets, which confused the C compiler.

I spent a while trying to understand ocaml-yaml, but I had trouble getting it so I could use my type definitions in my bindings. They seem to be using casts through void *, at which point I don’t know why I’d bother with ctypes at all. Could be user error—it seems very fiddly and I might have messed something up.

What do you mean by “you can try dash2 here?” Do you mean copying libdash.a into libdash2.a somewhere? I tried something like that (probably not exactly what you’re suggesting) and the compiler had trouble finding the right bindings.

My overall feedback is that dune feels like a “total” approach. Works great when everything is in OCaml, but less great when it encounters the outside world. dune believes it knows how C projects are compiled, but the landscape of C builds is… quite rich.

mgree · December 21, 2023, 2:34pm

I created a branch where I think I’ve implemented what you intended. Doesn’t work for building the library, sadly.

$ dune build --verbose
Shared cache: disabled
Shared cache location: /home/mgreenbe/.cache/dune/db
Workspace root: /home/mgreenbe/libdash
Auto-detected concurrency: 12
Dune context:
 { name = "default"
 ; kind = "default"
 ; profile = Dev
 ; merlin = true
 ; fdo_target_exe = None
 ; build_dir = In_build_dir "default"
 ; installed_env =
     map
       { "INSIDE_DUNE" : "/home/mgreenbe/libdash/_build/default"
       ; "OCAML_COLOR" : "always"
       ; "OPAMCOLOR" : "always"
       }
 ; instrument_with = []
 }
Actual targets:
- alias @@default
Error: Multiple rules generated for
_build/install/default/lib/libdash/libdash.a:
- ocaml/dune:8
- ocaml/dune:8
-> required by _build/default/libdash.install
-> required by alias all
-> required by alias default

Topic		Replies	Views
OPAM/packaging questions around OCaml bindings to custom C library Ecosystem ffi , build , package , ctypes , linking	15	1654	February 17, 2020
[ANN] ctypes 0.21.1, now with dune support Community announce , ffi , dune	0	403	July 24, 2023
Advanced C binding using ocaml-ctypes and dune Learning build , ctypes , dune , bindings	14	2820	April 21, 2022
I am looking for an example project with dune and ctype Learning dune	12	414	November 21, 2024
Dune project with a C/C++ ctypes foreign defined function Learning ctypes	2	768	July 27, 2023

Packaging foreign archives with bindings using ctypes

dune woes

linking woes

what should i do?

Related topics