Dune: depend on non-Dune libraries in the workspace

Learning
,
1

Hi,

I have a Dune-project (an application) A that depends on a third library L, this library is located in the workspace of the project, but it is built with a Make script (not using Dune).

The project tree is similar to:

A
|_ dune-project
|_ src
  |_ dune
  |_ ...
|_ L
  |_ src
    |_ Makefile

I would like my application to depend on this non-Dune library. I have been looking into the docs and I have tried a number of approaches without much success…

My preferred choice would be to package the library as a Dune-project, while still building it using the existing Make script. I tried following the Foreign build sandboxing idea without success:

(library
  (name L))

(rule
  (deps (source_tree src))
  (targets L.cmxa)
  (action
    (progn
      (chdir src (run make))
      (copy src/L.cmxa L.cmxa)
      )))

This doesn’t work because the “library” stanza (which is mandatory for declaring a library) must produce “L.cmxa” but it will not know how. I specify a custom rule that produces “L.cmxa”, but this rule conflicts with the “library” stanza.

Another option would be using a pre-compiled “L.cmxa” when building the application A. I could not find the way to specify such dependency. It seems that Dune can only make use of pre-compiled libraries when they are in the “installed world”. I tried copying the .cmxa file with (copy_files ../L/src/L.cmxa) but one cannot copy files from outside the current directory. Still, if I copy the L.cm* files of the library into A/src/lib, and then add (copy_files lib/L.cm*) to A/src/dune, this doesn’t work either (it seems that no file is being copied into _build/default/src).

I would appreciate any recommendations, hints, or pointers to examples.

Thanks in advance!

2

Hi @iago, the foreign build sandboxing is only meant for writing stubs in foreign languages such as C, not for OCaml libraries. What you are trying to do is currently not supported by Dune. The usual way to mix Dune and non-Dune libraries is to build and install the various libraries separately. Indeed, Dune is able to use globally installed libraries that were not built with Dune, however it is not able to build such libraries itself.

Adding support for this scenario would be possible, however it is a non-negligible amount of work. Have you tried porting the third library to Dune? The process is usually quite straightforward.

3

Thank your for you answer @jeremiedimino.

It’s not obvious to me how to build my library L with Dune, although I can believe it can be done. That is why I was trying to just “sandbox” it.

Let me rename L as mylib… the structure of mylib is something like:

mylib/
|_ c/
  |_ Makefile
  |_ ...
|_ ...
|_ ocaml/
  |_ Makefile
  |_ mylib_stubs.c
  |_ mylib.ml
  |_ mylib.mli
  |_ ...

The sources of the native C library are in mylib/c/ and the OCaml bindings in mylib/ocaml/.

I have given it a try at building mylib with Dune adding the following mylib/dune file:

(include_subdirs unqualified)

(ignored_subdirs (c))

(library
  (name mylib)
  (public_name mylib)
  (c_library_flags -ccopt -fPIC -L. -lmylib)
  (c_names ocaml/mylib_stubs))

This almost works but ocamlmklib cannot find mylib. This is expected because it is found in c/libmylib.a… I would need to get the library file copied to the build directory (e.g. _build/default), but I can’t manage to get that to work. I can add a rule like:

(rule
  (targets libmylib.a)
  (action (progn
    (chdir c (run make))
    (copy c/libmylib.a libmylib.a))))

But how do I force this rule to run?

4

The OCaml library containing the stubs can definitely be built with Dune. Regarding the native C library in mylib/c you have two choices:

  • if the library is simple enough, i.e. just a bunch of C files compiling in a straightforward way, then you can try to remove the c_library_flags field and simply add the various C files to c_names
  • if not, you should sandbox the build of the the C library only and pack as its own OCaml library, for instance mylib.c and then make mylib depend on mylib.c

To do the former, you can effectively follow the foreign build sandboxing. Technically you need to add something like this in mylib/dune:

(library
 (name mylib_c)
 (public_name mylib.c)
 (modules)
 (self_stubs_build_archive mylib_c_stubs)

(rule
 (deps (source_tree c))
 (targets libmylib_c_stubs.a dllmylib_c_stubs.so)
 (action (progn
          (chdir c (run make)))
          (copy libxxx.a libmylib_c_stubs.a)
          (copy libxxx.so dllmylib_c_stubs.so)))

Note that the filenames of the .a and .so files must be of the form libXXX_stubs.a and dllXXX_stubs.so. We could extend dune so that the intermediate mylib_c library is not necessary, however this is not supported at the moment.

2 Likes
5

Does copy now work? Last time I had tried that I had to shell out to copy the artifact

6

copy should definitely work, do you remember what didn’t work in your case?

7

The issue was the one discussed here: Strange error in dune building

8

Thanks again for the hints @jeremiedimino.

A small thing I noticed is that (copy c/libmylib.a libmylib_c_stubs.a) introduces a dependency on mylib/c/libmylib.a. I can see this dependency in the output of dune rules mylib/libmylib_c_stubs.a:

…
(In_build_dir default/mylib/c/libmylib.a)
…

And indeed Dune complains if I build from a clean workspace:

Error: No rule found for mylib/c/libmylib.a

So I am forced to make -C mylib/c before invoking Dune although this step is already part of the rule that generates libmylib_c_stubs.a and dllmylib_c_stubs.so.

Is there any way to work around this?

9

Ah yes, that’s the same issue @mseri pointed. We need to do something about this in dune. In the meantime, the simplest solution is to put the call to make and the various files copy in a shell script rather than use the DSL. For instance, you can write a shell script containing:

make -C c
cp c/libxxx.a libmylib_c_stubs.a
cp c/libxxx.so dllmylib_c_stubs.so

and call this shell script.

10

I created an issue to track this in GitHub instead: https://github.com/ocaml/dune/issues/2006