How to compile and link a Fortran library component with dune

Ecosystem
, ,
1

I’m wondering if I am compiling and linking a simple one-file Fortran sublibrary correctly into my OCaml library. Ideally I would like to be able to make my library installable on linux and macos with a simple opam install.

After some fiddling and googling I have arrived at the project structure shown below, which seems to work for loading the Fortran library via ctypes like so:

integrate_mvnd.ml

open Containers
open Ctypes
[...]
let mvnd =
  match
    Sys.getenv "CAML_LD_LIBRARY_PATH"
    |> String.split_on_char ':'
    |> List.cons (Sys.getcwd ())
    |> List.find_map (fun p ->
        try
          let filename = p ^ "/dllmvnd.so" in
          Some (Dl.dlopen ~flags:Dl.[RTLD_NOW;] ~filename)
        with Dl.DL_error _ -> None)
  with
  | Some l -> l
  | None -> failwith "unable to load mvnd shared library"

let ocaml_mvnd arg1 arg2 [...] = 
   Foreign.foreign ~from:mvnd "mvnormaldist" (ptr.integer @-> [...])

Some points I’m not content with:

  • I have to manually check for the CAML_LD_LIBRARY_PATH
  • I have to build a static library although afaict this will never work the way the library is loaded, because otherwise dune complains
  • I am using hardcoded dll filenames, but when trying @{dll_ext} in dune that also gave .so on macos not .dylib as i had expected.

Am I using dune correctly? Is there a way to switch to either fully static linking of the fortran sublibrary or only dynamic linking?

Relevant project structure bits follow:

dune:

(data_only_dirs fortran-lib)

(library
	(name vbar)
	(public_name vbar)
	(modules vbar)
	(libraries [...]))

(rule
	(deps (source_tree fortran-lib))
	(targets dllmvnd.so libmvnd.a)
	(action
		(no-infer
			(progn
				(chdir fortran-lib (run make))
				(copy fortran-lib/dllmvnd.so dllmvnd.so)
				(copy fortran-lib/libmvnd.a libmvnd.a)))))

(library
	(name integrate_mvnd)
	(public_name vbar.integrate_mvnd)
	(modules integrate_mvnd)
	(libraries [...] ctypes ctypes.foreign)
	(foreign_archives mvnd))

[...]

fortran-lib/Makefile:

.PHONY: clean all

all: mvnd.dylib libmvnd.a

mvnd.dylib: 
	gfortran -shared -O3 -fPIC -fcheck=bounds -std=f2008 -w -o dllmvnd.so mvndstpack_mod.f

libmvnd.a: 
	gfortran -c -O3 -fPIC -fcheck=bounds -std=f2008 -w mvndstpack_mod.f
	ar -rcs libmvnd.a mvndstpack_mod.o 

clean:
	rm -f *.o *.mod *.a *.dylib
2

I may be missing something, but I don’t see why you need to load your library dynamically. If you use foreign_archives, the Fortran library should have been linked statically into your final executable (at least on native code).

Cheers,
Nicolas

3

Ok, so how would this look? no ctypes and

external ocaml_mvnd : (int -> float -> [...] -> ()) = "mvnormaldist"

?
The Fortran function takes integer pointer and double pointer arguments and writes the result into one of the double pointers. I was under the impression that I would have to write a C stubs file if I want external functions but maybe that’s wrong?
Or can I use ctypes to build these pointers to pass to the external function and read them out later?
(clearly I don’t grok how external functions and/or ctypes work)

4

You probably still want ctypes to bind the functions, but you shouldn’t need the ~from:... argument.

Cheers,
Nicolas

5

This is something I had tried. If I remove let mvnd = [...] and also remove the ~from argument but leave everything else the same, I get a runtime error: exception Dl.DL_error("dlsym(RTLD_DEFAULT, mvnormaldist): symbol not found") . If i remove only ~from but leave the dynamic loading in place, with unused resulting value mvnd, then the symbol is found – presumably due to the mere act of loading the library.

6

Are you using bytecode or native-code?

Cheers,
Nicolas

7

I was trying to dune exec a native code executable (.exe) when this error was produced. i haven’t specified any special compilation/linking mode fields for the executables so i think native is chosen by default

8

It looks like the symbols from the Fortran lib are being dropped by the linker because they are not referenced from the main program (they are only being addressed dynamically using dlsym/ctypes). How to keep this from hapenning is rather OS-dependent (I don’t remember how to do it off the top of my head).

Cheers,
Nicolas

9

Ah thanks that could be an explanation. this reminds me of https://github.com/ocaml/ocaml/issues/10018 . Is it related?

10

I think the root issue is the same in both cases.

Cheers,
Nicolas

11

I’ve tried a little more. I remembered that -all_load can be used on macos to load unreferenced symbols, so changed:

(library
	(name integrate_mvnd)
	(public_name vbar.integrate_mvnd)
	(modules integrate_mvnd)
	(libraries util ctypes ctypes.foreign)
	(foreign_archives mvnd)
	(flags (:standard -cclib -all_load)))

and tried to compile. This time, the linker complains it cannot find __gfortran_runtime_error_at which seems to be a gfortran internal symbol. Also adding -cclib -lgfortran does not find a corresponding library. On my system I have libgfortran.dylib but as it appears no libgfortran.a. (I got this lead from https://askubuntu.com/questions/581905/undefined-references-to-gfortran-runtime-error-at)

Anyway, the static compilation option seems more painful than the dynamic library loading. I would be content with making the latter work without also producing a static library. Can I just leave out the (foreign_archives) field?

12

Yes, indeed.

Cheers,
Nicolas

13

hi, this seemed to work at first but it turns out to be not quite right. i tried this:

(rule
	(deps (source_tree fortran-lib))
	(targets dllmvnd.so libmvnd.a)
	(action
		(no-infer
			(ignore-stdout 
				(progn
					(chdir fortran-lib (run make))
					(copy fortran-lib/libmvnd.a libmvnd.a)
					(copy fortran-lib/dllmvnd.so dllmvnd.so))))))

(library
	(name integrate_mvnd)
	(public_name vbar.integrate_mvnd)
	(modules integrate_mvnd)
	(foreign_archives mvnd)
	(libraries util ctypes ctypes.foreign))

which installs both the static and dynamic libraries. if i leave out foreign_archives neither get installed. if i keep foreign_archives but prevent building of the static library in the Makefile and in the rule listed first, then dune complains that the static lib is missing – apparently foreign_archives wants the static library.

i can’t seem to convince dune that only a dynamic library should be produced and then installed.