Compiling a shared file, including an external library

Learning
1

I am compiling an Ocaml project (named ‘cody’) which needs to use the ‘batteries’ external module (for UTF8 handling). By the way, I am not using Dune for this project.

If I compile as a standalone with:

ocamlfind ocamlopt -o cody -linkpkg -package batteries cody.ml

there is no problem.

but I need to share this project by creating a *.cmxs file:

ocamlfind ocamlopt -linkpkg -package batteries -shared cody.ml -o cody.cmxs

That also compiles, but trying to load it throws up a Dynlink error when I go to load it:

Dynlink.Error (Dynlink.Cannot_open_dll "Dynlink.Error (Dynlink.Cannot_open_dll \"Failure(\\\"/../cody.cmxs: undefined symbol: camlBatUTF8__look_87\\\")\")")

About the only part of that I understand in the ‘BatUTF8’ which appears as an import in my project.

As a relative newcomer to the OCaml ecosystem, I have no idea what I am doing wrong, and would be very grateful for any pointers.

2

Do you reference BatUTF8.look somewhere in the code of the plugin?

This error means that some code that is required by your plugin has not been linked either to the plugin itself or to the main program loading the plugin. Can you provide the full command-line used to build your main program? Using the -linkall flag (either when linking the plugin or the main program or both) may also be relevant, but may increase the size of your compiled artifacts.

Note that normally for dependencies of plugins (like batteries) you have a choice between linking them to the plugin itself or to the main program. The latter is typically better if the dependencies are shared between plugins; the former is preferrable if the dependencies are only used by the one plugin. At the moment you are linking batteries to the plugin, I believe (by using the -linkpkg flag in the command-line for the .cmxs).

EDIT: also, the output of ocamlobjinfo cody.cmxs may be useful to know what has been linked into the plugin.

Cheers,
Nicolas

1 Like
3

Thanks for the reply. ‘ocamljobinfo’ gives the following output

Interfaces imported:
	6f39f075b573943ab932cfc8fc5bfd5d	Stdlib__Uchar
	4a64380180d45f6d850ce24ab1076b18	Stdlib__Seq
	cbde478960c3a756d3d7c559b25871ed	Stdlib__Bytes
	ae6f92ba6bb2f608ab1494d80d56dc6a	Stdlib__Buffer
	2d082666be7fc2ba916e7233397491df	Stdlib
	c4b583a727ec28f5bc9ba36adc64cfc7	CamlinternalFormatBasics
	57e8734a8a0883d6ac16196325233b72	Batteries
	da435b516b15c4651b8ba92e09827a41	BatUTF8
	adae9039076875a56204566cf097377a	BatUChar
	4e54f9e3d4bf53cd1ab3e5501f4e1ad5	BatOrd
	dbe3ef0bf093df8b41bfce43e4df12e4	BatInterfaces
	4260dcec5b0d9ce654bd597e94550614	BatInnerIO
	76c7ecda51f006036ada27f3f1f885c6	BatEnum
	2fd50b1b43acef962878bac57f2196bc	BatConcurrent
Implementations imported:
	f8b8b4ee0b917ebd74ad56082a6500fe	Stdlib
	ecafee687b6f7c62ccf67031892a6255	BatUTF8

I wasn’t quite accurate before - I had to remove the -linkpkg switch in the compile code because that gives the error.

ImportError: Dynlink.Error (Dynlink.Module_already_loaded "Unix")

Of course, I don’t know whether this error or the previous error is the one that is easier to fix.

There is no BatUTF8.look called in the code, only BatUTF8.validate, BatUTF8.length, BatUTF8.get. Also BatUChar.code .

To load the plugin in my main (Python) program, I use a package called ocaml-in-python, as follows:

import ocaml
ocaml.loadfile("cody.cmxs", "Cody")

I have often used this approach successfully in the past, though never when using an external Ocaml library in the plugin before.

Thank you for looking at this for me.

4

I suspect that you are in the following situation:

  • The main program (your python program using ocaml-in-python) contains a number of libraries already loaded: the standard library, some libraries specific to ocaml-in-python, and (I assume) the unix library.
  • Your plugin is made of your own code (cody), and depends on batteries, which depends on unix.

So if you use -linkpkg, ocamlfind will resolve all libraries needed (except for the standard library) and add them to the linking command. This results in a .cmxs file containing cody, batteries, and unix. This means that you cannot load this file from a program already using unix, as it would lead to duplicate mofules.

If you don’t use -linkpkg, you only get cody, not batteries, and your .cmxs file cannot be loaded on its own.

I see two simple modifications that should enable you to progress:

  • You could manually add batteries.cmxs to your linking command (without -linkpkg). This should produce a cody.cmxs file containing both cody and batteries, but not unix, so you should be able to load it normally
  • You could call ocaml.loadfile("batteries.cmxs", "Batteries") in your python code before loading cody.cmxs (you may need to specify a path for batteries.cmxs, maybe something like "batteries/batteries.cmxs"). I’m not sure it will actually work (it depends on how ocaml.loadfile is implemented, and I’m not familiar with ocaml-in-python).
2 Likes
5

Many thanks for your helpful and insightful analysis of this. Plenty to think about there, such as how to locate batteries.cmxs

I will give your solutions a try and report back.

6

Just to complement @vlaviron’s explanation, I think it should be batteries.cmxa, not batteries.cmxs. If you are using ocamlfind you should be able to just write batteries.cmxa in the command-line. The -I flags needed to find that file will have been added by ocamlfind in response to the -package batteries argument.

Cheers,
Nicolas

7

Right.

I have tried 3 methods:

  1. A load procedure that appears in the ocaml-in-python documentation,
  2. Locating and manually adding the batteries.cmxs file
  3. Locating and manually adding the batteries.cmxa file

Method 1:

import ocaml
ocaml.require("batteries")

that gave me the error:

ImportError: Dynlink.Error (Dynlink.Cannot_open_dll "Dynlink.Error (Dynlink.Cannot_open_dll \"Failure(\\\"/~/.opam/4.13.0/lib/batteries/batteries.cmxs: undefined symbol: caml_mutex_lock\\\")\")")

Now, that should not happen, as the documentation suggests that is the way to do it.

Method 2: gave me the identical error as above.

Method 3: (using the .cmxa file) gave the error:

ImportError: Dynlink.Error (Dynlink.Cannot_open_dll "Dynlink.Error (Dynlink.Cannot_open_dll \"Failure(\\\"/~/.opam/4.13.0/lib/batteries/batteries.cmxa: invalid ELF header\\\")\")")

This means that my way forward would have to be @vlaviron first suggestion, which is to add batteries.cmxs into the Ocaml compile statement somehow.

Needless to say, I am not sure how that would look compared to my current compile statement, which is:

ocamlfind ocamlopt -package batteries -shared cody.ml -o cody.cmxs

Any help would be gratefully received - even if there is no solution to this, you are still helping me find my way around the Ocaml ecosystem.

8

What I had in mind was passing adding batteries.cmxa to your command-line (without -linkpkg):

ocamlfind ocamlopt -package batteries batteries.cmxa -shared cody.ml -o cody.cmxs

and loading it as before.

Could you try that and report if it works?

Cheers,
Nicolas

9

I tried that, but still got the original error …

ImportError: Dynlink.Error (Dynlink.Cannot_open_dll "Dynlink.Error (Dynlink.Cannot_open_dll \"Failure(\\\"~~/Batteries/cody.cmxs: undefined symbol: camlBatUTF8__look_87\\\")\")")

If I also do the manual load, the error changes to the … undefined symbol: caml_mutex_lock\\\")\")")

The real problem here is my appalling ignorance of this ecosystem - the error messages mean nothing to me, and I can’t even be sure if this is an ocaml-in-python issue (though I assume it is).

I suppose that as a last resort, I could simply take the BatUTF8.ml file from the package and use it in my code. It’s not the ideal way, but I guess it may get the job done.

Thanks for the help, anyway. I am open to further suggestions …

EDIT: At least I can see where the problem Dynlink error of loading modules more than once is coming from - it was a patch applied several years ago to solve some other issues. They also talk about how it might affect plugins:

However we believe these problems can be solved by such systems using their own caching layer that can catch the new error value returned (Dynlink.Error(Module_already_loaded ...)).

Not sure I can do anything to solve that, though.

10

It is hard to go further without access to the code, but can you try adding -linkall to the command-line and retry?

ocamlfind ocamlopt -linkall -package batteries batteries.cmxa -shared cody.ml -o cody.cmxs

Cheers,
Nicolas

11

Thanks again for the help, although that tweak still gave me the :

undefined symbol: caml_mutex_lock\\\" error

I guess it’s something to do with ocaml-in-python.

To get round this, I have written my own small UTF8 decoder, turns out that is trivial once you know the code points.

12

Note that recent version of the standard library comes with UTF8 and UTF16 decoders: OCaml library : String .

13

Thanks. Yes, I looked at that, at it didn’t seem to do what I wanted.

I am interfacing between a PythonQT GUI front-end an an OCaml back-end.

Python treats UTF8 strings very effectively, that is, getting the character at position 4 will be the actual Unicode character, whereas in OCaml the same function gives back the 4th byte.

So I had to write small Ocaml routines to expose functions that behave like Python would. OCaml really shines in this department, and the implementation is trivial - 15 lines of code.