I am compiling an Ocaml project (named ‘cody’) which needs to use the ‘batteries’ external module (for UTF8 handling). By the way, I am not using Dune for this project.
If I compile as a standalone with:
ocamlfind ocamlopt -o cody -linkpkg -package batteries cody.ml
there is no problem.
but I need to share this project by creating a *.cmxs file:
ocamlfind ocamlopt -linkpkg -package batteries -shared cody.ml -o cody.cmxs
That also compiles, but trying to load it throws up a Dynlink error when I go to load it:
Dynlink.Error (Dynlink.Cannot_open_dll "Dynlink.Error (Dynlink.Cannot_open_dll \"Failure(\\\"/../cody.cmxs: undefined symbol: camlBatUTF8__look_87\\\")\")")
About the only part of that I understand in the ‘BatUTF8’ which appears as an import in my project.
As a relative newcomer to the OCaml ecosystem, I have no idea what I am doing wrong, and would be very grateful for any pointers.
Do you reference
BatUTF8.look somewhere in the code of the plugin?
This error means that some code that is required by your plugin has not been linked either to the plugin itself or to the main program loading the plugin. Can you provide the full command-line used to build your main program? Using the
-linkall flag (either when linking the plugin or the main program or both) may also be relevant, but may increase the size of your compiled artifacts.
Note that normally for dependencies of plugins (like
batteries) you have a choice between linking them to the plugin itself or to the main program. The latter is typically better if the dependencies are shared between plugins; the former is preferrable if the dependencies are only used by the one plugin. At the moment you are linking
batteries to the plugin, I believe (by using the
-linkpkg flag in the command-line for the
EDIT: also, the output of
ocamlobjinfo cody.cmxs may be useful to know what has been linked into the plugin.
Thanks for the reply. ‘ocamljobinfo’ gives the following output
I wasn’t quite accurate before - I had to remove the
-linkpkg switch in the compile code because that gives the error.
ImportError: Dynlink.Error (Dynlink.Module_already_loaded "Unix")
Of course, I don’t know whether this error or the previous error is the one that is easier to fix.
There is no
BatUTF8.look called in the code, only
To load the plugin in my main (Python) program, I use a package called ocaml-in-python, as follows:
I have often used this approach successfully in the past, though never when using an external Ocaml library in the plugin before.
Thank you for looking at this for me.
I suspect that you are in the following situation:
- The main program (your python program using ocaml-in-python) contains a number of libraries already loaded: the standard library, some libraries specific to ocaml-in-python, and (I assume) the unix library.
- Your plugin is made of your own code (cody), and depends on batteries, which depends on unix.
So if you use
-linkpkg, ocamlfind will resolve all libraries needed (except for the standard library) and add them to the linking command. This results in a
.cmxs file containing cody, batteries, and unix. This means that you cannot load this file from a program already using unix, as it would lead to duplicate mofules.
If you don’t use
-linkpkg, you only get cody, not batteries, and your
.cmxs file cannot be loaded on its own.
I see two simple modifications that should enable you to progress:
- You could manually add
batteries.cmxs to your linking command (without
-linkpkg). This should produce a
cody.cmxs file containing both cody and batteries, but not unix, so you should be able to load it normally
- You could call
ocaml.loadfile("batteries.cmxs", "Batteries") in your python code before loading
cody.cmxs (you may need to specify a path for
batteries.cmxs, maybe something like
"batteries/batteries.cmxs"). I’m not sure it will actually work (it depends on how
ocaml.loadfile is implemented, and I’m not familiar with ocaml-in-python).
Many thanks for your helpful and insightful analysis of this. Plenty to think about there, such as how to locate
I will give your solutions a try and report back.
Just to complement @vlaviron’s explanation, I think it should be
batteries.cmxs. If you are using
ocamlfind you should be able to just write
batteries.cmxa in the command-line. The
-I flags needed to find that file will have been added by
ocamlfind in response to the
-package batteries argument.
I have tried 3 methods:
- A load procedure that appears in the ocaml-in-python documentation,
- Locating and manually adding the batteries.cmxs file
- Locating and manually adding the batteries.cmxa file
that gave me the error:
ImportError: Dynlink.Error (Dynlink.Cannot_open_dll "Dynlink.Error (Dynlink.Cannot_open_dll \"Failure(\\\"/~/.opam/4.13.0/lib/batteries/batteries.cmxs: undefined symbol: caml_mutex_lock\\\")\")")
Now, that should not happen, as the documentation suggests that is the way to do it.
Method 2: gave me the identical error as above.
Method 3: (using the .cmxa file) gave the error:
ImportError: Dynlink.Error (Dynlink.Cannot_open_dll "Dynlink.Error (Dynlink.Cannot_open_dll \"Failure(\\\"/~/.opam/4.13.0/lib/batteries/batteries.cmxa: invalid ELF header\\\")\")")
This means that my way forward would have to be @vlaviron first suggestion, which is to add
batteries.cmxs into the Ocaml compile statement somehow.
Needless to say, I am not sure how that would look compared to my current compile statement, which is:
ocamlfind ocamlopt -package batteries -shared cody.ml -o cody.cmxs
Any help would be gratefully received - even if there is no solution to this, you are still helping me find my way around the Ocaml ecosystem.
What I had in mind was passing adding
batteries.cmxa to your command-line (without
ocamlfind ocamlopt -package batteries batteries.cmxa -shared cody.ml -o cody.cmxs
and loading it as before.
Could you try that and report if it works?
I tried that, but still got the original error …
ImportError: Dynlink.Error (Dynlink.Cannot_open_dll "Dynlink.Error (Dynlink.Cannot_open_dll \"Failure(\\\"~~/Batteries/cody.cmxs: undefined symbol: camlBatUTF8__look_87\\\")\")")
If I also do the manual load, the error changes to the …
undefined symbol: caml_mutex_lock\\\")\")")
The real problem here is my appalling ignorance of this ecosystem - the error messages mean nothing to me, and I can’t even be sure if this is an
ocaml-in-python issue (though I assume it is).
I suppose that as a last resort, I could simply take the
BatUTF8.ml file from the package and use it in my code. It’s not the ideal way, but I guess it may get the job done.
Thanks for the help, anyway. I am open to further suggestions …
EDIT: At least I can see where the problem Dynlink error of loading modules more than once is coming from - it was a patch applied several years ago to solve some other issues. They also talk about how it might affect plugins:
However we believe these problems can be solved by such systems using their own caching layer that can catch the new error value returned (
Not sure I can do anything to solve that, though.
It is hard to go further without access to the code, but can you try adding
-linkall to the command-line and retry?
ocamlfind ocamlopt -linkall -package batteries batteries.cmxa -shared cody.ml -o cody.cmxs
Thanks again for the help, although that tweak still gave me the :
undefined symbol: caml_mutex_lock\\\" error
I guess it’s something to do with
To get round this, I have written my own small UTF8 decoder, turns out that is trivial once you know the code points.
Note that recent version of the standard library comes with UTF8 and UTF16 decoders: OCaml library : String .
Thanks. Yes, I looked at that, at it didn’t seem to do what I wanted.
I am interfacing between a PythonQT GUI front-end an an OCaml back-end.
Python treats UTF8 strings very effectively, that is, getting the character at position 4 will be the actual Unicode character, whereas in OCaml the same function gives back the 4th byte.
So I had to write small Ocaml routines to expose functions that behave like Python would. OCaml really shines in this department, and the implementation is trivial - 15 lines of code.