Dune, dynlink, and undefined symbol

Learning
,
1

Hello.

I have this interface for plugins to register plugins in my program:

let ht : (string, (string * (_ -> _ -> bool))) Hashtbl.t  = Hashtbl.create 0
let assets = ref ""
let register ~ns m fn = let fn = fn !assets in Hashtbl.add ht m (ns, fn)
let get = Hashtbl.find_opt ht

Then I have the main executable loading plugins this way:

let register_plugin dir =
  let assets = Filename.concat dir "assets" in
  GwdPlugin.assets := assets ;
  Secure.add_lang_path assets ;
  Dynlink.loadfile (Filename.concat dir "plugin.cmxs") ;
  GwdPlugin.assets := ""

But I have a lot of problems when trying to use libraries which are not included in my main program.

I tried to use embed_in_plugin_libraries and -linkall. Each time I get un undefined symbol, I add the corresponding lib in embed_in_plugin_libraries.

(executable
  (name plugin)
  (libraries aaa)
  (embed_in_plugin_libraries bbb ccc ddd)
  (flags -linkall)
  (modes (native plugin))
)

aaa si the library of my main program (plugin registration + the rest of its world), bbb ccc ddd libraries not used in the main program.

But now, I am stuck because the undefined symbol is actually

undefined symbol: camlStdlib__lexing__engine_113
  • Is it possible for my main program to link the whole stdlib without the compiler removing ? It is ok for me to state that the main program to embed unused code if it is only the stdlib (maybe this would be sufficent to solve the issue)
  • Is it possible for my plugin to embed stdlib as I did it with other libs ? Can’t find a package for this.
  • Am I just doing it wrong?
2

I can add a dummy module with include of all stdlib’s modules in my main program which seems to solve this problem, but is it what I should do?

3

Ok, I managed to get my plugin running:

Added findlib.dynload as main program dependency, and:

Called this in main program

let () = Findlib.init ()

Used this in my plugin

  Fl_dynload.load_packages ~debug:true dependencies (* hand-written hardcoded deps *)

And finally, using Dynlink on my file passed as command line argument to my program

Dynlink.loadfile (Filename.concat dir "plugin.cmxs") ;

So far, it works, which is okay, even if I am not confident at all about portability and reliability of this.

4

The last time Idid this sort of thing (which was a good long while ago), it was with bytecode. And I remember that there was something I had to do, to ensure that modules were linked that were needed by any libraries I wanted to Dynload. Maybe that’s what you’ve done with your hack. [I realize that you’re doing this with native code.] Does it work when you try all this with bytecode?

5

A few general remarks:

  • typically you would use, either Dynlink, or Findlib. Findlib is a an extra layer that takes care of loading plugin dependencies recursively.
  • for your “stdlib” problem, you should add -linkall to the link_flags of the main program.
  • If you want to use Findlib, then you can use it to load your main plugin as well, by putting the Findlib.load_packages call in the main program (in lieu of Dynlink.loadprivate).

Cheers,
Nicolas

6

Thanks.

I will try with the -linkall option for my main program.

And I think that I can not use findlib in the final product since my use case is to distribute binary (main program) and plugins without the whole ocaml environnement. Also the reason why I can not use bytecode and need native plugins.

So, I’ll try to linkall my main program and to use embed_in_plugin_libraries for my plugins (which leads me to an error of already loaded module, if I rember correctly, but I need to double check this)

7

This field is meant for libraries that need to be statically linked into your plugin (ie which are not linked into the main program). In other words, libraries that are used simultaneously from several plugins should be linked into the main program, and not put in this field.

Cheers,
Nicolas

8

I see. Thanks for all these explanations.

In other words, libraries that are used simultaneously from several plugins should be linked into the main program, and not put in this field.

This restriction may make building independent plugins very difficult. I am trying to make mutliple simple plugins in order to export libs one by one. Is it completely insane or should it work?

$ cat plugins/jingoo/dune
(executable
  (name plugin_jingoo)
  (embed_in_plugin_libraries jingoo)
  (flags -linkall)
  (modes (native plugin))
)

$ cat plugins/jingoo/plugin_jingoo.ml
let () = ()

$ cat plugins/re/dune
(executable
  (name plugin_re)
  (embed_in_plugin_libraries re)
  (flags -linkall)
  (modes (native plugin))
)

$ cat plugins/re/plugin_re.ml
let () = ()

And I finally launch my program with:

_build/default/bin/prog.exe -plugin _build/default/plugins/re/plugin_re.cmxs -plugin _build/default/plugins/jingoo/plugin_jingoo.cmxs -plugin _build/default/plugins/gwxjg/plugin_gwxjg.cmxs -plugin _build/default/plugins/v8/plugin_v8.cmxs

And it works, but as I said, I feel like it is completely insane to do so.
The dependencies hell and the extra work required by this workflow is ok for me, as I planned to have a package manager to handle it for me anyway (or as I do not really care about having to add all that options).

9

The (flags -linkall) is a bit fishy. If anything, it should be (link_flags -linkall), but my guess is that you don’t want to put this flag in at all in your “plugin” stanzas.

There is an alternative approach to building plugin systems in dune with automatic dependency handling, which does not depend on findlib, called “sites”. It may or may not be suitable for your use-case: you can read more about it here: dune/doc/sites.rst at main · ocaml/dune · GitHub

cc @bobot

Cheers,
Nicolas

10

I will try without the flag (and actually did not know about link_flags) but I thought that compiler might eliminate library’s code if not using it (since my plugins do absolutly nothing).

Nerver heard about this! I will have a look, thanks.

11

The sites are new in the futur 2.8, there is also a dune install --relocatable --prefix=<dir> mode for installing the executable, the librairies and the plugins without everything else inside a relocatable directory <dir>. However all the libraries that are used by the plugins but not used by the binaries should be in the dune workspace (or you can copy by hand the installed libraries inside <dir> ).