Delay Running "main"

Learning
,
1

It is my understanding that in OCaml doesn’t have a C-like “main” function in the sense that it runs the files top-to-bottom in lexical order, files ordered by the order in which they were linked at build time.

In short
I think I would like to delay running these things top-to-bottom until after a certain condition is met, or otherwise control when ocaml’s “main” is executed in a shared library (in a nice way). I am using the C FFI and Ctypes library. Is this possible?

The long story
I am writing Godot 4.2 bindings for OCaml. The idea is you define a C symbol in a shared library that Godot runs and calls at runtime. For example, I’ve defined hello_extension_entry in an external file as the symbol to use as the entry point to my OCaml extension. This is what it looks like:

#include "gdextension_interface.h"
#include <caml/mlvalues.h>
#include <caml/memory.h>
#include <caml/callback.h>
#include <caml/alloc.h>

GDExtensionBool hello_extension_entry(
    GDExtensionInterfaceGetProcAddress p_get_proc_address,
    GDExtensionClassLibraryPtr p_library,
    GDExtensionInitialization *r_initialization
    ) {
        char** argv_fake = malloc(sizeof(char*) * 2);
        argv_fake[0] = "hello_extension_entry";
        argv_fake[1] = NULL;
        caml_main(argv_fake);
        CAMLparam0();
        CAMLlocal3(get_proc_addr, library, initialization);

        get_proc_addr = caml_alloc(1, Custom_tag);
        *(GDExtensionInterfaceGetProcAddress*)Data_custom_val(get_proc_addr) = p_get_proc_address;

        library = caml_alloc(1, Custom_tag);
        *(GDExtensionClassLibraryPtr*)Data_custom_val(library) = p_library;

        initialization = caml_alloc(1, Custom_tag);
        *(GDExtensionInitialization**)Data_custom_val(initialization) = r_initialization;

        const value * entry = caml_named_value("hello_extension_entry");
        if (entry == NULL) {
            puts("Could not initialize the extension! Returning 0...");
            return 0;
        } else {
            puts("Initializing extension from OCaml!");
            int ret = Int_val(caml_callback3(*entry, get_proc_addr, library, initialization));
            printf("Done initializing from OCaml! Returning %d...\n", ret);
            return ret;
        }
    }

This all works fine: I can link used (native shared_object) and my OCaml main function is called. Great!

The problem arises with that function p_get_proc_address: because OCaml is being loaded as a shared library, you need a portable way to retrieve Godot’s functionality. It’s all from this function: you basically give it strings and it returns functions for you to call, which you C-cast to the correct type. I need to pass this function to OCaml, which is what the above code does.

So far so good.

The problem arises when I want to define foreign functions in terms of p_get_proc_address. I’d like to be able to write something like (and indeed generate the code that looks like this…)

  module ZIPReader = struct
    open! ApiTypes
    include ApiTypes.Object
    include RefCounted

    (** Opens the zip archive at the given [param path] and reads its file index. *)
    (* has type GodotString.t structure ptr -> ZipReader.t structure ptr -> GodotInt.t structure ptr *)
    let open_ =
      foreign_method1 "open"
        (BuiltinClass0.String.typ @-> Class0.ZIPReader.typ
        @-> returning GlobalEnum.Error.typ)
        GlobalEnum.Error.s GlobalEnum.Error.to_variant
        GlobalEnum.Error.of_variant BuiltinClass0.String.to_variant

   <...snip!>

with foreign_method1 looking like

    let foreign_method1 :
        string ->
        ('a -> 'base ptr -> 'r ptr) fn ->
        'r typ ->
        ('r ptr -> variant_ptr structure ptr) ->
        (variant_ptr structure ptr -> 'r ptr) ->
        ('a -> variant_ptr structure ptr) ->
        'a ->
        'base ptr ->
        'r ptr =
     fun method_name _fn ret_typ ret_to_variant ret_of_variant x_to_variant ->
        <...snip!>
        (* set up some stuff, then perform the call with: *)
        let () = variant_call () base string_name arr count ret err in
        let ret = coerce_ptr variant_ptr.plain ret in
        if is_error err then raise (to_exn err) else ret_of_variant ret

where

    let variant_call () = get_fun "variant_call" variant_call.typ

and, finally

let get_proc_address : (string -> InterfaceFunctionPtr.t) ref =
  ref (fun (_ : string) ->
      Stdio.print_endline "get_proc_address -> it does nothing!";
      assert false)

<...snip!>

let get_fun fun_name typ =
      coerce InterfaceFunctionPtr.t typ (!get_proc_address fun_name)

So the plan is this: get the get_proc_address function from C, shove it in the ref, then run all the FFI code like let open_ ... above. The problem is that the FFI code gets run first! Thus, on startup, I get an assert false and the message "get_proc_address -> it does nothing!".

Here is my “entry point” for the ocaml, the function that C calls.

let hello_extension_entry (get_proc_address : nativeint) (_library : nativeint)
    (initialization : nativeint) =
  let open C in
  let initialization =
    coerce (ptr void) (ptr Initialization.s) (ptr_of_raw_address initialization)
  in

  let get_proc_address =
    coerce (ptr void) interface_get_proc_address.typ
      (ptr_of_raw_address get_proc_address)
  in

  (* SET THE REF *)
  Foreign_api.get_proc_address := get_proc_address;

  let initialize (_userdata : unit ptr) (p_level : int) =
    Stdio.print_endline @@ "up: " ^ Base.Int.to_string p_level;

    (* ME TRYING TO FFI DERP *)
    let my_int = Conv.Int.of_ocaml (Int64.of_int 5) in
    let my_other_int = Conv.Int.of_ocaml (Int64.of_int 8) in
    let my_ret_int = Conv.Int.of_ocaml (Int64.of_int 0) in
    BuiltinClass.Int.(my_int + my_other_int) my_ret_int;
    Stdio.printf "%d BANG\n" (Int64.to_int_exn (Conv.Int.to_ocaml my_ret_int))
    (* END DERP *)
  in

  let deinitialize (_userdata : unit ptr) (p_level : int) =
    Stdio.print_endline @@ "down: " ^ Base.Int.to_string p_level
  in

  Initialization.(initialization |-> initialize_f <-@ initialize);
  Initialization.(initialization |-> deinitialize_f <-@ deinitialize);

  1

let () = Stdlib.Callback.register "hello_extension_entry" hello_extension_entry

Okay great. The problem is that BuiltinClass.Int.(+) requires get_proc_address filled in with the passed in function, but it gets defined at when caml_main is called in the original C. This is before the function is passed in, and leads to the errors described above.

Here are some things I’d like to avoid in a solution:

  • Return a big-ass runtime record of functions instead of using modules and “normal” OCaml programming
  • have to initialize each function individually, i.e. let open_ = super_foreign_method1 (* other args here *) () in use_open_here or otherwise fetch them
  • Have to dereference every function by storing them in a ref and using !. I’d like them to look like normal functions, basically.

So I have this chicken-and-egg problem. I need OCaml running when I pass it the get_proc_address function, so caml_main must have been called. At the same time, I want to delay running the top-to-bottom definitions in my library apart from the entry point until after OCaml is running.

Some ideas:

  • plugins?
  • Another shared library that hooks into this one and grabs the ref that gets loaded after (basically a plugin with a plugin-loader)
  • delay the top-to-bottom running of OCaml’s main until after some other OCaml code has run (kind of dubious)

Any one have an idea?

Thanks for coming to my TED Talk.

2

Just as a note, I think the problem was me coming from Haskell-land and not understanding deeply the difference between returning a function and binding it to a value, and defining a function that passes it’s arguments (i.e. eta reduction is not always valid in OCaml). Still interested in answers, but consider this mostly solved.