Cargo/Opam packaging of a Rust/OCaml project

Ecosystem
, , ,
9

Sorry for bumping this old thread, but it seems the solution is still to be found for this problem.

I’ve achieved some success binding asynchronous Rust to asynchronous OCaml (Lwt flavour, but that’s technicality), and have it open-sourced (see ocaml-lwt-interop project).

But it’s very challenging right now to use this in your own libraries. Currently established workflow of statically linking everything Rust-related into large .a that is linked to OCaml binary in the end does not scale beyond proprietary codebase in a monorepo.

There is a discussion on this matter in this ocaml-rs issue. Turns out that Rust is trying to formalize some official approach to separate compilation with external linking (see rust-lung#73632), but it seems that it won’t help us much as the best strategy is to pull all Rust sources for each binary into a single staticlib which will be linked into that binary.

Ideally some tool should scan all the dependencies of an OCaml binary, and for each OCaml library which embeds a Rust crate as foreign stubs, that tool should compose a top level crate for that binary, which should look something like this:

src/lib.rs:

pub use ocaml_lib1;
pub use ocaml_lib2;
pub use ocaml_lwt_interop;

Cargo.toml:

[package]
name = "ocaml-rust-deps"
version = "0.1.0"
edition = "2021"

[lib]
crate-type = ["staticlib", "cdylib","rlib"]
path = "src/lib.rs"

[dependencies]
ocaml-lib1 = { path = "_opam/lib/rust-lib1" }
ocaml-lib2 = { path = "_opam/lib/rust-lib2" }
ocaml-lwt-interop = { path = "some-local-dir/ocaml-lwt-interop" }

and dune:

(rule
 (targets libocaml_rust_deps.a dllocaml_rust_deps.so)
 (deps
  (glob_files_rec ../*.rs)
  (glob_files_rec ../*.toml)
  ../Cargo.lock
  ../.cargo/config
  (source_tree ../vendor))
 (action
  (no-infer
   (progn
    (chdir
     %{workspace_root}
     (run
      cargo
      build
      --target-dir
      %{workspace_root}/../../_build_rust
      --release
      --offline
      --package
      ocaml-rust-deps))
    (run
     mv
     %{workspace_root}/../../_build_rust/release/libocaml_rust_deps.a
     libocaml_rust_deps.a)
    (run
     mv
     %{workspace_root}/../../_build_rust/release/libocaml_rust_deps.so
     dllocaml_rust_deps.so)))))

(library
 (name rust_deps_lib)
 (foreign_archives ocaml_rust_deps)
 (c_library_flags
  (-lpthread -lc -lm)))

(Ugly hack to move rust build dir into the source directory is to speed up the build, dune will remove it on any changes if it’s inside the build dir it seems).

Resulting rust_deps_lib just needs to be included into the binary. This approach works and is future-proof it seems.

Open question is how such tool can be build and what kind of tool it needs to be. Will dune be able to read some library metadata out of already built library somewhere inside an opam switch, to figure out if it contains Rust crate that it needs to potentially take care of? Should that metadata go to .opam files, and opam could read it and build a list of rust dependencies for provided .opam file? Does opam support some arbitrary metadata in opam files? If yes, probably some external tool can be prototyped for this, it should read all local .opam files, build depedency tree, scan it for that specific metadata, and generate a dune library like I outlined above for each .opam package. Does that sound sane? :thinking:

1 Like