Defns: By cross-compiling I mean having host executables and libraries that use a host ABI which is different from the ABI the C/ASM/OCaml compilers are targeting. gcc -m32, Hostx86\x64\cl.exe, nvcc and aarch64-linux-android-clang are all cross-compilers.
From my light grokking, it looks like:
reason-mobile creates a OCaml compiler that targets the target ABI by a) first creating a complete host ABI OCaml environment, and b) using the host OCaml plus the C+ASM cross-compiler to create a OCaml native code compiler for the target ABI. Doesn’t seem like many packages need to be patched for an OCaml project to be built, because Esy takes over all the OCaml build steps and can easily (I think) use a target ABI sysroot
ocaml-cross replaces many Opam packages including the base ocaml package. The changes involve using the C cross-compiler and perhaps specifying a target ABI sysroot in Opam build steps. The host OCaml compiler and the target OCaml compiler live together through the use of findlib.conf toolchains.
Mirage keeps the host ABI for its build tools, downloads all source code including transitive dependencies using opam monorepo, configures Dune so that it uses the C cross-compiler, and then Dune builds the entire monorepo.
Sorry if I screwed up / oversimplified describing your project.
Am I missing other cross-compiling implementations, or missing key ingredients on how cross-compiling is done today? I’m working through using CMake to build both C and OCaml code: CMake C + ASM compiler configuration that is available in a CMake project is fed into Opam and Dune. Since CMake makes it simple to cross-compile C code, it would be nice if I could pick an approach for cross-compiling OCaml code that avoids problems that others have already uncovered.
I’d like to add my own solution using Nix to the mix.
The implementation lives here. It mostly follows Eduardo’s approach (in fact, he helped me set it up).
The way it works: we build an OCaml cross-compiler and then use Nix’s builtin cross-compilation support (along with dune’s -x target) and multiple findlib toolchains to tie it all together.
My Nix overlays CI, in the same repo, cross-compiles some well-known (to me) OCaml libraries and applications to both musl64 and aarch64-musl.
@anmonteiro I know nothing about Nix but that was fairly easy to understand. Thanks!
What I see as a requirement for this approach (other than Nix) is that you have the ability to call Dune directly for each package. In some sense it would evolve into the Opam monorepo of Mirage if I needed many transitive packages (or a very large Nix file). Is that a fair assessment?
Dune is not required, just more common. Some of the straggler packages that are still on OCamlbuild (or use topkg) can also build just fine, e.g. with the TOPKG_CONF_TOOLCHAIN environment variable.
That is correct, but not because esy handles it, but because reason-mobile generates all the patched files, similar to how nix does with overlays.
This doesn’t work with an opam environment because the sandboxing is required for anything that communicates through the environment(think C packages), which is also why Nix works really well here.
Another important thing is that for reason-mobile CMake is also setup, as that was required for the revery project.
Also it’s communicated through findlib.conf
Biggest problem is that the Makefile script doesn’t work, if you want to compile all OCaml packages you don’t want only an ocamlc targeting the platform, as compiler-libs.optcomp will not be available, which is why my solution works 100% of the time even on old and bizarre packages. Also having a native cross compiler leads to great performance.
The second biggest problem is that OCaml just cannot emit 32bits on 64bits platforms which is required for armv7(important), so you need to be able to run OCaml 32bits somehow, on x86 that’s by using the i386 backend, on some ARM platforms you can do armv7 to armv7, but on Apple Silicon there is no way.
I would highly recommend trying the one on Nix by @anmonteiro if you want the most reproducible environment and if it doesn’t work well enough for you, or you want more flexibility, you can use opam monorepo if that works for you, or reason-mobile if you want to compile something fancy.
I have a WIP branch somewhere with Windows as a target for reason-mobile and it would make the toolchain complete. Also ESP32 missing, but this one is not hard.
Gotcha. So regardless of which choice I make I’ll try out your technique.
Yes, without modification the bwrap.sh sandbox will get in the way. That is also trivial to change with Opam wrap-*-commands config fields. But I’m going to stick closely to what experienced OCaml cross-compilers like you have done, so I’m not going down that path.
Yikes yikes yikes. This is exactly the sharp edge I would have stumbled on if I hadn’t asked my question! Thanks for that news.
My primary dev environments are Windows, Ubuntu on WSL2 and mac M1 (Apple Silicon). To get ARM64 with aarch32 CPU support sounds like I need to add my Nvidia Jetson TX1 dev kit as another primary dev environment.
All the free CI providers I am familiar with (GitHub Actions; GitLab CI) have shared CI runners that don’t support ARM32. I believe GitLab CI will support Apple Silicon in the near future but that doesn’t sound like an option. Mostly because I’ll be vending security conscious software I do intend to a) make my source available, b) let others easily build it and c) let others build on top of it. That is partly why I released my Windows build tools. Anyway, it sounds like my only solution for ARM32 and public CI is slow QEMU user-mode emulation.
If I believe what is on the Internet, Nix only works under WSL2 on Windows; that is a non-starter for my requirements. Correct me if I’m wrong. So I’ll try out opam monorepo next, but coupled with your host/target OCaml environment trick.
Can you elaborate on this? The ocaml-cross/opam-cross-windows has been updated to opam 2 and seems to work well with sandboxing. Are you referring to a different sandbox?
macOS: So far it doesn’t look like exporting all relevant directories or even the legacy OPAM_USER_PATH_RO directories is present in macOS yet. But there is that escape hatch I mentioned earlier; just set the Opam configuration field on macOS to use your own wrapper with your own mounts. (Changing wrappers seemed to work very well when I did it on macOS with Opam 2.1; you can even compose the global and switch-specific wrappers)
I’m not sure to understand this. Unless I’ve missed something, the macos sandbox doesn’t hide files from the file system and the sandbox script simply makes everything readonly by default. In the same PR you mentioned I’ve removed the code handling OPAM_USER_PATH_RO on the macOS sandbox script because it doesn’t do anything (everything is accessible readonly by default already)
Yes, opam-cross doesn’t communicate through the environment, also it doesn’t handle C libraries unlike esy, which is a big pain when cross compiling big projects with C deps. As an example Revery uses Skia, and how does it know which Skia to use when there is multiple in the environment? It doesn’t, it always uses SKIA_SOMETHING which is the same for both platforms, but because esy has sandboxing that’s not a problem, as skia-wrapper will see only it’s dependencies, and it doesn’t not depend on the arm64-skia. You could use pkg-config for that, but then every package needs to use it and a couple wrapped packages in esy don’t do it.
IIRC an opam dependency A can see B even if B it’s not on it’s dependency list.
On windows it handles C libraries via MXE dependencies. I have never looked at the other cross building setup but I’m happy to do knowledge transfer to see if a similar solution could be implemented.
I took a look at your release assets and your GitHub CI. That is a substantial build matrix. So yes, I’m curious if you are using autoscaling ARM hardware (ex. AWS Graviton EC2) or if you doing a pure cross-compile. I do see some use of “docker buildx” which I was unfamiliar with; the Overview of Docker Build | Docker Docs page looks like it uses QEMU binfmt emulation underneath which I am familiar with.
Yes, docker buildx will allow you to do any kind of cross-compilation from a x86 host, however, it is extremely slow. If you got time and wanna stick to the free, open source plan, that’s a solution. Otherwise, renting a fast hardware machine and using it as a self-hosted runner works great.
Could anybody point me to the right tutroial about crosscompilation of OCaml using Nix?
After a few tutorial I stumbled at the following:
➜ nix-ocaml-overlays git:(master) ✗ nix shell 4.14.1+flambda
warning: Using saved setting for 'extra-substituters = https://anmonteiro.nix-cache.workers.dev' from ~/.local/share/nix/trusted-settings.json.
warning: Using saved setting for 'extra-trusted-public-keys = ocaml.nix-cache.com-1:/xI2h2+56rwFfKyyFVbkJSeGqSIYMC/Je+7XXqGKDIY=' from ~/.local/share/nix/trusted-settings.json.
error: flake 'git+file:///mnt/old/nix-work/nix-ocaml-overlays' does not provide attribute 'packages.x86_64-linux.default' or 'defaultPackage.x86_64-linux'