# \[ANN\] Esperanto, when OCaml meets Cosmopolitan

**URL:** <https://discuss.ocaml.org/t/ann-esperanto-when-ocaml-meets-cosmopolitan/10501>\
**Category:** Community\
**Tags:** cross-compilation, portability, mirageos, announce\
**Created:** [September 20, 2022, 9:49pm UTC](https://discuss.ocaml.org/t/ann-esperanto-when-ocaml-meets-cosmopolitan/10501 "2022-09-20T21:49:53Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![dinosaure](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/dinosaure/32/16_2.png) [@dinosaure](https://discuss.ocaml.org/u/dinosaure)\
**Post date:** [September 20, 2022, 9:49pm UTC](https://discuss.ocaml.org/t/ann-esperanto-when-ocaml-meets-cosmopolitan/10501/1 "2022-09-20T21:49:54Z")

</div>

I am delighted to present the first **experimental** release of Esperanto. This project is a new OCaml _toolchain_ that creates binaries compiled with the [Cosmopolitan C library](https://justine.lol/cosmopolitan/) and linked with the [αcτµαlly pδrταblε εxεcµταblε](https://justine.lol/ape.html) link script. The binary produced is then portable to different platforms:

![Linux](https://worker.jart.workers.dev/redbean/linux.png) ![Windows 10](https://worker.jart.workers.dev/redbean/windows10.png) ![](https://worker.jart.workers.dev/redbean/msdos60.png) ![MacOS](https://worker.jart.workers.dev/redbean/macos.png) ![FreeBSD](https://worker.jart.workers.dev/redbean/freebsd64.png) ![OpenBSD](https://worker.jart.workers.dev/redbean/openbsd.png) ![NetBSD](https://worker.jart.workers.dev/redbean/netbsd2.png)

The main objective of Esperanto is to provide a toolchain capable of producing a portable binary from an existing project. This would allow to finally be able to distribute software for all these platforms without having to:

1. manage multiple platforms orthogonally, the Cosmopolitan C library offers you the POSIX API for all platforms (including Windows)
2. Produce several versions of the same software for each platform. Only the binary is needed to run on all platforms

Cosmopolitan **does not** however produce a binary with a multi-platform assembler. At this stage, our distribution only supports the `x86_64` assembler (the most common one) but we are working on the possibility to produce a binary with different assemblers.

I would like to give special thanks to Justine, the author of the Cosmopolitan project (to develop [redbean](https://redbean.dev/), a small portable HTTP server) for her excellent work.

## A _toolchain_

In OCaml, the “toolchain” principle allows the existence of several compilers within an OPAM switch and to choose one of them when it comes to cross-compiling a project. This principle, even though it is not clearly defined and even though its use remains very limited, exists through the `ocamlfind` tool.

You can find these toolchains in your switch:

```sh
$ ls $(opam var lib)/findlib.conf.d/
esperanto.conf solo5.conf

```

From our experience with Mirage as well as the work done in `dune` regarding cross-compilation, the choice to propose a new _toolchain_ in order to allow cross-compilation of projects with OPAM is both a historical choice but also the most relevant one in our opinion[1](#fn1).

### Why we need to cross-compile?

The term cross-compilation can be misunderstood if we only consider the question of the assembler issued by the compiler (does it match the host assembler or not). In our case, cross-compilation is a broader term that implies the use of external artefacts to the compiler that are different from the default and the use of compiler options that must be used throughout the production of the final binary.

In other words, even though we are emitting the same assembler, we are doing so in a different “context” which requires the definition of a new _toolchain_ which includes our artefacts and compiler options.

One of these artefacts is of course the C library used by the compiler which will then be systematically used by the _runtime caml_, the well named `libasmrun.a`. This is why, for example, there is a version of OCaml with [musl](https://musl.libc.org/). So there must be a version of OCaml with Cosmopolitan.

This new _toolchain_ also allows you to include the necessary options for compiling C files because, yes, you can compile a C file with `ocamlopt`.

In order to provide a coherent _workflow_ for a project, we need to provide not only a `libasmrun.a` compiled with our Cosmopolitan C library but also an OCaml compiler capable of invoking the C compiler with the right options required by Cosmopolitan.

Finally, we also need to describe in this _toolchain_ how to link the object files together to actually produce a portable binary using the APE script.

### A simple example with this new _toolchain_

Installing Esperanto is very easy with OPAM. It will install the cross-compiler and the necessary files so that `ocamlfind`/`dune` can recognise this new _toolchain_:

```sh
$ opam install esperanto

```

Finally, let’s try to produce a simple binary that displays “Hello World!”:

```sh
$ cat >main.ml <<EOF
let () = print_endline "Hello World!"
EOF
$ ocamlfind -toolchain esperanto opt main.ml
$ objcopy -S -O binary a.out
$ file a.out
a.out: DOS/MBR boot sector

```

The binary produced can already be executed. However, there are still some issues that have been fixed since then but which are probably not yet integrated in your system. They concern `zsh` and `binfmt_misc` in particular.

The first problem with `zsh` is that it does not recognise the binary correctly. This problem has been fixed in the latest version of `zsh.5.9.0`.

```sh
$ zsh --version
zsh 5.8.1
$ zsh
$ ./a.out
zsh: exec format error: ./a.out

```

The second problem concerns `binfmt_misc` which intervenes upstream at the execution of your programs in order to choose how to execute them. In this case, `binfmt_misc` recognises Cosmopolitan binaries as Windows software by default.

Here too, a solution is available and described by the author of Cosmopolitan here: [APE loader](https://justine.lol/apeloader/#binfmt_misc)

#### Execution & Assimilation

If you are not concerned by the above problems, you can simply run the program:

```sh
$ ./a.out
Hello World!

```

There is a final solution that requires a little explanation of what αcτµαlly pδrταblε εxεcµταblε is. Indeed, the latter makes it possible to create a polyglot binary whose first point of entry is not your program but a small program which tries to recognize on which platform the binary tries to run.

After this recognition, this little program will “inject” values corresponding to the platform in which you try to run your program in order to finally let Cosmopolitan manage the translation between its interface and the real POSIX interface that your system offers.

Of course, this step has a cost as it adds an indirection between what your program wants to do and what is available on the system running your program. However, APE offers a very special option that allows the program to be assimilated to the platform in which it wants to run.

```sh
$ file a.out
a.out: DOS/MBR boot sector
$ sh -c "./a.out --assimilate"
$ file a.out
a.out: ELF 64-bit LSB executable, x86-64
$ ./a.out
Hello World!

```

This option makes your application truly native to the platform in which you run it. This means above all that the program is **no longer** portable.

#### Esperanto, `dune` & `opam monorepo`

The `dune` software also incorporates this toolchain idea using the `-x` option. More pragmatically, it is possible to define a new dune context to use Esperanto as a compilation toolchain.

However, the original aim of Esperanto is to produce a portable binary. This implies, among other things, that it should not depend on remaining artefacts in order to run and, in this sense, the compilation of your project should be a static compilation. This means that all dependencies of your project must be available to compile in the same context as your project.

Again, this is particularly necessary if any of your dependencies include C files, so they need to be compiled in some way.

This is where `opam monorepo` comes in, it will simply “vendor” your dependencies into a “duniverse” folder. Here are the steps needed to compile a project with Esperanto. We’ll take [`decompress`](https://github.com/mirage/decompress) as an example which produces a binary that can compress/decompress documents:

```sh
$ git clone https://github.com/mirage/decompress
$ cd decompress
$ cat >>bin/dune <<EOF
(rule
 (target decompress.com)
 (enabled_if
  (= %{context_name} esperanto))
 (mode promote)
 (deps decompress.exe)
 (action (run objcopy -S -O binary %{deps} %{target})))
EOF
$ cat >dune-workspace <<EOF
(lang dune 2.0)
(context (default))
(context
 (default
  (name esperanto)
  (toolchain esperanto)
  (merlin)
  (host default)))
$ opam monorepo lock --build-only
$ opam monorepo pull
$ dune build bin/decompress.com
$ sh -c "echo 'Hello World' | ./bin/decompress.com -d | ./bin/decompress.com"
Hello World

```

## Issues

Apart from the outcomes described above, however, the Esperanto toolchain is not complete. Indeed, the OCaml distribution gives several libraries such as `unix.cmxa` and `threads.cmxa`. A little work has been done to make the former available. The second one is however unavailable for the moment since Cosmopolitan only partially implements `pthread`.

However, it seems that the author of Cosmopolotian wants to implement the rest of the `pthread` API which will then allow us to provide support for `threads.cmxa` and OCaml 5.

This of course makes support for the projects more limited than we imagined (and that’s why this release is experimental) however, an effort has already been made to lwt into Cosmopolitan’s hypothetical future support for `pthread`.

## Future

As explained above, support for `threads.cmxa` and OCaml 5 remains the priority. however, an effort has already been made to support [Lwt](https://github.com/ocsigen/lwt/) via Cosmopolitan’s hypothetical future support for `pthread`.

However, it is possible that Cosmopolitan could become a target for the MirageOS project in the same way as Solo5 (or our recent experiment on Raspberry Pi 4).

In this sense, we will surely propose an integration in MirageOS so that projects can both produce unikernels with Solo5 or portable binaries with Cosmopolitan.

* * *

1: However, the question remains open at several levels, that of the compiler, that of OPAM and of course that of `dune`. It is clear that the current situation is not the best in terms of what we need to do to produce such a cross-compiler. Only the feedback from Solo5 (which requires cross-compilation) allows us to say that it is surely the right choice for what we want to offer.

## Conclusion

We hope that this project will facilitate the distribution of software. You can read a more technical article about our work [here](https://blog.osau.re/articles/esperanto.html). Finally, I would like to thank [robur.io](https://robur.io/) (an association you [can help](https://robur.io/Donate)) for allowing me to do this project.

**EDIT** : The author of Cosmopolitan just released Cosmopolitan with `pthread` support. So we will definitely try to improve our distribution to include OCaml with `threads.cmxa` support and move forward with OCaml 5!

---

<div class="post-metadata">

**Author:** ![dra27](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/dra27/32/5294_2.png) [@dra27](https://discuss.ocaml.org/u/dra27)\
**Post date:** [September 21, 2022, 8:02am UTC](https://discuss.ocaml.org/t/ann-esperanto-when-ocaml-meets-cosmopolitan/10501/2 "2022-09-21T08:02:13Z")

</div>

This is a glorious piece of work - we can even easily write UEFI programs in OCaml 🙂 It’s also unusually refreshing in OCaml-world to have an application which runs without caveats on Windows, but needs a few side-notes for running on Linux 😏😉

---

<div class="post-metadata">

**Author:** ![dinosaure](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/dinosaure/32/16_2.png) [@dinosaure](https://discuss.ocaml.org/u/dinosaure)\
**Post date:** [January 18, 2023, 3:08pm UTC](https://discuss.ocaml.org/t/ann-esperanto-when-ocaml-meets-cosmopolitan/10501/3 "2023-01-18T15:08:22Z")

</div>

After a long time, Esperanto [was upgraded](https://github.com/dinosaure/esperanto/releases/tag/v0.0.2) with the last version of Cosmopolitan which includes `threads.cmxa` now. A PR on `lwt` (to support Esperanto) [is pending](https://github.com/ocsigen/lwt/pull/945) but we found [an issue](https://github.com/ocaml/dune/issues/6843) with `dune` when we tried to compile a small program. This paves the way for OCaml 5 but, in the meantime, most OCaml applications can be compiled with this new toolchain (if the C files are not too fancy). A sample application is currently available here: [Bob](https://bob.osau.re). You can also see the reproducibility of the software [here](https://builds.osau.re/job/bob/build/ac20dfce-58b5-4b81-9e53-4fd11f484064) ([here](https://blog.osau.re/articles/reproducible.html) is an article for more information).

---

<div class="post-metadata">

**Author:** ![dinosaure](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/dinosaure/32/16_2.png) [@dinosaure](https://discuss.ocaml.org/u/dinosaure)\
**Post date:** [December 21, 2023, 3:27pm UTC](https://discuss.ocaml.org/t/ann-esperanto-when-ocaml-meets-cosmopolitan/10501/4 "2023-12-21T15:27:10Z")

</div>

I’m glad to announce the release of Esperanto (0.0.5). This release integrates the last release of [Cosmopolitan 3.1](https://twitter.com/justinetunney/status/1726141024597324189?t=K1nIh-2Y7JjaoOc9Y6bUzA).

The latter incorporates a great deal of work on the Cosmopolitan toolchain:

1. a modification of the GCC compilation so that code no longer needs to be patched to transform constants into variables
2. cross-compilation support for aarch64
3. support for Apple Silicon

This release introduces a change in the construction of the portable artifact from OCaml source code, which is explained [here](https://github.com/dinosaure/esperanto). More generally, the steps are:

- compilation to `x86_64`
- compilation to `aarch64`
- linking the two artifacts with `apelink`.

Currently, this version of Esperanto has been tested with [bob](https://github.com/dinosaure/bob), an OCaml file-sharing program. We have tested its portability on Debian, Ubuntu, Mac M1, FreeBSD and Windows. As we make clear in the description, Esperanto’s sole purpose is to assist in the distribution of software (rather than to provide a supportive environment for developers). We therefore advise you to use Docker (or related technology) to create your portable executable from your sources ([bob](https://github.com/dinosaure/bob/blob/main/.github/workflows/esperanto.yml) has, for example, a GitHub CI that produces such an executable).

Of course, we’re planning to integrate with OCaml 5, since `pthread` support is now available in Cosmopolitan (it wasn’t at first). So we only support OCaml 4.14 and OCaml 4.13. We’re waiting for a review of another project which is preparing its transition to OCaml 5 ([`ocaml-solo5`](https://github.com/mirage/ocaml-solo5/pull/124)) before integrating it into Esperanto.

Finally, we would like to thank the [robur.coop](https://robur.coop) cooperative (which you can [help](https://robur.coop/Donate)) for giving us the freedom to maintain this project.

PS: Esperanto is still in a release process (and not immediately available via OPAM), but the process should be completed in the next few days.