MirageOS on Unikraft

Community
,
1

On behalf of all the developers involved (namely @fabbing, @Firobe, @n-osborne and me), it’s my pleasure to announce that the first release of the Unikraft backend support in MirageOS unikernels.

Unikraft is a unikernel development kit: it is a pretty large collection of components that can be picked up, or not, in the unikernel tradition of modularity. The scope of Unikraft is much larger than Solo5, as it aims to make it easy to turn any Unix server into an efficient unikernel.
This was in fact a first motivation to explore using Unikraft as MirageOS backend: to experiment and see what performance we could get, in particular using their virtio-based network interface, as virtio is implemented currently only for one specific x86_64-only backend in Solo5.

Some of the immediate performance differences we observed are detailed further, but that is not all we hope from this Unikraft backend in the long-term. In particular, Unikraft is on the road to be multicore-compatible (i.e. having one unikernel use multiple cores). While this is not ready today and there are still significant efforts to get there, it means that this MirageOS backend will be able to benefit from these efforts and eventually support the full feature set of OCaml 5.

Furthermore, the Unikraft community (which is quite active) is experimenting with a variety of other targets such as bare-metal for some platforms or new hypervisors (e.g. seL4). Any new target Unikraft supports can be then supported “for free” by MirageOS too. For example, this already brings firecracker as a new supported VMM for MirageOS.

Lastly, since Unikraft is POSIX-compatible (for a large subset of syscalls), this potentially enables MirageOS unikernel to embed OCaml libraries that have not been ported to use the Mirage interfaces in the future. This would be useful for large libraries which are hard to port (owl comes to mind).

Overview of the Unikraft support

To add new MirageOS backends requires to create or modify a series of components:

Using Unikraft with a QEMU or a Firecracker backend is as simple as choosing the unikraft-qemu target or the unikraft-firecracker one when configuring a unikernel.

The OCaml/Unikraft cross compiler

To build the OCaml cross compiler to Unikraft, we use the Unikraft core, the Unikraft lib-musl and musl itself. musl is the C library recommended by Unikraft to build programs using the POSIX interface. This made it easy to build the OCaml 5 runtime, in particular because it provides an implementation of the pthread API which is now used in many places in the runtime[1]. This could also make it easier to port some libraries that depend on Unix to work on Unikraft backends.

The OCaml cross compiler per se builds upon the work that has been upstreamed to ease the creation of cross compilers, using almost the same series of patches than for ocaml-solo5. So the only version of the compiler that is currently supported for OCaml/Unikraft is OCaml 5.3. Almost all the patches will be in the upcoming OCaml 5.4 and there should no longer be any patches required by OCaml 5.5.

Note that we didn’t go with the full standard Unikraft POSIX stack, which includes lwIP to provide network support. We had a prototype at some point relying on lwIP to validate our progress on other building blocks but it raised many incompatibility issues with the standard MirageOS network stack so we dropped support for lwIP in that first release; we developed instead the libraries required to plug the MirageOS stacks into the low-level interfaces provided by the Unikraft core.

The new MirageOS libraries for Unikraft support

The Unikraft support comes with packages using the standard names: mirage-block-unikraft and mirage-net-unikraft to support the block and network devices. Those libraries are implemented directly on top of the low-level Unikraft APIs, and so are using virtio on both QEMU and Firecracker VMMs.
To evaluate the quality of the implementations for those devices, we ran a couple of small benchmarks. You can find those benchmarks (the unikernels along with some scripts to set them up and run them) in the benchmarks directory in @Firobe’s fork of mirage-skeleton, benchmarks branch.

Network device

To measure the performance of the network stack, we have tweaked the simple network skeleton unikernel to compute some statistics and used a variable number of clients all sending 512MB of null bytes. We have run this benchmark both on a couple of x86_64 laptops and on a LX2160 aarch64 board, all running a GNU/Linux OS.

We have observed a lot of variability in the performance of the solo5-spt unikernel (sometimes better, sometimes worse than unikraft-qemu) depending on the actual computer used, so those measures should be read with a grain of salt.

On two different x86_64 laptops:

network-x86_64
network-x86_641117×848 80.9 KB

network-x86_64-other
network-x86_64-other640×480 11.2 KB

On the LX2160 aarch64 board:

network-lx2160
network-lx21601155×867 77 KB

Block device

To measure the performance of the block devices, we wrote a simple unikernel copying data from one disk to another. We can see that the performance of unikraft-qemu is lower than solo5-hvt for small buffer sizes; fortunately, the situation improves with larger buffer sizes. We ran this benchmark only on a x86_64 laptop as there’s currently an issue with two block devices on aarch64 on Unikraft.

block-x86_64
block-x86_641810×1280 227 KB

It is worth mentioning that I/Os can be parallelised, which also gives a significant performance boost. Indeed, mirage-block-unikraft can leverage the parallelised virtio backend of QEMU and Firecracker; it takes care of limiting I/Os to what the hardware supports in terms of both parallelism and sector size.

Current limitations

  1. In our tests only Linux appeared well supported to compile Unikraft at the moment so we’ve restricted our packages to that OS for now.
  2. Unikraft supports various backends itself; in this first release, we’ve only added support and tested its two major ones: QEMU and Firecracker.

How to use

To try the new Unikraft backend for MirageOS, you need to use an OCaml 5.3 switch, so create one first if needed. Then add our opam overlay to get access to our latest versions of the packages until they are published on the standard repository and install mirage and the OCaml/Unikraft cross compiler. The short version could be:

$ opam switch create unikraft-test 5.3.0
$ opam repo add mirage-unikraft-overlays https://github.com/Firobe/mirage-unikraft-overlays.git
$ opam install mirage ocaml-unikraft-backend-qemu ocaml-unikraft-x86_64

See below for some explanations about the numerous OCaml/Unikraft packages.

From then on, you can follow the standard procedure (see how to install MirageOS and how to build a hello-world unikernel) to build your unikernel with the Unikraft backend of your choice.

$ mirage configure -t unikraft-qemu
$ make

Details about the various packages for the OCaml/Unikraft cross compiler

The OCaml cross compiler to Unikraft is split up into 14 packages (see the PR to opam-repository for more details) so that users can:

  • choose which of the backends (QEMU or Firecracker) and which of the architectures (x86_64 and arm64) they want to install, where all combinations can be installed at the same time,
  • choose which architecture is generated when they use the unikraft ocamlfind toolchain by installing one of the two ocaml-unikraft-default-<arch> package,
  • install the ocaml-unikraft-option-debug to enable the (really verbose!) debugging messages.

The virtual packages can be installed to make sure one of the architecture-specific packages is indeed installed:

  • ocaml-unikraft can be installed to make sure that there is indeed a unikraft ocamlfind toolchain installed,
  • ocaml-unikraft-backend-qemu and ocaml-unikraft-backend-firecracker can be intalled to make sure that the unikraft ocamlfind toolchain supports the corresponding backend.

Those virtual packages will be used in particular by the mirage tool when the target is unikraft-qemu or unikraft-firecracker.

All those packages use one of two version numbers. The backend packages use the Unikraft version number they are using, while the OCaml compiler packages per se use version 1.0.0.

Conclusion

This is a first release, which we are experimenting with; we expect to run it in production in the coming months but it may need improvements nevertheless. Notably absent from this release is an early attempt to leverage Unikraft’s POSIX compatibility to implement Mirage interfaces instead of hooking directly to Unikraft’s internal components. This early version used Unikraft’s lwIP-based network stack instead of Mirage’s (fooling Mirage into thinking it was running on Unix), and it may be interesting to revisit this kind of deployment, in particular for easy inclusion of unix-only OCaml libraries in unikernels.

We are eager for reviews, comments and discussion on the implementation, design and approach of this new Mirage backend, and hope it will be useful to others.

  1. Adding support for Thread-Local Storage has been a large part of the work to get OCaml 5 working on Solo5: even if the creation of threads is not supported, TLS is still necessary to get the runtime to compile. ↩︎

18 Likes