Bin_prot in mirage unikernels?

Learning
1

Is janestreet bin_prot library [https://github.com/janestreet/bin_prot] compatible with mirage unikernels? i.e. it doesn’t seem to have a dependency on Unix module so I gather it is okay to use in mirage unikernels. Correct?

2

I don’t know any reason why it wouldn’t be compatible with mirage. There’s a xen directory that appears to have mirage-specific C stubs.

3

Thanks for confirming.

4

It seems true for Xen but it should fail for Solo5/KVM (freestanding) In such case, you probably need to follow the same layout than bigstringaf to correctly provides C stubs for any targets.

1 Like
5

Ah. Okay. In that case does the standard Marshal module work correctly then?

6

Marshal should be ok, I never use it but I would like to say that the idea of a file-system (and by extension a file-descriptor) does not really exist into a Mirage (only an Mirage with UNIX should be able to play with in_channel and out_channel as we expect).

For example, for some operations such as open_in, due to the use of nolibc for hvt/spt targets (Solo5), they always raise an error. It’s the reason of some projects such as:

  • ocaml-crunch to copy some files into the unikernel (read-only)
  • irmin to have an access to a external database (read-and-write)

More generaly, if your case is to serialize some data and be able to retrieve them outside your unikernel, the best is to use irmin.

7

With regards to bigstringaf, is this required/recommended when building mirage unikernel? I don’t have experience with this lib and after looking at its api, it seems to offer an alternative api to stdlib Buffer data structure along with getting/setting bytes of BigEnding/LittleEnding integers. Stdlib with ocaml version 4.08 seems to provide these API too in its Bytes module. Is bigstringaf more performant than these stdlib APIs?

1 Like
8

So the problem is not precisely bigstringaf but “the required layout for a library to be compatible with MirageOS”. Generally, an usual library in OCaml can be used with MirageOS only if it does not depends on unix.cmxa.

dune comes with a mechanism, forbidden_libraries to ensure that you don’t depend on unix.

C stubs and MirageOS

However, for a library which has some C stubs, the requirement is a bit more complex. In fact, MirageOS can compile your librairies with something else than the usual standard C library. If you want to make a the spt/hvt unikernel (Solo5/ocaml-freestanding), your C stubs must be compiled with some extra flags and some specific headers.

The usual layout in such case (which appears for bigstringaf, digestif or mirage-crypto) is to provide into the distribution cross-compiled C artifacts for MirageOS. The example of bigstringaf is better than others because the library is small.

Finally, the library provides:

  • libbigstringaf_stubs_freestanding.a to be linked with a Solo5 unikernel
  • libbigstringaf_stubs_xen.a to be linked with a Xen unikernel

Both are compiled with specific rules:

  • lib/freestanding/dune
    It provides bigstringaf.freestanding which is a compilation of the copied bigstringaf_stubs.c with specific flags. Currently these flags can be get by pkg-config
  • lib/xen/dune
    It’s the same than freestanding but with Xen’s flags

Then, the library must provides the place where are located these specific C libraries. For that, we use the META file and we add these informations:

xen_linkopts = "-l:xen/libbigstringaf_xen_stubs.a"
freestanding_linkopts = "-l:freestanding/libbigstringaf_freestanding_stubs.a"

With all of that, you are able to use these C stubs with MirageOS! It’s a bit complex than usual but we must respect that to be able to correctly link and use your C stubs in any proposed targets by MirageOS.

The same layout exists for digestif, mirage-crypto, cstruct, bigarray-overlap or any libraries largely used by MirageOS which contains C stubs. To come back to your initial case, bin_prot seems to provide the Xen target (eg. xen directory) and it follows the same pattern:

  • a sub-library bin_prot.xen
  • a call to pkg-config to get extra flags
  • a modification of the META file

So, as far as I can tell, bin_prot is compatible with MirageOS only for Xen targets. However, it should be easy to extend it to Solo5 unikernel (and, again, follow the same layout).

Bigstringaf

bigstringaf was ported to MirageOS mostly to facilitate the port of httpaf to MirageOS (as one of its dependencies). It is about an issue from the OCaml community to have a decent API to use a bigstring (or more specially, a (char, unsigned_int8_elt, c_layout) Array1.t.

Some others projects like it exist:

  • bigstring
  • bigstringaf
  • ocplib-endian
  • base_bigstring

And an issue exist to integrate such API into the standard library: https://github.com/ocaml/ocaml/issues/7772

Then, about the question if we should use Buffer or Bigstringaf, I think it depends on your use-case. As explained Hannes Mehnert interview about MirageOS and OCaml by Evrone, Bigarray has some advantages and some inconveniences.

1 Like