A user of a library I maintain wants to make it work in Mirage OS, but I’m worried that since I don’t use Mirage, it will be hard for me to not accidentally break support for it.
What I’m wondering is, how do other people ensure their packages stay working in Mirage? I considered building the tests as unikernels but that seems non-trivial. Is there some mode where I can forbid non-Mirage deps in a build or something?
I think the responsibility is more on people who develop MirageOS and be able to accept any libraries like yours. Of course, the reality is a bit hard but we can safely ensure that if:
unix or lwt-unix
Your library is compatible with MirageOS. Some linters exist such as forbidden_libraries with dune. For the second, it still possible to use C stubs but you must do some plumbing.
With a quick look into your project, I can say that your library should be compatible - then, just a try with Solo5 (eg. mirage configure -t hvt) will give you a true response 
. If you want more details about C stubs, I can explain it in this topic of course.
If I try, this, I get an error:
$ mirage configure -t hvt
mirage: unknown option-t'. Usage: mirage configure [OPTION]... Trymirage configure --help’ or `mirage --help’ for more information.configuration file config.ml missing
It seems like there’s not really a simple way to test if a library will work without having a full executable?
I realize it’s not really “our responsibility” to make it work in Mirage, but we’d like to make it a supported platform.
mirage configure -t hvt
Is the initial command when we want to configure an unikernel. However, it requires a config.ml and your unikernel.ml as you can find here: plenty of unikernels. So, it’s normal that the command does not work (see configuration file config.ml missing).
I realize it’s not really “our responsibility” to make it work in Mirage, but we’d like to make it a supported platform.
Again, it should be the case for any usual OCaml projects. The hard part when you want to be compatible with MirageOS is mostly about C stubs or libraries which uses directly unix. So, of course, you probably need to lint your dependencies. But the MirageOS team wants to avoid as much as possible constraints to be compatible with MirageOS.
(this is mostly the same as @dinosaure comment, and adds some sample code)
I’m sure there are other ways (including extraction of the build commands used by mirage), but I’d use the following approach:
mirage in your projectmirage/config.ml:open Mirage
let main = foreign ~packages:[package <my-to-be-tested-package>] "Unikernel" job
let () = register "hello" [main]
mirage/unikernel.ml:let start = Lwt.return_unit
then in your CI execute:
$ opam pin add -y <my-to-be-tested-package> , #not needed if your CI already opam pins your package
$ opam install -y mirage
$ cd mirage && mirage configure -t spt && make depend && mirage build
(the configure argument -t spt could vary, -t unix is the cheapest (but doesn’t fail if you link against C libraries), -t xen and -t hvt are also good candidates)
While making an pgx instantiation for Mirage, I had one issue with the API, which I hope Mirage experts can comment on. The preliminary signature I came up with is:
module Make :
functor (Random : Mirage_random.S) ->
functor (Clock : Mirage_clock.MCLOCK) ->
functor (Stack : sig include Mirage_stack.V4 val singleton : t end) ->
Pgx.S with type 'a monad = 'a Lwt.t
As you can see, I’m passing a singleton of the network stack in the functor. This is needed to implement
val open_connection : sockaddr -> (in_channel * out_channel) t
which is passed to Pgx.Make. Apart from being inelegant and lacking IPv6 support, this is impractical, since it means Pgx cannot be instantiated at the module top level. So, while this works, it looks like Pgx would need to generalize the way of establishing a PostgreSQL connection in order to provide a MirageOS-style API.
I presume Mirage developers have sees similar issues in the past, as it relates to the difference between the design of MirageOS and the regular API. So my question is, do you have a good solution, or suggestion for how the pgx-mirage adapter should look like?
BTW, integrating this into a sub-package of pgx, could be a way to test compatibility.
I’m not sure about your problem but mirage works with functoria which is able to describe how to connect a device and give the result to the start function. For example, the syslog device provided by the mirage distribution needs several implementations and there representations into mirage:
let syslog = impl @@ object
method ty = console @-> stackv4 @-> syslog
...
end
The object which is the description of a device can describe how to connect (or create) the syslog representation:
method! connect _ modname = function
| [ console; stack; ] ->
Fmt.strf "%s.connect %s %s" modname console stack
| _ -> assert false
Where modname is the name of the syslog module and console, and stack are already connected values. Then, mirage will generates a main.ml which will call all connect of all used implementations such as our stackv4 and pass them to others devices where it’s needed.
I use some terms which can misleading the comprehension of mirage. When we talk about connect or device, it’s an abstraction where syslog in our example is a device. A device should have a witness, a type t and you must have something to create/connect this type t. Usually, MirageOS libraries provides a connect function which is able to create the witness from some arguments.
In your case, we can imagine than Pgx can be described with:
type pgx = Pgx
let pgx = Type Pgx
let pgx = impl @@ object
method ty = random @-> clock @-> stackv4 @-> pgx
...
And the connect function should be:
method connect _ modname =
| [ random; clock; stack; ]
Fmt.strf "Pgx.open_connection %s %s %s" random clock stack"
Then, your config.ml should describe how to create/connect your value such as:
let pgx = pgx default_random default_clock default_stack
let () =
register "my_unikernel"
[ my_unikernel $ pgx ]
In your unikernel.ml you should be able to put a new functor such as:
module Make (Pgx : Pgx.S) = struct
let start my_pgx_connection = ...
end
mirage will generate the main.ml which will call your connect function applied with right arguments. Then, it will call Unikernal.start with what it created.
Of course, all of that is an example but a look into the mirage distribution should help you about how to use functoria. I think your problem is mostly about the lack of documentation on this specific part of mirage. However, when you understand then how you can describe your device, the life is much more easier for any !
Thanks! I didn’t consider using functoria, having only used it from the application point of view, but this is clearly the right solution.
Thanks @hannes, that worked!
https://circleci.com/gh/arenadotio/pgx/255
At some point I might make the entrypoint run some of the tests but for now this gives us the main check I wanted (Pgx compiles against Mirage).
That’s fantastic! When you’re happy with it, we’d be delighted to have a simple skeleton application using pgx over at https://github.com/mirage/mirage-skeleton. We use that as a place for testing out new features (specifically right now, the port to Dune that @samoht is leading), as well as a tutorial area for newcomers.
I’ll work on adding a pgx_mirage library this week and I can make a pull request to mirage skeleton after that. Would it be okay to use a pinned version of pgx_mirage for the example or should I wait till we can make a release to opam?
Best,
Anurag
A released version is easier, but you’re very welcome to open a draft PR to mirage-skeleton with pin-depends to test out your release, if that helps.
Thanks. I will open a draft PR as i’ve mostly just used mirage from an application stand point, and I might miss some best practices when it comes to a library that’s meant to be consumed by a mirage application. Feedback on the draft PR could be useful to iterate on the implementation before a release.
Thanks to @paurkedal I made some progress in splitting pgx_lwt, into a non unix dependent pgx_lwt, pgx_lwt_unix, and a pgx_mirage. https://github.com/arenadotio/pgx/pull/78
So far I’ve only tried a few small mirage application examples so i’m not sure if what I did is a good approach for a library intended for mirage, but i ended up with an interface like:
This interface worked fine in a small skeleton unikernel app that i tried to setup (its part of the pull request i linked to), so i’m able to test and verify that i can build a unikernal that uses pgx_lwt_mirage. That being said, I don’t know a lot about how to setup the network properly for testing on my laptop (without ethernet), to make a successful client connection while using the hvt or spt backends. So i’ve been limited to just verifying that I can access a postgres instance and perform operations on it while configuring the unikernel to use the unix target.
Once I figure out enough about the networking setup, i’ll feel a little more confident about contributing an example to mirage-skeleton.