Eio 0.1 - effects-based direct-style IO for OCaml 5

I think that isn’t actually the best encoding for the non-capability version because it doesn’t handle multiple uses of Eio_main.run correctly. I think the best encoding is to use algebraic effects (using the old syntax for brevity):

effect Load : path -> string
effect Save : path * string -> unit

let main_run fn =
 Env_main.run
   (fun env ->
     match fn () with
     | () -> ()
     | effect Load path, k ->
         let data = Eio.Dir.load (Eio.Stdenv.fs env) path in
         continue k data
     | effect Save(path, data), k ->
         Eio.Dir.save (Eio.Stdenv.fs env) path data ~create:(`Exclusive 0o666);
         continue k ()

let load path = perform (Load path)
let save path data = perform (Save(path, data))

Given such an interface, you can also implement the capability-based version:

type env = { perf : 'a. 'a eff -> 'a }
type fs = env

let fs env = env

let load { perf } path = perf (Load path)

let save { perf } path data = perf (Save(path, data))

let main_run' fn =
  let effect Perf : 'a eff -> 'a in
  let effect Tunnel : 'a eff -> 'a in
  let env = { perf = fun eff -> perform (Perf eff) } in
  match
    main_run (fun () ->
      match fn env with
      | () -> ()
      | effect Perf eff, k -> reperform k eff
      | effect (Load _| Save _) as eff, k -> reperform k (Tunnel eff))
  with
  | () -> ()
  | Tunnel eff, k -> reperform k eff

The encoding is a bit awkward with the multicore algebraic effect handlers because they lack some more advanced facilities (e.g. shifting) for manipulating handlers.

I think that the “algebraic effect” version is the more natural design if you have an effect system to type things properly, since the types already give you more precise information about which facilities are being used than the capabilities do.

That’s not to say that the capability approach is a bad design though. On the contrary, I think that it might combine very nicely with some of the work we’ve been doing at Jane Street on local allocations, allowing us to have a safe effect handlers without/before having a full blown effect system in the language. I’m very interested in exploring this direction.

I just think that your reasoning that you can build the non-capability version on top of the capability version but not vice versa is incorrect: each version can in fact be implemented in terms of the other.

Eio uses objects internally, while mostly exposing a functional API

It still requires users to understand class syntax to read the API and to decode the more difficult type errors from object types. Taking a method call on every operation also seems like an unnecessary cost. Since all you are using them for is basic higher-order functions accepting products of functions, it would seem better to provide a version using records or first-class modules or just abstract product types, and then add a layer using classes in another module or library if someone actually wants the additional structural typing for some reason.

e.g.

val run_webserver :
  www_root:Eio.Dir.t ->
  certificates:Eio.Dir.t ->
  Eio.Net.listening_socket ->
  unit

gives much more information than:

val run_webserver :
  unit -[filesystem,network]-> unit

FWIW, whilst earlier prototypes of effect typing for OCaml might look something like that, at this point I would expect any effect system that makes it into the language to have named effects like:

val run_webserver :
  unit -[www_root: Eio.Dir.t;
         certificates: Eio.Dir.t;
         network: Eio.Net.listening_socket] -> unit

Although I’m not sure it would be quite as convenient as capabilities when used at this fine a granularity.

8 Likes

Depends what you want to do with it. If you just want to wait for data and then handle it yourself, Eio_unix.await_readable should do.

There’s no public API at the moment to e.g. wrap a Unix.file_descr as an Eio.Flow (though that would be useful). You can always implement any interface yourself, though.

But if you want e.g. an optimised read from a tap device using io_uring then it needs to be in the abstract API so all the backends know to provide an implementation of it. PRs to fill out the current APIs are very welcome!

(and, yes, opening the device from Eio and then getting the Unix FD with Eio_unix.FD is the best option if you can)

The API itself is split in two main parts: the “Concurrency primitives” and the “Cross-platform OS API”. However, the backends tend to combine them. For example the Eio_luv backend runs a libuv event loop, which works with libuv abstractions such as Luv.File. You can’t use a Luv.File with an io_uring event loop, for example, so the event loop also implements the OS abstractions.

Each backend is designed for some platform (Linux, macos, Windows, libuv, etc), and should provide an API for whatever that platform provides. Then there is a standard set of types that common operating systems provide, and Eio_main can select one of the backends that implements this common set.

However, there will also be other backends that don’t implement all of the suggested interfaces. For example, an Eio_browser.run wouldn’t provide a file-system (and browser code wouldn’t use Eio_main).

Some backends won’t implement any of the interfaces. For example, an Eio_xen.run would only provide low-level Xen APIs (allowing sharing of pages of memory with other VMs and sending inter-Xen-domain interrupts). Then OCaml libraries would be used to build OS abstractions on top (such as Ethernet devices, IP networking, block devices, file-systems). Ideally, these higher-level abstractions would implement the standard Eio interfaces and would therefore be compatible with other Eio code.

The exact APIs are very much up for debate, and will combine features from Mirage, the stdlib, Lwt and other platforms.

Taking the sole example of fibers, I would certainly like to see a core set of effects/primitives to deal with them in the Stdlib but in a way that allows me to interpret them in possible different ways than is exposed here.

The ones Eio uses itself are here: Effects (eio.Eio.Private.Effects). Something like this could go in the stdlib one day. The main opinionated item in this API is support for cancellation.

3 Likes

These places are all very old code that is one of the greatest maintenance and contributor headaches of Lwt. I am speaking, of course, as a long-time Lwt maintainer. I strongly suggest, if reasonably possible, to avoid classes and objects anywhere in a core library.

10 Likes

I strongly agree with this. A “core” I/O library with object capabilities is the wrong way to go for OCaml. It will create a learning barrier, and constant “noise” when programming.

What is the reasoning, safety, or composability benefit, in the wide context of all program analysis tools and methods in use today? What is the actual benefit of this kind of capability passing, when capabilties are not considered in isolation?

In the EIO README, I saw it mentioned that there may be some future ocap extension that makes capabilities worthwhile. Are there any concrete, practical plans for actually bringing this to OCaml?

On security or reasoning, given how varied real OCaml code is, including linked-in libraries, what realistic threat do these capabilties actually protect against? Are OCaml function calls and modules really the right “boundary” at which analysis should be done? Is there a discussion of this available?

I don’t agree that there is an analysis advantage to passing object capabilties. In the OCaml context, as far as I can tell, passing these capabilties around, from an analysis point of view, amounts to forcing users to manually execute pieces of the work of a static analyzer, but given the realities of OCaml, discovering genuine information about a program will still take running a static analyzer, or performing it manually to some extent. This fine-grained threading-through of capabilties will also be a constant burden on refactoring. Is there a discussion of this somewhere?

On composability, I don’t see what pervasive required use of capabilities offers over ordinary composable streams, extra arguments for testing, and the like, used where necessary instead of forced everywhere by a foundational library.

Is there an example of a widespread object capability-passing language or library ecosystem, with many developers and some years of real-world experience?

If the answer to these questions is no (as I think), object capabilities are only a burden, and moreover will bring a bad reputation to OCaml as impractical if they are found in a library presented as foundational. This in turn strongly invites a fork of EIO or a separate project to create an alternative.

Also, speaking from the point of view of designing a very clear API for Dream, I would be greatly disappointed to have the underlying I/O library, that people have to use elsewhere, be something that is unnecessarily complicated and fundamentally going the opposite way.

13 Likes

I like @lpw25 's argument that foundational libraries should be unopinionated and conservative in their design. It’s something seen again and again in the ecosystem, including in bindings: for example, we have sqlite3 which is a robust thin layer on top of C, whose design can’t be contested; and then various people build their own favorite abstractions on top of it, or just use the C-like API directly if they prefer (or if they want to control everything for performance reasons).

Similarly, an IO/concurrency/effects library should be as small and thin as possible (like luv). We don’t know that this particular flavor of capabilities or concurrency primitives is the one that will work for everyone. Heck, people already have lots of different ways of addressing both (lwt’s style, async’s style, which has a different semantics in bind, dune’s fibers with structured concurrency, ocamlnet, grumpy people like me who like threads, etc.). Why should we commit to one experimental and very opinionated design?

It seems like passing everything explicitly by argument is great for Mirage, where one deals with many backends (raw hardware ones, sometimes) for random values, clocks, storage, etc. That’s laudable and impressive. But for the average desktop/server OCaml programmer, who writes websites or APIs (or theorem provers or compilers or…), that makes IOs extremely awkward and ceremonious. If I want to capture the current time, I don’t want anything more complicated than Mtime_clock.now() or Ptime_clock.now() (even Eio.Clock.now ~clock:Global.clock () seems quite annoying if you add the initialization bit). I certainly don’t want the rest of the code to know I’m calling that (it might just be for tracing or debugging or logging edit: @talex5 addressed that, sorry). We’re writing OCaml, not Haskell, and side effects and impure functions are normal.

Another question I’ve had for quite a while, is: why do we need a new library, instead of working effects into Lwt in a retrocompatible way? Something to get val await: 'a Lwt.t -> 'a would be incredibly helpful, and then more and more APIs could provide effect-based versions that eschew promises. At the same time, that would allow the whole ecosystem of Lwt users to gradually migrate to effects without having to go through the pains of show-stopping migrations (remember python3?). Personally I really try to keep my code compatible with relatively old versions of OCaml (sometimes as old as 4.03); if lwt was the way forward for effects it’d help a lot of people support both pre- and post- 5.0 OCaml.

8 Likes

Thanks for your answer. I was able to do pretty much what I wanted.

I was surprised that I had to first await_readable and then call Unix.read. I guess that’s quite expected for a 0.1 version though.

By the way, does creating fibres from inside fibres changes anything to their scheduling?

A genuine thank you for all the comments everyone. Firstly, we are not proposing this for the OCaml stdlib: this thread is about the first ever tag in the git repository, and so we welcome all the discussion. I wanted to describe some of the goals of the eio project to clarify our aims more clearly.

We began eio in late 2020 to a backdrop of a flurry of design work on multicore OCaml itself, and needed a way to evaluate whether effects were practical for their promised use at all. The main goal (in the first para of the README) was specifically to build a direct-style alternative to Lwt, with structured concurrency for resource-safe cancellation and backtraces without exposing effects in the library interface.

The bulk of our work has been to test the hypothesis that direct-style IO could be significantly more performant and observably optimal with how they invoke modern OS async interfaces like io_uring, GCD or IOCP. The performance results here speak for themselves, and I hope others are also impressed with just how good it is. Some of the speedups are due to the use of modern IO interfaces, but Eio makes it ergonomic to use those async interfaces without much programmer knowledge. These IO backends are not monolithic: as they stabilise, new libraries (e.g. uring) have all been released separately. Their integration into eio is also modular as separate opam libraries, with the eio core being very portable.

A secondary aim (what with @talex5 and I both being long-time core MirageOS developers) is to investigate how to replace the need for functors in Mirage libraries, and indeed to form the basis for MirageOS 5. We’ve published hundreds of libraries to opam that use these device signatures, and the addition of a single argument to eio calls is sufficient to eliminate those functors while improving performance and retaining portability. We would like MirageOS code to be “just like normal code”, and this has been true since 2013 when the effects research started.

While we’re not set in stone on the use of objects and classes yet; it was not a design decision taken lightly. We want eio to be performant, portable and safe, and the interfaces it exposes so far achieve that balance pretty well. If there are other ways to achieve the subtyping that we need (for unidirection and two-way flows, for example) without lots of syntactic overhead, we can give that a try. But what is currently exposed to developers is very succinct at the moment.

I find the argument that foundational libraries should be conservative and unopinionated to be somewhat uncompelling at this stage in OCaml’s life. We’ve just designed, implemented, evaluated and have almost released two mega features that are transformative to the language: effects and multicore. If we don’t push the boundaries now and explore these features to reap the benefits, what was the point of all the effort? OCaml 5.0.0 will be evolving throughout 2022, and this is the perfect year for the community to go wild and try out other experiments along the same lines as eio and domainslib.

Not all the experiments will succeed, but that is the nature of scientific endeavour. What’s left behind to form a standard IO library for OCaml 5.0 will be stronger as a result. But such endeavour does also require engagement so that we learn as we go along: I don’t think we’ll succeed if it’s just “team eio” trying to build one true library. Some questions i have are:

  • There’s been little discussion of Eio’s Fibre and cancellation semantics. Is this sufficient to unify both Lwt and Async’s current models? I do not believe that it is to anyone’s benefit to have two such models in the wild.
  • What might migratory codecs look like for existing code, along the lines of @dbuenzli’s nbcodec or lwt_eio for example.
  • How might live tracing and logging improve with eio-like backends? Can we create an even fancier alternative to the awesome tokio-console for OCaml code?
  • Is the decision to make eio sequential-by-default the right one? It works well for IO-bound code, but does it work when there is more pure computation happening?

I wonder if a ‘IO working group’ along the same lines as we’ve done for the core OCaml upstreaming effort might be appropriate to form at this stage. We need to glue together multiple design requirements, and eventually end up with a single IO library that everyone’s happy to use in the longer term.

26 Likes

A large proportion of the objections EIO is currently receiving is on this issue, not on the content of the rest of your message. As far as I can tell, just about everyone is happy about the rest. However, capability objects are a major source of concern. To keep the discussion productive and not sway towards only stating things that nobody disagrees with, I will pick out this quote and comment on it.


This is the fundamental issue with proposing EIO in its current form as a “standard” I/O library: it would amount to a “Mirage-ification” of how I/O is done throughout the OCaml ecosystem, i.e. forcing the very real syntactic and conceptual overhead of Mirage-style programming on all users even though the vast majority of users don’t need it. A potentially foundational I/O library should not “use” the “opportunity” presented by the release of multicore and effects to sneak in and impose an unnecessary and very visible burden, which is of sectional interest to Mirage, on the rest of OCaml users.

Mirage is truly the only explanation I can think of for capability objects in EIO, and the multiple references to Mirage in your and @talex5’s messages makes me think that this is the real explanation.

  1. Oblique mention has been made of JavaScript in this thread. Speaking as someone who routinely releases projects that are portable between native code, js_of_ocaml, and ReScript, I want to emphasize that from all my experience, capability objects in an I/O library have nothing to do with porting to JavaScript.

  2. Speaking also as someone who has led work on two major I/O library projects (Lwt and Luv), capability objects have nothing to do with ordinary native I/O, neither in terms of composability, nor security, nor any other property I have ever been aware of to date, at least in an OCaml setting.

  3. I challenged all of the other arguments (known to me) in favor of capability objects in my earlier post — are you able to address them?

If not, I think we would have to come to the conclusion that the capability objects aspect of EIO is really related specifically to Mirage:

and is a service to Mirage at the expense of the rest of OCaml. It is essentially a way to make people pay the cost of being portable to Mirage (or almost) even though they don’t need that.

It might then be an example of a conflict that arises from the same group of people working on the future of OCaml in general, and also developing Mirage. If EIO is meant to be foundational, it is wrong to insert something so pervasive, visible, awkward, and annoying, but useful primarily only for Mirage.

Note that I have nothing against Mirage or its development, in fact I intend to use it myself.


Neither I nor @lpw25 made an argument that EIO should be different because it should be conservative or unopinionated.

We made the argument that EIO should be different because of specific technical and design issues, which we began to detail. Only after that, we summarized that detailing as “conservative” or “unopinionated” on those aspects of EIO’s design which we object to.

More directly, even if EIO will be opinionated, it should not be opinionated specifically in favor of Mirage.

There is no question about EIO being conservative, opinionated, or experimenting on any of the other things. Your comment is therefore dismissive, because it deliberately blurs focus, from real technical objections on a specific issue, to generalities, rhetoric, and listing everything else about EIO.


There is another argument about what stage of life OCaml is in:

  1. OCaml effects and multicore are probably the largest opportunity OCaml will ever get to attract a large general-purpose user base.
  2. We should not waste this opportunity with awkward, overly experimental libraries that pursue sectional, strongly biased, or academic interests and will naturally turn away users.

Note that I am not saying that everything EIO is doing is biased, academic, or impractical. I am commenting on one, but very visible and pervasive aspect of its current API. It is the very first thing someone will notice when using EIO (and, I claim, will keep noticing and being justifiably annoyed by).

14 Likes

I’ll be the first to volunteer :slight_smile:

5 Likes

I have to agree with @antron here. It seems like the decision was made to cram a massive number of features that are Mirage-exclusive into a library all OCaml users want in general. There is experimentation going on in this library with features people outside of Mirage simply do not care about or want anything to do with. The result will be the exact opposite of what the purported aim was: rather than unifying the community, it will cause further fragmentation.

As a further general point, I should mention that I’m not sure how invested we should be into effects in general before typed effects have arrived on the scene. My fear with introducing effects too early was that introducing the typed version later on will be difficult, and yet without typed effects OCaml loses a large amount of its type safety. I sincerely hope this isn’t going to be the case.

I’m still not quite sure what you mean about multiple uses of run. Running two event loops in a single domain will never work (Lwt will raise an exception if you try, and Eio probably should too).

To run a new event loop in a new domain, you can spawn one using env. That ensures that the new loop will be of the same type, and so the resources from one will work in the other. e.g.

let () =
  Eio_main.run @@ fun env ->
  Eio.Domain_manager.run env#domain_mgr (fun () ->
      Eio.Flow.copy_string "Hello, from a new domain!\n" env#stdout
    )

We need to check a bit to make sure everything is thread-safe. I’m a bit nervous about sharing FDs across domains because if one domain closes an FD as another it using it, there is a possible race where it might access an unrelated FD that just got opened with the same number. However, given that the stdlib already has a much worse version of this problem even within a single domain, we can probably live with it for now!

[ Example of implementing directory sandboxing using nested effect handlers ]

Interesting. I suspect this won’t quite work if you try to pass a directory from one fibre to another, though, since only the original fibre will have the appropriate handler in its stack.

That’s not to say that the capability approach is a bad design though. On the contrary, I think that it might combine very nicely with some of the work we’ve been doing at Jane Street on local allocations, allowing us to have a safe effect handlers without/before having a full blown effect system in the language. I’m very interested in exploring this direction.

I remember hearing about some local allocations stuff in the Signals and Threads podcast. That would be very useful! We have several APIs where we pass a slice of a buffer to a callback, and ask users not to continue using it after the callback returns. Would be great to ensure that statically.

It still requires users to understand class syntax to read the API…

odoc hides the class definition by default, which is useful here. Possibly it needs a comment saying “You don’t need to read this unless you’re implementing the API yourself” or something?

e.g. in odoc you see this:

class virtual source : object ... end

val read : #source -> Cstruct.t -> int

So the only things you really need to know are:

  1. class foo = ... can be treated as type foo, and
  2. #source can be treated as just source.

However, we do make use of row-polymorphism. For example, in addition to source and sink, we have two_way, which includes both APIs. You have to know that the # allows you to call read on a socket, even though it looks at first like a separate type.

I would note though that Python has objects, methods and classes and is regularly recommended as a first language for children. I’m not saying this is a good thing (objects shouldn’t be first choice if something else will do), but it’s not an advanced topic.

and to decode the more difficult type errors from object types.

Using functions to access objects seems to avoid that problem. Here’s an example, where we try to write to stdin instead of stdout:

let () =
  Eio_main.run @@ fun env ->
  let dst = Eio.Stdenv.stdin env in
  Eio.Flow.copy_string "Hello!\n" dst
                                  ^^^
Error: This expression has type Eio.Flow.source
       but an expression was expected of type #Eio.Flow.sink
       The first object type has no method copy

Taking a method call on every operation also seems like an unnecessary cost.

I did some benchmarking, comparing various schemes here GitHub - talex5/flow-tests: Just for testing. In particular, this compared Conduit 3 (using first-class modules and GADTs) with objects. The conclusion was that for accessing OS resources the speed of a method call hardly matters. In fact, Conduit 3 was slower, but for other reasons, I think. Here’s a simple benchmark with my own GADT version:

module Object = struct

  class type source =
    object
      method read : bytes -> int -> int -> unit
      method close : unit
    end

  let of_channel ch =
    object (_ : source)
      method read buf off len = really_input ch buf off len
      method close = close_in ch
    end

  let read (source : #source) = source#read
end

module Gadt = struct
  module type SOURCE = sig
    type t

    val read : t -> bytes -> int -> int -> unit
    val close : t -> unit
  end

  type source = Source : (module SOURCE with type t = 'a) * 'a -> source

  module Channel_source = struct
    type t = in_channel

    let read = really_input
    let close = close_in
  end

  let of_channel ch =
    Source ((module Channel_source), ch)

  let read (Source ((module Source), source)) buf off len =
    Source.read source buf off len 
end

let time_object ch =
  let source = Object.of_channel ch in
  let buf = Bytes.create 4096 in
  let t0 = Unix.gettimeofday () in
  for _i = 1 to 1_000_000 do
    Object.read source buf 0 4096
  done;
  let t1 = Unix.gettimeofday () in
  Printf.printf "Time with object: %.3f\n" (t1 -. t0)

let time_gadt ch =
  let source = Gadt.of_channel ch in
  let buf = Bytes.create 4096 in
  let t0 = Unix.gettimeofday () in
  for _i = 1 to 1_000_000 do
    Gadt.read source buf 0 4096
  done;
  let t1 = Unix.gettimeofday () in
  Printf.printf "Time with module: %.3f\n" (t1 -. t0)

let () =
  let zero = open_in "/dev/zero" in
  time_gadt zero;
  time_object zero
$ dune exec -- ./test.exe
Time with module: 0.243
Time with object: 0.243

This is for reading from /dev/zero, which is very fast for the kernel. If you’re reading from a file or socket then obviously the kernel will be doing more work and the speed benefit (if any) will be smaller.

However, we also want to use sub-types, which is much easier with objects. As well as source, sink and two_way, some flows can be closed while others can’t. File sources should also allow pread, and sockets should allow send_msg. When using io_uring, you should be able to get the file descriptor, etc.

One other thing I forgot to mention: Eio isn’t entirely made of objects! For example, the buffered reader is a regular record type. So when you do e.g.

let r = Buf_read.of_flow flow ~max_size:1_000_000 in
Buf_read.line r

Here, reading lines or characters is a direct static call. We only make a method call when the buffer needs refilling, to get the next chunk.

Finally, for really high-performance tasks you can also query an object for faster options. In the cat example, there are only two methods calls for the entire transfer: one asking the destination to copy the source, and one with the destination file object asking the source if it has an FD. The actual copying happens by calling splice in a loop on the FD, with no objects involved.

2 Likes

Good :slight_smile:

I was surprised that I had to first await_readable and then call Unix.read. I guess that’s quite expected for a 0.1 version though.

I was assuming you had some existing API that needed a Unix.file_descr and you wanted to use it within an Eio program. If you control all the code, you shouldn’t need to use Unix at all. Just open it as a flow and use Flow.read.

By the way, does creating fibres from inside fibres changes anything to their scheduling?

No, all fibres are equal within a domain. See e.g. https://ocaml-multicore.github.io/eio/eio/Eio/Fibre/index.html#val-both for the details.

1 Like

There is a lively discussion of the “Cross-Platform OS API” so far, but I haven’t seen much on the concurrency primitives.

(I wonder if it would be possible to split the two parts, so that people want to use the Concurrency API even if they don’t like the particular style of the OS API. @talex5 had a good answer about why it’s natural to tie the two together, but one still wonders about whether we could have a good API, with effect handlers, to build the concurrency abstractions on a small core, and provide the event loops separately.)

I had a brief skimming look at the Concurrency API and here are some idle comments.

  • As @talex5 mentioned, the main design choice that stands out is the choice to support cancellation pervasively, with a Switch abstraction to group fibres together for cancellation.
  • The description of Promise makes it look like a Stream of size 1. (Except one cannot push several times to it, so I guess it’s a single-push stream of size 1?). Is there an actual connection in the implementation, or should the relation be mentioned in the documentation?
  • The documentation highlights that Promises are not Lazy values. Are there concurrent-Lazy-values somewhere, or how hard are they to build on top of what there is?
  • I see Semaphore but no Mutex. Is the idea to push everyone to the more “reentrant” behavior of semaphores? (Should this be documented?) Is there a performance impact?
  • Spawning a Fibre uses functions called fork, the name comes with some Unix baggage that isn’t relevant here. Maybe this could be pointed out.
5 Likes

This is all quite contradictory. “You don’t need to read this…”, but then you also do need to know these details to understand what it enables you to do.

In Python, objects, methods and classes are idiomatic. It’s used so pervasively that it becomes internalized, and the cost of using it for an additional feature is virtually zero. In OCaml, on the other hand, it’s not idiomatic. And so using this one feature adds an additional cost of having to learn a bunch of new and non-idiomatic concepts. Whether or not the topic is “advanced” isn’t as significant as the fact that there’s still a lot to it, and it’s quite “weird” from both a functional and mainstream OOP POV.

1 Like

Is this a good time to ask a different question?

How similar and different are eio’s fibers to go’s goroutines and elixir’s processes?

I’d like some suggestions on how best to implement an experimental in-memory pubsub library with the availability of domainslib and eio. In Go and Elixir, the mental model to declaring processes and sending messages are clear: use goroutines, tasks, channels, etc.

I had partially implemented a pubsub library with domainslib channels (and spawning different domains from a taskpool). But now I’ve discovered eio fibers which may be a better fit. Point being is that I’m still unaware of the expected behaviors of ocaml’s concurrency and parallel utils. Can fibers communicate across domains? If a fiber crashes, how do I handle that? Caveat, I have not read all available docs on domains and eio yet.

End goal: build a Phoenix equivalent in Ocaml.

2 Likes

There is even a past one, as mentioned in the HP Emily report from 2006. Though that was clearly very much a proof-of-concept (and I haven’t tried using it). It’s based on E, which I used for a few years and is/was fairly complete. But this bit of the report is the key:

The file objects used here are not part of the OCaml standard library. In OCaml, the functions for file manipulation are spread across a number of modules and functions. Unsurprisingly, they are ill-suited for an object-capability environment.

It’s very hard to add a safe mode to a language if it prevents using the standard IO features! But on the other hand, having an ocaps style IO library is still pleasant and useful without the safe mode, so let’s start there instead.

On security or reasoning, given how varied real OCaml code is, including linked-in libraries, what realistic threat do these capabilties actually protect against?

The biggest one is access outside a service’s intended directory. With the stdlib, if I do:

let tmpdir = Filename.concat downloads name in
Unix.mkdir tmpdir 0o700;
Zip.extract ch tmpdir

then we really have no idea what it might do. There could very easily be a bug in the Zip module that allows an archive to write outside of tmpdir. For example, by first extracting a symlink foo -> /root and then extracting foo/.bashrc on top of it.

By contrast, if we do:

Eio.Dir.mkdir downloads name ~perm:0o700;
Eio.with_open_dir downloads name (fun tmpdir ->
  Zip.extract flow tmpdir
)

then this can’t happen. tmpdir only provides Zip with access to the new directory. Also, if name itself is malicious (perhaps we foolishly let the user specify the name when they uploaded the archive) then at least we can’t create anything outside of the parent downloads directory.

If the Zip module itself is malicious, this doesn’t help us (it could ignore Eio and use Unix directly), but it does still protect from malicious input data.

Are OCaml function calls and modules really the right “boundary” at which analysis should be done? Is there a discussion of this available?

Capabilities work at many levels: e.g. controlling access within a programming language, between processes on a computer, or between computers on a network. See Robust Composition: Towards a Unified Approach to Access Control and Concurrency Control for a gentle introduction. It’s a thesis, so quite long, and goes much further than Eio does.

I don’t agree that there is an analysis advantage to passing object capabilties. In the OCaml context, as far as I can tell, passing these capabilties around, from an analysis point of view, amounts to forcing users to manually execute pieces of the work of a static analyzer,

The key here is that a capability both grants access to and designates a resource. It’s not just about granting access. For example, in the code above we’re not just granting Zip access to a directory, but also telling it which one to use (which we had to do anyway).

Likewise, instead of just passing a clock to allow a module to perform a timeout, it might be more useful to specify what the timeout should be too. Though I don’t want to get hung up on clocks; it’s not a particularly interesting case, and we can make an exception for that if needed.

On composability, I don’t see what pervasive required use of capabilities offers over ordinary composable streams

Capabilities are ordinary streams. Normal OCaml programming is capability programming. If you create two hash tables and pass them to a function, you’re doing capability programming:

let () =
  let a = Hashtbl.create 10 in
  let b = Hashtbl.create 10 in
  foo a b

What is not capability programming is using global variables and then passing the names of things in the global state:

let root = Hashtbl.create 100

let foo name1 name2 = ...

let () =
  Hashtbl.add root "a" @@ Hashtbl.create 10 in
  Hashtbl.add root "a" @@ Hashtbl.create 10 in
  foo "a" "b"

Of course, no OCaml programmer would suggest doing that for hashtables. Eio is simply treating directories here the same as we would treat hashtables.

Is there an example of a widespread object capability-passing language or library ecosystem, with many developers and some years of real-world experience?

FreeBSD’s Capsicum is one example. It’s used by many of its standard executables. I see talks going back to 2010. There are various case-studies listed on that page.

That’s working at the OS level, of course. So you program in that style, but the language doesn’t enforce it, which is the same thing you get with Eio (except that it is enforced at the OS level). Hopefully Eio programs will be able to work with that without changes, given a FreeBSD backend.

7 Likes

But on the other hand, having an ocaps style IO library is still pleasant and useful without the safe mode, so let’s start there instead.

I think that’s where the pushback is. Not everyone seems to agree with that. Generally speaking, I’ve tried in the past to do advanced type magic in OCaml, to increase type safety (it was a lot of GADTs), and it turned out to be a lot of pain for little gain; I think it’s generally the case that OCaml has a natural balance between type safety and simplicity and it’s more or less what we’ve been using for decades. Forcing every user to trade simplicity for more safety (how much isn’t that clear to me) is the wrong way.

The example about Zip seems like twice the same code, just with a nicer API in the second case. I’d also use the with_… idiom for, say, temporary files, no capability required.

Citing a lot of ocap papers is not the point here. E is really interesting, as is capnproto, but these are still niche technologies that are less mainstream than even OCaml is. Betting the 5.0 temporary hype farm on this seems dangerous to me. I’d hope for lwt but without the monadic part, personally. No experimental change of API beyond that.

4 Likes

This is a very long way of saying “no.” I specifically asked for future plans, and you replied with past examples (of which I am well aware). The issue is that you are promoting a programming style in a language in which that style’s usefulness is very limited, and you vaguely refer to extension that may increase the usefulness of this style in the future, yet when asked you cite past examples, and state that programming in the style you are imposing is simply “pleasant,” and useful, which is debatable.

Why start somewhere else? I directly asked a question about future plans based on this paragraph in the README:

Since OCaml is not a capability language, code can ignore Eio and use the non-capability APIs directly. However, it still makes non-malicious code easier to understand and test and may allow for an extension to the language in the future. See Emily for a previous attempt at this.

The subject of my question is directly relevant to the usefulness of capability objects. Why simply redirect with a bunch of rhetoric?

Interestingly, even quote you picked out in this paragraph directly suggests one of the reasons capability objects are not that useful when considered in the context of OCaml and the existence of other approaches, as opposed to their advantages in an idealized setting where you are not comparing them to anything.


As you already said,

This doesn’t protect you from bugs either, since a bug in the Zip module is the same case as it being malicious (at best a matter of degree). This discussion again sounds plausible in an idealized world that is not OCaml. In OCaml, when you are faced with an unknown, unanalyzed Zip module with unknown bugs, malicious behavior, and responses to user inputs, you will be highly motivated to run it in a container, run program analysis tools on it, or review it, whether it is (or appears to be) written in an object capability style or not.

No competent person actually interested in security or correctness could ever look at an OCaml program, see that it appears to be written in an object capability style, and conclude ANY security or correctness properties from that without doing all the other work they would have to do anyway.

That means that the incremental advantage of capabilities is extremely low. Capability objects seem advantageous in an idealized theoretical setting, or a setting in which much of real OCaml has been somehow disabled or disallowed for the sake of the exercise. As you implied above, there are no concrete plans to do something like that to general OCaml in the future.

Furthermore, just like there is no reason why Zip must be safe even if it appears to use capability objects, there is no reason why Zip can’t be written in a capability object-like style even when the underlying libraries don’t require it, and many OCaml libraries or functions are already written in such a style. The question is why would EIO, if taken to be foundational, impose such a style regardless of whether it is needed or not in any particular case? Especially when in OCaml, there is no actual connection between object capability style and any actual security or safety properties that one could depend on. Object capability style in OCaml is fundamentally a gimmick or something some APIs provide as a convenience for their users. That is the dual of imposing it on users, as EIO would do if foundational.


To play along with the rhetoric here, why don’t we just use streams then? Lwt has Lwt_stream and uses it in a few places. Does that make it a library written in object capability style? Does Lwt impose this style on all its users, or muse at length about the advantages of “streams” in an idealized setting?


Capsicum looks like a counter-example to object capabilities at function and module granularity. Note the specifics of my objection:

Capsicum, AFAICT, inherently affects programs only at the system call boundary, and from the outside at the process boundary. It does not itself impose a pervasive programming style on the entire program, its internal control flow, its syntax, etc. It’s very much similar to using containers and other such techniques, as I already mentioned above. It’s an example of the “other techniques” that I am asking you to compare capability objects to, in order to see if they actually offer any incremental improvement in realistic settings. Especially when compared to their much higher cost than containers, Capsicum, etc. (that one will pay anyway).