# Monadic Library for Eio Capabilities? **URL:** \ **Category:** Learning\ **Tags:** effects\ **Created:** [February 27, 2024, 3:18pm UTC](https://discuss.ocaml.org/t/monadic-library-for-eio-capabilities/14202 "2024-02-27T15:18:08Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![bluddy](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/bluddy/32/104_2.png) [@bluddy](https://discuss.ocaml.org/u/bluddy)\ **Post date:** [February 27, 2024, 3:18pm UTC](https://discuss.ocaml.org/t/monadic-library-for-eio-capabilities/14202/1 "2024-02-27T15:18:08Z")
I would like to get into programming with OCaml 5 effects, but I’m not sure which scheduling library to use. From what I can tell, `eio` requires threading capabilities through the program, which seems like it could be painful, especially after hearing some feedback from community members. I wonder - can we use a monad to carry the capabilities around painlessly? Does anyone know of a monadic library that is able to do such a thing? `miou` seems like it doesn’t require this, but doesn’t seem to be as popular yet. Then there’s `moonpool` as well, made by Container’s own @c-cube. But perhaps a monadic solution with eio is a good way to go? ---
**Author:** ![talex5](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/talex5/32/353_2.png) [@talex5](https://discuss.ocaml.org/u/talex5)\ **Post date:** [February 27, 2024, 5:13pm UTC](https://discuss.ocaml.org/t/monadic-library-for-eio-capabilities/14202/2 "2024-02-27T17:13:43Z")
> [@bluddy](#): > > From what I can tell, `eio` requires threading capabilities through the program `eio` itself doesn’t, and the main backends (eio\_linux, eio\_posix, eio\_windows) all define a `Low_level` API that treats the network and filesystem as global variables, like Lwt does, and then add a capability layer on top of that. However, you will likely find programming much easier without globals everywhere, so I would encourage giving it a try first! Every program I’ve converted from Lwt to Eio got quite a bit shorter, so whatever the pain is, it’s less than the pain of using Lwt. The [Lambda Capabilities](https://roscidus.com/blog/blog/2023/04/26/lambda-capabilities/) blog explains why you usually don’t want to hide references, and gives some reasons why it shouldn’t be a problem (and how to do it if you still want to). ---
**Author:** ![yawaramin](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/yawaramin/32/3384_2.png) [@yawaramin](https://discuss.ocaml.org/u/yawaramin)\ **Post date:** [February 27, 2024, 11:03pm UTC](https://discuss.ocaml.org/t/monadic-library-for-eio-capabilities/14202/3 "2024-02-27T23:03:54Z")
Are there some examples out there of real-world-ish codebases that were ported from Lwt to Eio? I think it would motivate a lot of people if we could see Eio in action. ---
**Author:** ![bluddy](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/bluddy/32/104_2.png) [@bluddy](https://discuss.ocaml.org/u/bluddy)\ **Post date:** [February 28, 2024, 7:11am UTC](https://discuss.ocaml.org/t/monadic-library-for-eio-capabilities/14202/4 "2024-02-28T07:11:01Z")
> [@talex5](#): > > `eio` itself doesn’t, and the main backends (eio\_linux, eio\_posix, eio\_windows) all define a `Low_level` API that treats the network and filesystem as global variables, like Lwt does, and then add a capability layer on top of that. So is this a viable thing? To put the capabilities in a global variable? There are no race conditions? Would that work for multiple domains as well? Does eio’s interface support just filing the capabilities away if you don’t care about them? > [@talex5](#): > > Every program I’ve converted from Lwt to Eio got quite a bit shorter, so whatever the pain is, it’s less than the pain of using Lwt. I’ve seen several people on discord saying that they are thinking about going back to Lwt rather than keep using capabilities. > [@talex5](#): > > The [Lambda Capabilities](https://roscidus.com/blog/blog/2023/04/26/lambda-capabilities/) blog explains why you usually don’t want to hide references, and gives some reasons why it shouldn’t be a problem (and how to do it if you still want to). I have to try it myself, but my concern is that here was an opportunity to finally unify the community around a concurrency library, and instead, something was added to the most heavily promoted library, that is causing even more fracturing than we had before. From what I understand, even Mirage doesn’t use eio, and that seems to me to be the most potentially useful application of capabilities in the ecosystem. From other people I’ve talked to in the community, virtually all of them dislike it and find the costs of capabilities outweigh what they consider fairly niche benefits. And they’re also concerned that they made their concerns known and were ignored. ---
**Author:** ![raphael-proust](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/raphael-proust/32/415_2.png) [@raphael-proust](https://discuss.ocaml.org/u/raphael-proust)\ **Post date:** [February 28, 2024, 1:13pm UTC](https://discuss.ocaml.org/t/monadic-library-for-eio-capabilities/14202/5 "2024-02-28T13:13:25Z")
This message from a relative novice of Eio. I haven’t used it at scale. I welcome corrections. It seems to me Eio was designed to take the place of both Lwt and Async in the ecosystem (to remove the community split). In doing so, it seems the decision was made to make a library which could do all that Lwt could do and all the Async could do and more. But this is mostly a problem of opinionation. More specifically: It’s actually a modular set of libraries sold as a whole package. - The `eio.core` library exposes the basic concurency concepts (fibers, promises, switches, etc.) as well as their semantic (exception management, yielding, etc.). - As mentioned above the backends expose a `Low_level` module so you can interact more or less directly with the backend (e.g., `Eio_linux.Low_level` has basically all of the `Unix` functions albeit somewhat fancier with some labels and such). - The `eio` library does backend-agnostic, capabilities-enforced OS interaction - The `eio_main` library can run your code regardless of which of the modular bits above you are using. And so sure you can go capability-less, but - a module called `Low_level` located in a separate library is not very inviting, - you are writing code for one backend specifically so your code won’t run on windows, - you have to read through a large-ish amount of documentation before you learn that it is possible, - you still have the capabilities stuff in your cone of dependence. Essentially, Eio is branded as a all-in-one capability-full plateform-agnostic concurrency library. The code is also organised this way. And going capability-less is going against the grain of the library. * * * > [@talex5](#): > > Every program I’ve converted from Lwt to Eio got quite a bit shorter, so whatever the pain is, it’s less than the pain of using Lwt. Conciseness and terseness is a bad measure for pleasantness. E.g., `↑0 1↓↑+.×/5/⊂2 2⍴1 1 1 0` ---
**Author:** ![patricoferris](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/patricoferris/32/3621_2.png) [@patricoferris](https://discuss.ocaml.org/u/patricoferris)\ **Post date:** [February 28, 2024, 1:44pm UTC](https://discuss.ocaml.org/t/monadic-library-for-eio-capabilities/14202/6 "2024-02-28T13:44:26Z")
I thought I would do a minimal experiment to have a networking interface that doesn’t require a networking capability: [GitHub - patricoferris/neio](https://github.com/patricoferris/neio) – this is just using the posix and linux (I don’t have a windows machine currently) low level interfaces and wrapping them in what is mostly a small subset of the `Eio.Net` interface and making some different opinionated choices (e.g. no capabilities, removing some of the uses of subtying with polymorphic variants and instead providing conversion functions to flows or fds etc.). Hopefully this is useful to imagine what Eio offers in terms of taking care of a lot of the _low level_ OS interactions in a safe way (structured concurrency, reference counted file descriptors etc.) and you are free to choose how to wrap that and present it to an end-user :)) ---
**Author:** ![emillon](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/emillon/32/1206_2.png) [@emillon](https://discuss.ocaml.org/u/emillon)\ **Post date:** [February 28, 2024, 2:16pm UTC](https://discuss.ocaml.org/t/monadic-library-for-eio-capabilities/14202/7 "2024-02-28T14:16:37Z")
I don’t have a lot of experience with effects, but it seems to me that you could hack your way using the following technique: instead of passing capabilities as arguments, use a dynamically scoped variable (similar to react’s contexts; that can be expressed using effects) to pass them down the program without an explicit argument being passed. Would that work? ---
**Author:** ![c-cube](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/c-cube/32/1727_2.png) [@c-cube](https://discuss.ocaml.org/u/c-cube)\ **Post date:** [February 28, 2024, 3:04pm UTC](https://discuss.ocaml.org/t/monadic-library-for-eio-capabilities/14202/8 "2024-02-28T15:04:47Z")
> [@raphael-proust](#): > > It seems to me Eio was designed to take the place of both Lwt and Async in the ecosystem (to remove the community split). I like this phrasing, and that’s what I thought Eio was aiming to be. But instead we get an opinionated monolithic library. It might be well designed and tasteful, but opinions are bound to create disagreements. Packaging/maintaining the foundational libraries (fibers, low level IOs, set of effects, local storage) separately in libraries other people can build on _should_ have been the way. It was not done this way and I’ve lost hope that it will. ---
**Author:** ![dinosaure](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/dinosaure/32/16_2.png) [@dinosaure](https://discuss.ocaml.org/u/dinosaure)\ **Post date:** [February 28, 2024, 3:22pm UTC](https://discuss.ocaml.org/t/monadic-library-for-eio-capabilities/14202/9 "2024-02-28T15:22:49Z")
> [@raphael-proust](#): > > The `eio` library does backend-agnostic, capabilities-enforced OS interaction OS-agnostics seems a little optimistic to me in view of the `Path` module offered by `eio`, which seems very `Unix`-directed (by providing elements that don’t exist on Windows). I’m reminded of [this article](https://fasterthanli.me/articles/i-want-off-mr-golangs-wild-ride) on Go which points out the supposed simplicity of the API. Personally, I prefer to expose the complexity of a system rather than hide it behind a “beautiful API”. Finally, let’s not even talk about unikernels, which may not have a file system (and where `Path`’s usefulness then becomes nil). ---
**Author:** ![talex5](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/talex5/32/353_2.png) [@talex5](https://discuss.ocaml.org/u/talex5)\ **Post date:** [February 28, 2024, 3:23pm UTC](https://discuss.ocaml.org/t/monadic-library-for-eio-capabilities/14202/10 "2024-02-28T15:23:01Z")
> [@yawaramin](#): > > Are there some examples out there of real-world-ish codebases that were ported from Lwt to Eio? I think it would motivate a lot of people if we could see Eio in action. Here are a couple of mine: - [GitHub - talex5/wayland-proxy-virtwl: Allow guest VMs to open windows on the host](https://github.com/talex5/wayland-proxy-virtwl) - [GitHub - ocurrent/solver-service: An OCurrent service for solving opam dependencies](https://github.com/ocurrent/solver-service/) The Wayland proxy went from 6661 lines of Lwt to 6527 lines of Eio (it’s a bit longer now as new features were added afterwards). It’s easy to measure the cost of capabilities by starting from the main entry-point [main.ml#L127](https://github.com/talex5/wayland-proxy-virtwl/blob/02d8d9f3faa3178fd5ca89efa7321525965d6bc8/main.ml#L127): - Without caps, there would be no `env`, just `()` (saving one character, though we could just a shorter name of course). - It extracts `fs` and uses it to configure tracing. - It passes `env` to `main`, which extracts `fs`, `net` and `proc_mgr`. - `fs` is used to find the virtio-gpu device file in `/dev`. - `proc_mgr` is used to run the user’s application and to launch Xwayland. - `net` is used to connect to the host Wayland compositor, to listen for incoming Wayland connections, and to listen for incoming X11 connections. Following all those uses through, I count 358 characters that wouldn’t have been needed without capabilities. `wc **/*.ml** /*.mli` says there are 231,949 characters in total, so I make that 0.15% extra code. On the other hand, you now have a good idea of every way that this process interacts with its environment, and anyone unfamiliar with this code could also have written the above summary in a few minutes. (note: the tracing code there looks a bit of a mess; I think I wrote that when Eio was missing some file-system functionality) The solver service went from 3247 lines of Lwt to 2792 lines of Eio. However, it’s still using some Lwt libraries (`git-unix` and `capnp-rpc`), so counting extra characters might not get the full cost. But so far I count 302/90,988 = 0.33% overhead. > [@raphael-proust](#): > > Conciseness and terseness is a bad measure for pleasantness. I agree, but as far as I can tell this is the only thing people are complaining about. In terms of maintainability, security or formal verification, I don’t think anyone is disputing the advantages of capabilities? ---
**Author:** ![c-cube](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/c-cube/32/1727_2.png) [@c-cube](https://discuss.ocaml.org/u/c-cube)\ **Post date:** [February 28, 2024, 3:37pm UTC](https://discuss.ocaml.org/t/monadic-library-for-eio-capabilities/14202/11 "2024-02-28T15:37:38Z")
> [@talex5](#): > > In terms of maintainability, security or formal verification, I don’t think anyone is disputing the advantages of capabilities? I think people loudly complain about the maintainability, yes. Threading more stuff around is just a pain an a hindrance to composability, just like monads, but in a different way. If you now need networking in a part of the program that didn’t (e.g for caching via Redis or whatever), here you go, add capabilities to the whole path from main to that part of the program. Personally I already have enough parameters as it is. The benefits to security in a language that doesn’t actually enforce capabilities is, imho, dubious, but let’s have the benefit of the doubt here. In terms of formal verification, I mean, I have no idea? Has anyone used Gospel with capabilities yet? There is research about verification based on capabilities but I doubt it applies in an environment where capabilities are just a convention. ---
**Author:** ![yawaramin](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/yawaramin/32/3384_2.png) [@yawaramin](https://discuss.ocaml.org/u/yawaramin)\ **Post date:** [February 28, 2024, 3:52pm UTC](https://discuss.ocaml.org/t/monadic-library-for-eio-capabilities/14202/12 "2024-02-28T15:52:19Z")
I don’t think you would need to thread the ‘net’ capability throughout your program upto the point you want to use Redis though? You would need the ‘net’ capability at or near the top of your application stack to open a connection to Redis, then you would have to pass that open connection (or a connection pool with automatic reconnection etc.) down to the part of your app that needs to do caching. This would not be very different architecturally from what you would do without capabilities. You’d still need to pass the Redis client/connection to the part of your app that needed it, either way. ---
**Author:** ![raphael-proust](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/raphael-proust/32/415_2.png) [@raphael-proust](https://discuss.ocaml.org/u/raphael-proust)\ **Post date:** [February 28, 2024, 4:30pm UTC](https://discuss.ocaml.org/t/monadic-library-for-eio-capabilities/14202/13 "2024-02-28T16:30:59Z")
> [@talex5](#): > > as far as I can tell this is the only thing people are complaining about I don’t think people complaining about having to thread some arguments through a call graph is the same as people complaining about having more characters in their source code. For example, say you want to add some logging to a function somewhere deep in your call graph and have to add some capability argument to the function and all the stack of functions above it, the cost is not in the characters, it’s more like - The cost of having to go through the whole graph, opening file after file, jumping around the source code, etc. You might get help from your IDE/lsp/editor, but the reason help for these tasks is available in your IDE/lsp/editor is precisely because it is a costly task. - The cost of having multiple merge-request conflict because two small changes that would otherwise be very local are now affecting multiple files adding arguments in a lot of places. - Etc. ---
**Author:** ![beajeanm](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/beajeanm/32/189_2.png) [@beajeanm](https://discuss.ocaml.org/u/beajeanm)\ **Post date:** [February 28, 2024, 11:35pm UTC](https://discuss.ocaml.org/t/monadic-library-for-eio-capabilities/14202/14 "2024-02-28T23:35:19Z")
That’s a design choice. You could also read a config file and open a redis connection deep in the middle of the code and never thread anything. No matter what we think of that approach, it’s not the place of a concurrency to impose these choices to its users. ---
**Author:** ![beajeanm](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/beajeanm/32/189_2.png) [@beajeanm](https://discuss.ocaml.org/u/beajeanm)\ **Post date:** [February 28, 2024, 11:46pm UTC](https://discuss.ocaml.org/t/monadic-library-for-eio-capabilities/14202/15 "2024-02-28T23:46:03Z")
> [@talex5](#): > > I agree, but as far as I can tell this is the only thing people are complaining about. In terms of maintainability, security or formal verification, I don’t think anyone is disputing the advantages of capabilities? Sorry, but may I ask what are the advantages exactly? We’ve read on this thread that some consider it as an hindrance to some code maintenance. The security can be bypassed by not using the eio api. And the portion of software projects which care about formal verification is rather small. ---
**Author:** ![yawaramin](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/yawaramin/32/3384_2.png) [@yawaramin](https://discuss.ocaml.org/u/yawaramin)\ **Post date:** [February 29, 2024, 12:54am UTC](https://discuss.ocaml.org/t/monadic-library-for-eio-capabilities/14202/16 "2024-02-29T00:54:16Z")
A real-world concurrency library could definitely make a design choice to make resources available at the top layer and force users to inject them into lower layers. This is how Scala’s ZIO (and in fact even Erlang) does it. This helps developers of all backgrounds fall into the pit of success with using the library with correct concurrency and safety patterns. Of course they can always bypass it if they want (there is the low-level API after all). ---
**Author:** ![c-cube](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/c-cube/32/1727_2.png) [@c-cube](https://discuss.ocaml.org/u/c-cube)\ **Post date:** [February 29, 2024, 3:38am UTC](https://discuss.ocaml.org/t/monadic-library-for-eio-capabilities/14202/17 "2024-02-29T03:38:51Z")
That sounds like an excellent design choice for a high-level opinionated library for people who like this kind of thing, yes. Less so for something that tries to establish consensus and provide common abstractions to _reduce_ fragmentation (ha!). ---
**Author:** ![yawaramin](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/yawaramin/32/3384_2.png) [@yawaramin](https://discuss.ocaml.org/u/yawaramin)\ **Post date:** [February 29, 2024, 3:50am UTC](https://discuss.ocaml.org/t/monadic-library-for-eio-capabilities/14202/18 "2024-02-29T03:50:22Z")
It sounds like this would be basically solved by splitting Eio into a separate low-level library (core effects and I/O bindings) and a high-level cross-platform capability-driven library in the same repo? ---
**Author:** ![c-cube](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/c-cube/32/1727_2.png) [@c-cube](https://discuss.ocaml.org/u/c-cube)\ **Post date:** [February 29, 2024, 4:15am UTC](https://discuss.ocaml.org/t/monadic-library-for-eio-capabilities/14202/19 "2024-02-29T04:15:22Z")
Absolutely, it would be good to have a clean separation so that other people cab build their preferred abstractions on top of the lower level library (which would be (mostly) dependency-free, ideally). It’s a bit trickier for a foundation of fibers because it’s harder to not put opinions in there (e.g. structured concurrency and `spawn` are closely tied together) but I haven’t seen a lot of complaints about structured concurrency 🙂. [Picos](https://github.com/ocaml-multicore/picos) is a library designed explicitly to be a foundation other libraries can build on. I think it’s a very interesting effort and it’s pushed by someone who has a lot of multicore experience from other ecosystems (something we should do more, e.g. by porting Tokio from Rust?). ---
**Author:** ![talex5](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/talex5/32/353_2.png) [@talex5](https://discuss.ocaml.org/u/talex5)\ **Post date:** [February 29, 2024, 9:52am UTC](https://discuss.ocaml.org/t/monadic-library-for-eio-capabilities/14202/20 "2024-02-29T09:52:04Z")
> [@c-cube](#): > > If you now need networking in a part of the program that didn’t (e.g for caching via Redis or whatever), here you go, add capabilities to the whole path from main to that part of the program. Interestingly, the blog post mentioned above uses this exact scenario (using Redis for caching) as an example of a case where you might think you’d need to pass `net` around but don’t (see [Plumbing capabilities everywhere](https://roscidus.com/blog/blog/2023/04/26/lambda-capabilities/#plumbing-capabilities-everywhere)). > [@c-cube](#): > > In terms of formal verification, I mean, I have no idea? The blog post gives an example of reasoning about the safety of a web-server; see [Web-server example](https://roscidus.com/blog/blog/2023/04/26/lambda-capabilities/#web-server-example). > [@c-cube](#): > > There is research about verification based on capabilities but I doubt it applies in an environment where capabilities are just a convention. There are two cases to consider: depending on a library or application that is actively malicious vs one that is merely buggy/confused. Without support from the compiler, we can only fix the second case. The web-server example above illustrates this with the case of a web-server that is tricked into revealing its private key. It is possible to ask the OS to drop privileges soon after a program starts, and if the program is using only the high-level (recommended) Eio APIs then it should continue working just as before. The [Filesystem Access](https://github.com/ocaml-multicore/eio?tab=readme-ov-file#filesystem-access) section of Eio’s README shows an example of this (at the end), where using the stdlib as `open_in "/etc/passwd"` is rejected with `exception Sys_error("/etc/passwd: Not permitted in capability mode")`. However, even this won’t protect you from malicious code, as OCaml runs static initialisers in all dependencies before your application can drop privileges. > [@raphael-proust](#): > > For example, say you want to add some logging to a function somewhere deep in your call graph and have to add some capability argument to the function and all the stack of functions above it There’s no need to do that (the example programs I mentioned above both have logging). Modules writing log messages can’t use that to communicate with each other - logging is write-only from their point of view - and restricting logging isn’t useful from a security point of view, so we don’t bother to restrict it. Notice that Eio provides `Eio.traceln` that can write to stderr (by default) from anywhere. It does this using fiber local storage. This is discussed in the blog post in the section [Thread-local storage](https://roscidus.com/blog/blog/2023/04/26/lambda-capabilities/#thread-local-storage). > [@beajeanm](#): > > Sorry, but may I ask what are the advantages exactly? In the example I gave above, anyone can see how the Eio version of the wayland-proxy interacts with the outside world after studying the code for a couple of minutes. If you were asked to maintain or deploy this code, you’d probably find that pretty useful. For example, the Lwt version of the solver service I linked above has a security vulnerability. Can you find it? > [@bluddy](#): > > my concern is that here was an opportunity to finally unify the community around a concurrency library, and instead, something was added to the most heavily promoted library, that is causing even more fracturing than we had before. I think this is backwards. Rust made the opposite choice, not using capabilities in `std`. But this forced people wanting to write secure code to create the separate [cap-std](https://github.com/bytecodealliance/cap-std) library, splitting the community. It’s much better to have a default library that everyone can use. For example, someone made a Rust version of sudo using `std` and in [sudo-rs’ first security audit](https://lobste.rs/s/dpahjz) they found a bug allowing users to delete arbitrary files (in rare cases). The highest-rated reply in that discussion says: > The cap-std library in Rust makes this easier than C/C++, so why not just do the right thing from the start if you’re doing greenfield software development? [Next page](https://discuss.ocaml.org/t/monadic-library-for-eio-capabilities/14202.md?page=2)