# Effects with Lwt, a dead end for now?

**URL:** <https://discuss.ocaml.org/t/effects-with-lwt-a-dead-end-for-now/15002>\
**Category:** Learning\
**Tags:** lwt, effects, runtime, eio\
**Created:** [July 20, 2024, 7:23pm UTC](https://discuss.ocaml.org/t/effects-with-lwt-a-dead-end-for-now/15002 "2024-07-20T19:23:48Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![kentookura](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/kentookura/32/5187_2.png) [@kentookura](https://discuss.ocaml.org/u/kentookura)\
**Post date:** [July 20, 2024, 7:23pm UTC](https://discuss.ocaml.org/t/effects-with-lwt-a-dead-end-for-now/15002/1 "2024-07-20T19:23:48Z")

</div>

This thread is basically a discussion about this issue:

> <https://github.com/ocsigen/lwt/issues/1003>
>
> The following code is a minimized version of https://github.com/aantron/dream/is…sues/297 that no longer calls into Dream at all:
> 
> \`\`\`ocaml
> type \_ Effect.t += E : unit Effect.t
> 
> let () =
> Effect.Deep.try\_with
> begin fun () -\>
> Lwt\_main.run begin
> Lwt.bind (Lwt\_unix.sleep 1.) @@ fun () -\>
> Effect.perform E;
> assert false
> end
> end
> ()
> {
> effc = fun (type a) (e : a Effect.t) -\>
> match e with
> | E -\>
> Option.some @@ fun (k : (a, \_) Effect.Deep.continuation) -\>
> prerr\_endline "handling E";
> Effect.Deep.continue k ()
> | \_ -\> None
> }
> \`\`\`
> 
> Running this on OCaml 5.1.0 with Lwt 5.7.0 results in
> 
> \`\`\`
> Fatal error: exception Stdlib.Effect.Unhandled(Dune\_\_exe\_\_Json.E)
> \`\`\`
> 
> Replacing \`Lwt\_unix.sleep 1.\` by, say, \`Lwt.pause ()\` results in
> 
> \`\`\`
> handling E                           
> Fatal error: exception File "dream/example/e-json/json.ml", line 9, characters 8-14: Assertion failed
> \`\`\`
> 
> which is the correct output, and what I would expect in both cases.

I am trying to integrate [irmin](https://github.com/mirage/irmin) into an [application](https://git.sr.ht/~jonsterling/ocaml-forester) that makes heavy use of effects, in particular [Algaeff](https://github.com/RedPRL/algaeff), [Asai](https://github.com/RedPRL/asai) and Eio.

Now, irmin uses lwt and while there is a [solution](https://github.com/ocaml-multicore/lwt_eio) for running lwt and eio code together, but it seems that in general, I can’t make use of code that uses effect handlers.

The question I have is: What is the best option for making progress?

- Is my only option to wait for the issue at the lwt repo to resolve?
- Should I study [Lwt\_eio](https://github.com/ocaml-multicore/lwt_eio/blob/master/lib/lwt_eio.ml)? My thinking is that maybe that code can be generalised/adapted somehow, since it is an example of using effect handlers in conjunction with lwt. It might be a problem that the effect handlers that are causing issues in my code are library code, so I can’t modify it.
- …?

There is an open PR for an Eio backend for irmin, but I assume that using irmin would still require using the monadic lwt API (can’t tell from the PR alone).

> <https://github.com/mirage/irmin/pull/2316>
>
> Follow-up on #2149 to fix the Eio backend of \`irmin-fs.unix\`:
> \- The effects to …access the eio capabilities had to go, as they break as soon as we spawn fibers which will escape the handler scope
> \- Rather, the eio capabilities are passed through the irmin \`Conf\` as this is the least invasive solution that we found... The main issue is that \`irmin-cli\` expects to be able to deserialize the conf values from the command-line / config file: the underhanded solution to this is to parametrize the conf \`Spec\` by the required eio capabilities. Cons: we technically loose spec uniqueness per backend, but this didn't seem to be used anyway other than for sanity checking irmin?
> 
> WDYT of renaming \`irmin-fs.unix\` to \`irmin-fs.eio\`?

I’ll work on putting together a minimal working example of the code that causes my issues so that we have something more concrete to talk about. In the meantime I am eager for discussion. Thanks!

---

<div class="post-metadata">

**Author:** ![patricoferris](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/patricoferris/32/3621_2.png) [@patricoferris](https://discuss.ocaml.org/u/patricoferris)\
**Post date:** [July 21, 2024, 7:25am UTC](https://discuss.ocaml.org/t/effects-with-lwt-a-dead-end-for-now/15002/2 "2024-07-21T07:25:44Z")

</div>

> [@kentookura](#):
>
> There is an open PR for an Eio backend for irmin, but I assume that using irmin would still require using the monadic lwt API (can’t tell from the PR alone).

Hey @kentookura,

The PRs you referenced from Irmin are to remove Lwt as much as possible. The API will be Lwt free – see [the updated store interface](https://github.com/mirage/irmin/blob/7186af2b0bbd1e7250b687277a4ac2865a133507/src/irmin/store_intf.ml). The implementation could use lwt\_eio under the hood for some libraries until they are also ported away from Lwt.

Do share that minimal example once you have it to help explain exactly the issues you are hitting :))

---

<div class="post-metadata">

**Author:** ![kentookura](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/kentookura/32/5187_2.png) [@kentookura](https://discuss.ocaml.org/u/kentookura)\
**Post date:** [July 22, 2024, 3:22pm UTC](https://discuss.ocaml.org/t/effects-with-lwt-a-dead-end-for-now/15002/3 "2024-07-22T15:22:38Z")

</div>

I created a repo:

> **[GitHub - kentookura/irmin-effects-demo](https://github.com/kentookura/irmin-effects-demo/)**
>
> Contribute to kentookura/irmin-effects-demo development by creating an account on GitHub.

---

<div class="post-metadata">

**Author:** ![patricoferris](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/patricoferris/32/3621_2.png) [@patricoferris](https://discuss.ocaml.org/u/patricoferris)\
**Post date:** [July 22, 2024, 11:11pm UTC](https://discuss.ocaml.org/t/effects-with-lwt-a-dead-end-for-now/15002/4 "2024-07-22T23:11:28Z")

</div>

Thanks for the reproduction code @kentookura! I think this might be a case of issues arising due to the nesting of the effect _handlers_ (I wrote about this [in a blog post](https://patrick.sirref.org/posts/escaping-effects.html)). In the example you have:

```ocaml
  Eio_main.run @@ fun env ->
  Reporter.run ~emit:Term.display ~fatal:Term.display @@ fun () ->
  Lwt_eio.with_event_loop ~clock:env#clock @@ fun _ ->

```

The third line forks a new fiber and is handled by the first line escaping the scope of the handler installed by the second line – hence “unhanded effect”. In this case, I think it is fine to fix this by changing lines 1 and 2 so the reporter sits on top. Eio could also perhaps offer a better mechanism for users to add additional effect handlers ([which it does have internally](https://github.com/ocaml-multicore/eio/blob/33d4e01a30aae0ae55747e29c1fc4ea6e741571b/lib_eio_posix/sched.ml#L375)).

I forked the repo and converted it to the Irmin interface in the PR you mentioned here: [irmin-effects-demo/bin/main.ml at escaping-effects · patricoferris/irmin-effects-demo · GitHub](https://github.com/patricoferris/irmin-effects-demo/blob/escaping-effects/bin/main.ml) – it doesn’t fork anymore and so the old code works. However, I added a fork just to help show what was happening. Hope this helps clear things up a little ?

---

<div class="post-metadata">

**Author:** ![kentookura](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/kentookura/32/5187_2.png) [@kentookura](https://discuss.ocaml.org/u/kentookura)\
**Post date:** [July 24, 2024, 2:12pm UTC](https://discuss.ocaml.org/t/effects-with-lwt-a-dead-end-for-now/15002/5 "2024-07-24T14:12:54Z")

</div>

I’ve successfully pinned irmin at the eio branch. However, the code you added does not work for me, as `Irmin_fs_unix.conf` is reported as an unbound value. I made some modifications to the code so it builds, which you can look at on github. Unfortunately, a similar issue arises:  
`Fatal error: exception Stdlib.Effect.Unhandled(Irmin__Conf.Env.Fs)`  
This happens both with both permutations of `Eio_main.run` and `Reporter.run`

---

<div class="post-metadata">

**Author:** ![kentookura](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/kentookura/32/5187_2.png) [@kentookura](https://discuss.ocaml.org/u/kentookura)\
**Post date:** [July 24, 2024, 5:45pm UTC](https://discuss.ocaml.org/t/effects-with-lwt-a-dead-end-for-now/15002/6 "2024-07-24T17:45:51Z")

</div>

This still happens when removing all asai code, so it seems that irmin-fs itself fails:

```auto
module Store = Irmin_fs_unix.KV.Make (Irmin.Contents.String)

let test () =
  let config = Irmin_fs.config "store" in
  let repo = Store.Repo.v config in
  let main = Store.main repo in
  let info () = Store.Info.v 0L in
  let key = "Hello" in
  let _ = Store.set main [key] ~info "world!" in
  let v = Store.get main [key] in
  print_endline v

let () = Eio_main.run @@ fun env -> test ()

```

---

<div class="post-metadata">

**Author:** ![patricoferris](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/patricoferris/32/3621_2.png) [@patricoferris](https://discuss.ocaml.org/u/patricoferris)\
**Post date:** [July 24, 2024, 6:42pm UTC](https://discuss.ocaml.org/t/effects-with-lwt-a-dead-end-for-now/15002/7 "2024-07-24T18:42:49Z")

</div>

Looks like maybe you are pinning Irmin to a slightly older WIP commit – I was trying it with @art-w’s latest irmin-fs library changes: [irmin-fs: eio backend · art-w/irmin@9d606d7 · GitHub](https://github.com/art-w/irmin/commit/9d606d78375abcd4c961331a1e8ee8f9be9212cb)

---

<div class="post-metadata">

**Author:** ![kentookura](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/kentookura/32/5187_2.png) [@kentookura](https://discuss.ocaml.org/u/kentookura)\
**Post date:** [July 24, 2024, 7:21pm UTC](https://discuss.ocaml.org/t/effects-with-lwt-a-dead-end-for-now/15002/8 "2024-07-24T19:21:15Z")

</div>

Yes, you’re right. Thanks a lot!