[ANN] Lwt.6.0.0~alpha (direct-style)

Ecosystem
, ,
21

If that turns up issues, I suppose one work-around, albeit sub-optimal, is to restore the effect of Lwt_unix.(set_default_async_method Async_none), and copiously document the need to call it if invoking anything in Lwt_process or Lwt_unix.fork and Lwt_unix.system and so forth. But the best advice is probably not to try spawning new processes in any Lwt program.

22

I think @raphael-proust 's answer is excellent, but Iā€™d like to add a couple things:

  • the implementation is roughly 130 loc, some of which related to local storage, so itā€™s reasonable to read it all to understand how it works
  • lwt_direct is designed to be a very light addition to lwt that does as little as possible and relies on lwt as much as possible, running in the same scheduler, etc. so it doesnā€™t add parallelism or a new kind of future.
  • run and await are similar to domainslibā€™s eponymous functions: await (the desirable part) allows direct style code to wait for promises, but it needs to run in a context with an event handler, hence run.
3 Likes
23

Not generators, but CPS. https://ocsigen.org/js_of_ocaml/latest/manual/effects.

2 Likes
24

Small update:

  • The branch lwt-6 now includes the multi-domain feature. You can start the Lwt scheduler once in each domain.
  • There is a known bug causing the direct-style tests to stall (waiting for some file descriptors to be closed). I will make an alpha01 release once this is resolved.

Some more details about the mutli-scheduler model can be found on this earlier announcement/request-for-feedback: Multi-domain, multi-scheduler Lwt

1 Like
25

It is worth reading the implementation of lwt_direct.ml.

Itā€™s an enjoyable 140 lines of OCaml code (only!).

As all things Lwt and effects, understanding the ā€œnormalā€ scenario is general easy but it quickly gets hairy when exceptions/errors enter the room. But reading lwt_direct.ml is still quite educational !

1 Like
26

version 6.0.0~alpha01 is now ready (2 packages from ocsigen/lwt at 6.0.0~alpha01 by raphael-proust Ā· Pull Request #28222 Ā· ocaml/opam-repository Ā· GitHub)

Anyone able to test the release on their existing code, Iā€™d welcome the feedback: does it work? There shouldnā€™t be breakages but then there might. If you are not sure how to test or what to test, donā€™t hesitate to ask (on here or via dm or via discord).

27

Loving this, but unless Iā€™m mistaken, lwt_direct depends on lwt.unix, making it unusable with jsoo ?

28

It does.
I think it can be made to work with jsoo but jsoo needs to provide an equivalent of the scheduler hooks. Pretty sure this is doable. One of the alpha bumps (maybe alpha02 maybe alpha03) will focus on jsoo.

Things that need to happen before a non-alpha release

  • make lwt_direct jsoo compatible
  • notify all packages that depend on (the long long deprecated) Lwt_main.exit_hooks (and other such hooks) so they use Lwt.Exit_hooks (and such) instead
  • notify all packages that depend on (the long long deprectaed) lwt_log library to use logs_lwt instead
  • restore backwards compatibility of send_notification and provide a separate send_notification_to_other_domain function
  • a lot of testing
  • whatever else testing uncovers
6 Likes
29

Hello, compilation is failing on the alpha01 tag:

opam switch create lwt_direct 5.3.0
eval $(opam env)
opam pin add lwt_direct git+https://github.com/ocsigen/lwt#6.0.0-alpha01
...
### output ###
# [...]
# (cd _build/default && /Users/yawarquadiramin/.opam/lwt_direct/bin/ocamlc.opt -w -40 -g -bin-annot -bin-annot-occurrences -I src/direct/.lwt_direct.objs/byte -I /Users/yawarquadiramin/.opam/lwt_direct/lib/bytes -I /Users/yawarquadiramin/.opam/lwt_direct/lib/lwt -I /Users/yawarquadiramin/.opam/lwt_direct/lib/lwt/unix -I /Users/yawarquadiramin/.opam/lwt_direct/lib/ocaml/threads -I /Users/yawarqu[...]
# File "src/direct/lwt_direct.ml", line 74, characters 19-34:
# 74 |     Domain.DLS.set current_storage (modify_storage k (Some v) (Domain.DLS.get current_storage))
#                         ^^^^^^^^^^^^^^^
# Error: The value "current_storage" has type "Lwt.Private.storage ref"
#        but an expression was expected of type "'a Domain.DLS.key"
# (cd _build/default && /Users/yawarquadiramin/.opam/lwt_direct/bin/ocamlopt.opt -w -40 -g -I src/direct/.lwt_direct.objs/byte -I src/direct/.lwt_direct.objs/native -I /Users/yawarquadiramin/.opam/lwt_direct/lib/bytes -I /Users/yawarquadiramin/.opam/lwt_direct/lib/lwt -I /Users/yawarquadiramin/.opam/lwt_direct/lib/lwt/unix -I /Users/yawarquadiramin/.opam/lwt_direct/lib/ocaml/threads -I /Users/y[...]
# File "src/direct/lwt_direct.ml", line 74, characters 19-34:
# 74 |     Domain.DLS.set current_storage (modify_storage k (Some v) (Domain.DLS.get current_storage))
#                         ^^^^^^^^^^^^^^^
# Error: The value "current_storage" has type "Lwt.Private.storage ref"
#        but an expression was expected of type "'a Domain.DLS.key"
30

The issue is that lwt_direct.6 depends on lwt.6 so you need to

opam pin add lwt git+https://github.com/ocsigen/lwt#6.0.0-alpha01

and then

opam pin add lwt_direct git+https://github.com/ocsigen/lwt#6.0.0-alpha01

Architecturally, lwt_direct is a separate library which provides a sort of ā€œside schedulerā€ using effects to plug into the main scheduler.

1 Like
31

I looked into this more.

The code seems to support async_none. So I dug a bit further.

There is actually an issue with the documentation of Lwt: async_none is still very much working. It was slated for removal/deprecation but plans changed after some discussion (Deprecate the Lwt_unix.async_method machinery and eventually remove it Ā· Issue #572 Ā· ocsigen/lwt Ā· GitHub). Iā€™ll remove the incorrect deprecation warning.

Iā€™ll also add some documentation about the safety of forking.

32

@cvine Here is the documentation fixes Iā€™ve made: Lwt 6 (NOT FOR MERGE AS IS) by raphael-proust Ā· Pull Request #1073 Ā· ocsigen/lwt Ā· GitHub