[ANN] Experimental tools for migrating from Lwt to Eio

Community
1

We developed tools to help us migrate Ocsigen libraries and applications to direct-style concurrency and we are happy to share them.

There are 4 independent tools for different purposes:

  • Replacing every uses of lwt_ppx with Lwt function calls (e.g. replacing let%lwt ... in with let* ... in).
    This is purely syntactic and allows you to easily remove a PPX. It handles inserting open Lwt.Syntax at the top of the file.

  • Warning about occurrences of let _ = ... and ignore ..., which make the next tools less reliable.
    These are called “implicit fork” because code run concurrently if the ignored value is an Lwt thread. This requires an explicit call to Fiber.fork with Eio.

  • Migrating uses of Lwt_log to Logs.
    It was used on ocsigenserver for example.
    At this point we generate code working as before and did not introduce a dependency on Eio.

  • Migrating uses of Lwt to Eio. It rewrites code written in monadic style into direct-style, for example this code:

    let _ =
  let* x = f 1 in
  let+ y = f 2 in
  Lwt.bind (f 3) (fun z ->
    Lwt.return (x + y + z))

    is rewritten to:

    let _ =
  let x = f 1 in
  let y = f 2 in
  let z = f 3 in
  x + y + z

    It also rewrite Lwt function calls to their equivalent in Eio and handles many IO operations.

    Unfortunately, this tool doesn’t generate fully equivalent code and requires manual modifications. Comments are inserted in many places where intervention is needed.

    An example of generated incorrect code is:

    let _ =
  let a = operation_1 () in
  let* b = operation_2 () in
  let* a = a in
  Lwt.return (a + b)

    which is rewritten into:

    let _ =
  let a = operation_1 () in
  let b = operation_2 () in
  let a = a in
  a + b

    This is incorrect because operation_1 and operation_2 are now sequential but were concurrent before. This is a case of an implicit fork that cannot easily be detected. The correct transformation would be:

    let _ =
  let a, b = Eio.Fiber.pair operation_1 operation_2 in
  a + b

    We could achieve the migration on eliom and ocsigenserver

All the tools use OCamlformat under the hood to generate the code. This has the inconvenient of enforcing the use of OCamlformat before using them as they will reformat the entire codebase.

The last two tools use Merlin under the hood to locate uses of Lwt functions in the source files. This works by reading Merlin’s ocaml-index files built by Dune to extract the location in the source files that we need.

Currently, it only supports Eio as a backend but can easily be adapted to other direct-style concurrency libraries.

This work was made possible thanks to the support of the NGI Zero Core fund through the Nlnet foundation, and is perfomed by Tarides.

11 Likes
2

This is super nice.

I suspect though that Claude Code might be able to perform similar transformation. I’ve been surprised recently at how good it is also with OCaml.