Advice for combining multiple monads

[raphael-proust]

In the Octez project we have face a similar issue: we use both Lwt and Result pervasively.

TL;DR:

• We use dedicated syntax module for each monad (Lwt, result, Lwt+result) which export binding operators (let*, let+, return, etc.).
• We have a Stdlib supplement exposing monadic variants of all the Stdlib’s traversal functions (like Lwt_list but much more extensive)

Some time ago

Until not so long ago we used infix operators for bind. And we would mix different operators depending on what sub-monad a specific expression would be in. So we would have >>=? for Lwt+result, >>? for result-only, and >>= for Lwt-only. Plus we had a dedicated operator for when you use a result-only expression in an Lwt+result context: >>?=. We don’t need the other specialised binder because Lwt-only and Lwt+result mix quite well: (_, _) result Lwt.t is just a specific case of _ Lwt.t so >>= just works.

We also had a very flat namespace where all the operators as well as some helper functions (e.g., we had error : 'err -> ('a, 'err) result and fail : 'err -> ('a, 'err) result Lwt.t) were exported by an Error_monad module which was opened everywhere (using -open as a build flag).

Now

We have changed a few things.

• We use binding operators (let* and such) which are exported by dedicated syntax modules.
  • Lwt_syntax is essentially our local version of Lwt.Syntax
  • Result_syntax is for the result-monad
  • Lwt_result_syntax is for the combination monad
  • (We also have Option_syntax and Lwt_option_syntax, but we don’t use those as often.)
• Our Lwt+result monad syntax includes dedicated binding operators for Lwt-only and result-only expressions
  • let*! is for Lwt-only (mnemonic: you must(!) wait for the promise to resolve)
  • let*? is for result-only (mnemonic: there may(?) be a value there or maybe an error)
  • Same for the Lwt_option_monad syntax module
• Internally, we recommend to only open those locally. So you start your function by let open Lwt_result_syntax (or whichever monad you are actually using there).
• We have an extensive Stdlib supplement with many of the monadic variants of the provided traversors baked in. E.g., List.map_s is the Lwt-only equivalent of List.map, List.map_e is the result-only, and List.map_es is the Lwt+result.
  • We also have concurrent variants (e.g., List.map_p and List.map_ep) for the traversals where it makes sense.
  • We have a consistent semantic regarding error management across all those traversors.
  • We have several modules: List, Map, Set, Seq, etc.

One thing I didn’t mention is that we actually have a specialised result called tzresult: 'a tzresult = ('a, tzrerror trace) result where tzerror is a custom error type and trace is a data-strucutre holding several errors. Traces (of errors) allows us to combine errors in different ways. The first way is you can add an error to a trace to add higher-level context about the lower-level error. The second way is for errors that happen concurrently (e.g., you evaluate concurrently several Lwt+result expressions and more than one fails). This ability to combine concurrent errors gives us a semantic for and* in the Lwt+result syntax module.

There are several downsides to our current approach, but all in all it works well enough.

• Even for functions of modest size, it’s not always immediately visible which monad you are located in. This is even worse when you are viewing just a chunk of a diff.
• Some parts of the Stdlib don’t lend themselves to our monadification (e.g., Seq) or require a bit of boilerplate because they don’t expose internal representations (e.g., Map’s monadic traversors are largely implemented in terms of Seq).
• The Stdlib supplement is a lot of code with a lot of tests and a lot of comments. It’s just a large volume of low-complexity code to deal with.

There are also some very pleasant upsides:

• The separate monad syntax modules encourages you to write each function with the smallest monad that it needs. This in turns
  • encourages you to split your function into smaller components which are more easily testable,
  • makes the type of functions informative: an Lwt.t function is very likely to actually be doing I/O at some point down the line, and a result function is very likely to actually return Error in some situations.
• The Stdlib supplement makes it quite easy to adapt code for one monad to another.

For more complete information, you can check the Error-monad tutorial.

Later

There are a few things we want to change. Albeit we have no urgent need to do so.

• Remove some legacy helper functions that are still hanging around.
• Organise the namespace better.
• Generate a lot of the code and doc of Lwtreslib automatically.
• Improve traces (currently, for mostly historical reasons, we use list).