# Best practices and design patterns for supporting concurrent IO in libraries
**URL:** https://discuss.ocaml.org/t/best-practices-and-design-patterns-for-supporting-concurrent-io-in-libraries/15001
**Category:** Learning
**Tags:** lwt, io, concurrency, eio, async
**Created:** [July 20, 2024, 12:18pm UTC](https://discuss.ocaml.org/t/best-practices-and-design-patterns-for-supporting-concurrent-io-in-libraries/15001 "2024-07-20T12:18:53Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@zoj613](https://discuss.ocaml.org/u/zoj613)
#### Post date: [July 20, 2024, 12:18pm UTC](https://discuss.ocaml.org/t/best-practices-and-design-patterns-for-supporting-concurrent-io-in-libraries/15001/1 "2024-07-20T12:18:53Z")
I am interested in learning about the best practices and recommended design patterns in 2024 for supporting concurrent IO in libraries. I noticed that several libraries release two packages, one using synchorious IO and another with a `-lwt` or `-async` suffix meant to be used in Lwt or Async workflows. An example is the [aws-s3 ([GitHub - andersfugmann/aws-s3: Ocaml library to access Amazon S3](https://github.com/andersfugmann/aws-s3)) library. So:
1. Why is that?
2. Is this considered best practice?
3. How does one avoid code duplication if creating modules for sync and async modules?
4. Are there alternatives to this style?
5. Are there existing tutorials on this topic that I can learn from?
Any amount of insight is welcome since I have no idea where to start.
---
### Author:  [@nojb](https://discuss.ocaml.org/u/nojb)
#### Post date: [July 20, 2024, 1:37pm UTC](https://discuss.ocaml.org/t/best-practices-and-design-patterns-for-supporting-concurrent-io-in-libraries/15001/2 "2024-07-20T13:37:06Z")
In short, `lwt` and `async` are two (incompatible) libraries exposing a monadic API for concurrency. Together, they are the de-facto standard for concurrent programming (at least before the arrival of OCaml 5, see below).
> [@zoj613](#):
>
> Are there existing tutorials on this topic that I can learn from?
- An `async` overview: [Concurrent Programming with Async - Real World OCaml](https://dev.realworldocaml.org/concurrent-programming.html) (see also the [`lwt` translation](https://github.com/dkim/rwo-lwt/))
- The Mirage `lwt` tutorial [Getting Started with Lwt threads | MirageOS](https://mirage.io/docs/tutorial-lwt)
> [@zoj613](#):
>
> Are there alternatives to this style?
With OCaml 5 it became possible to implement concurrency in “direct style” (ie without using monads or callbacks). Accordingly, there are now concurrency libraries which follow this style:
- [GitHub - ocaml-multicore/eio: Effects-based direct-style IO for multicore OCaml](https://github.com/ocaml-multicore/eio)
- [GitHub - robur-coop/miou: A simple scheduler for OCaml 5](https://github.com/robur-coop/miou)
(The documentation of both libraries is excellent. Also, this list is not exhaustive.)
Cheers,
Nicolas
---
### Author:  [@zoj613](https://discuss.ocaml.org/u/zoj613)
#### Post date: [July 20, 2024, 1:53pm UTC](https://discuss.ocaml.org/t/best-practices-and-design-patterns-for-supporting-concurrent-io-in-libraries/15001/3 "2024-07-20T13:53:04Z")
Thanks a lot for this reply, I wasn’t aware of the existence of miou. Are there any recommended best practices to follow when implementing new libraries, given all the options stated above? Is there any benefit with trying to support multiple concurrency libraries or is it better to just stick with one implementation and forget the rest?
---
### Author:  [@nojb](https://discuss.ocaml.org/u/nojb)
#### Post date: [July 20, 2024, 2:10pm UTC](https://discuss.ocaml.org/t/best-practices-and-design-patterns-for-supporting-concurrent-io-in-libraries/15001/4 "2024-07-20T14:10:56Z")
> [@zoj613](#):
>
> Are there any recommended best practices to follow when implementing new libraries, given all the options stated above?
I would say that any recommendation would depend on what you are trying to achieve.
In the past, people have abstracted over the Lwt/Async/eio/etc difference by functorizing their libraries, but this complicates the implementation, and can get tricky if you need to go beyond the “formal” properties of each concurrency framework (since you need to come up with a signature common to all concurrency frameworks). An example of this approach is [GitHub - mirage/ocaml-cohttp: An OCaml library for HTTP clients and servers using Lwt or Async](https://github.com/mirage/ocaml-cohttp).
The maximally flexible approach is to write your library in so that it does not do any I/O by itself, and instead asks the library user to do it on its behalf. Then, by construction, the library will be independent of any specific concurrency framework. The downside is that this often requires more work to design & implement and to develop the connectors to specific concurrency networks. An example of this approach is [GitHub - mirleft/ocaml-tls: TLS in pure OCaml](https://github.com/mirleft/ocaml-tls).
Cheers,
Nicolas
---
### Author:  [@njb](https://discuss.ocaml.org/u/njb)
#### Post date: [July 20, 2024, 4:46pm UTC](https://discuss.ocaml.org/t/best-practices-and-design-patterns-for-supporting-concurrent-io-in-libraries/15001/5 "2024-07-20T16:46:49Z")
Another option would be to investigate [picos](https://github.com/ocaml-multicore/picos)
---
### Author:  [@yawaramin](https://discuss.ocaml.org/u/yawaramin)
#### Post date: [July 20, 2024, 6:34pm UTC](https://discuss.ocaml.org/t/best-practices-and-design-patterns-for-supporting-concurrent-io-in-libraries/15001/6 "2024-07-20T18:34:34Z")
See also [Interaction of effect-based fibers with caml\_release\_runtime\_system?](https://discuss.ocaml.org/t/interaction-of-effect-based-fibers-with-caml-release-runtime-system/14807)
---
### Author:  [@zoj613](https://discuss.ocaml.org/u/zoj613)
#### Post date: [September 9, 2024, 3:34am UTC](https://discuss.ocaml.org/t/best-practices-and-design-patterns-for-supporting-concurrent-io-in-libraries/15001/7 "2024-09-09T03:34:39Z")
Functorizing turned out to be the most appropriate approach for my usecase since the I/O interface is uniform. See [GitHub - zoj613/zarr-ml: An implementation of the Zarr storage format specification for chunked & compressed multidimensional arrays.](https://github.com/zoj613/zarr-ml) or [this post announcing the library](https://discuss.ocaml.org/t/ann-zarr-v0-1-0/15259)