[ANN] New release of Conduit

Community
1

Conduit 3.0.0

Hello everyone,

We’re glad to announce the new release of conduit, a framework that allows to abstract over transfer protocols. One of its main advantages is allowing the implemententation of free-dependencies protocols.

Introduction

There are several ways to abstract over an implementation in OCaml. However, those solutions are often lost deep in the stack of protocols and allowing the user to choose the implementations of the sub-procotols implies growing complexity as we move up through the stack. (For example, allowing to abstract over the implementation of the TLS protocol from the implementation of the HTTP protocol)

One of those solutions, the functors, can rapidly become a hellish nightmare for the end-user. This is especially true in the case of MirageOS, which literally wants to abstract over everything!

This is why Conduit was implemented: it aims to provide to the user a cleaner abstraction mechanism which would allow the protocol developers to get rid of most of the responsibilities concerning the choice of sub-protocols (Like which TLS implementation use between OpenSSL or our great ocaml-tls library), while giving the end-users an easy way to compose the protocols of their choice and inject them in the stack via conduit.

Usage of Conduit

Such a framework allows us to separate the logic of a protocol from underlying implementation needed to communicate with a peer. The distribution of Conduit comes with a simple tutorial which explains step by step how to implement a ping-pong client & server and, most importantly, how to upgrade them with TLS.

With Conduit, we ensure the compatibility with MirageOS (and specially mirage-tcpip) while being useful for others. Of course, Conduit is not mandatory to ensure this compatibility, but it helps us for higher libraries such as ocaml-git/Irmin or Cohttp.

Specific improvements

Abstract and destruct it!

The most requested feature on the new version of Conduit is the ability to destruct the [Conduit.flow][conduit-flow]. The ability to abstract the protocol comes with the abstract type Conduit.flow. The new version permits to destruct it to a well-known value (such as an UNIX socket):

let handler flow = match flow with
  | Conduit_lwt.TCP.T (Value file_descr) ->
    let peer = Lwt_unix.getpeername file_descr in
    ...
  | flow -> ... (* other kind of protocol *)
  
let run =
  Cohttp_lwt_unix.serve ~handler
    { sockaddr= Unix.inet_addr_loopback }

The dispatch of the protocol

The second most interesting feature of Conduit is the full control over the dispatch between protocols by the end-user. From a concrete information such as an Uri.t, the end-user is able to describe how Conduit should choose the protocol (and with which value it should try to initiate the connection):

let my_tls_config = Tls.Config.client ...

let connect uri =
  let edn = Conduit.Endpoint.of_string
    (Uri.host_with_default ~default:"localhost" uri) in
  let resolvers = match Uri.scheme uri with
    | Some "https" ->
      let port = Option.value ~default:443 (Uri.port uri) in
      Conduit_lwt.add
        Conduit_lwt_tls.TCP.protocol
        (Conduit_lwt_tls.TCP.resolve ~port ~config:my_tls_config)
        Conduit.empty
    | Some "http" | None ->
      let port = Option.value ~default:80 (Uri.port uri) in
      Conduit_lwt.add
        Conduit_lwt.TCP.protocol
        (Conduit_lwt.TCP.resolve ~port)
        Conduit.empty in
  Conduit_lwt.resolve ~resolvers edn >>= fun flow ->
  ...

An explicit way to launch a server

Conduit comes with a new API for the server-side, where everything becomes explicit: no dispatch, no hidden choice. It proposes now a simple function to start the usual server loop:

let run handler =
  Conduit_lwt.serve ~handler
    Conduit_lwt.TCP.service
    { Conduit_lwt.TCP.sockaddr= Unix.(ADDR_INET (inet_addr_loopback, 8080)
    ; capacity= 40 }

Reverse-dependencies

Conduit is used by many libraries (~150 packages) and we spend 2 months to track this breaking-change. Currently, it’s mostly about Cohttp and Irmin and both have a PR according the new version of Conduit. These packages will be released as soon as we can with the new version of Conduit.

Conclusion

Conduit is a piece required by many libraries but nobody really uses it. This new version wants to replace and redefine more concretely what Conduit is. The update is huge for us but small for people where we tried to keep the same global idea of the abstraction.

I would like to thank many people (MirageOS core team, Cohttp peoples, some not so famous guys of the Reason/OCaml eco-system) who followed us on this deep development (and tried and iterated on our version). It does not change too much our world, but it paves the way for a better MirageOS/OCaml eco-system.

As a french guy, I just would like to say: Conduit est mort, Vive Conduit!

19 Likes