Upcoming Caqti Release - TLS, Packaging, and minor breakage

Community
, , ,
1

The next Caqti release is almost ready if I proceed as planned, though I would like to lift plans in advance in case anyone has feedback.

While Caqti 2 brought support for MirageOS (using PGX), it is missing TLS, which limits its usability in practise. Patching that hole will be the main delivery for the upcoming release. This involved a revision of the network abstraction and better performance under EIO.

Some more minor changes can be seen in the preliminary change log.

Minor Breaking Changes

My plan is the make the upcoming one a minor release, but there will be two breaking changes, so we could discuss whether a minor release is reasonable:

  • There will be a new optional ?config parameter for all functions which connect to a database or create a connection pool, which can break code which makes higher-order calls or aliases those parts of the signatures.

  • The Caqti_query.t type will have one additional constructor (V), which breaks any code which alias the type along with its constructors.

Packaging and TLS

TL;DR: This is mostly a request for comments about packaging.

The Caqti project currently has 11 packages in the official OPAM repository. With the current plan, there will be an additional 4 TLS packages:

packages description assessment
caqti core functionality required
caqti-dynload dynamic linking could be merged into caqti
caqti-driver-mariadb, caqti-driver-pgx, caqti-driver-postgresql, caqti-driver-sqlite required
caqti-async, caqti-eio, caqti-lwt, caqti-mirage required
caqti-type-calendar convenience could be retired
caqti-tls core TLS package see below
caqti-tls-async, caqti-tls-eio, caqti-tls-lwt specialized TLS engines see below

(For caqti-mirage, TLS is included, since the tls library is the only option to encrypt PostgreSQL communication for unikernels, if not implemented at the network level, AFAIK.)

As indicated above there could be options to eliminate two existing packages, but I’ll leave that for later. The alternatives I’ve considered for the TLS implementation and packaging is:

  1. The implemented choice is to use the tls-async, tls-eio, tls-lwt, and tls-mirage instantiations of the TLS engine, thus adding one package per flavour.

  2. TLS support for Async, EIO, and LWT could instead be merged as sublibraries of caqti-async, caqti-eio, and caqti-lwt, and caqti-tls could be merged into caqti. The main reason for not doing this, is the C bindings for MariaDB and PostgreSQL already have TLS built-in, so in most cases the tls dependency would be redundant. The dependencies could be made optional, though I have the impression many consider optional dependencies harmful.

  3. The Caqti network abstractions could use the pure TLS engine directly, thus requiring only one extra package, caqti-tls. This would be a elegant solution, except that it will not work with the ssl library should we want to support it, since it is tied to the network stack.

While switching to the second option is simple, switching to the third option would require more work on my part, delaying the release, but I could be convinced if we feel strongly for the end result.

The caqti-tls package is (currently) only needed to define a shared configuration key which depends on the tls library.

4 Likes
2

Am I right in thinking that the non-C (libpq) drivers for PostgreSQL don’t actually support the default PostgreSQL password format (scram) yet? In which case I don’t think the non-C options are a serious consideration anyway. I’m not trying to minimise the effort required to implement SCRAM (I had a quick look and knew it was more effort than I was interested in) but it’s been implemented for several years now.

(Disclaimer: I’m a long-time postgres user and don’t really use/care about other platforms much)

3

You are right about no SCRAM support. I don’t think it’s a deal-breaker if the CA is handled properly, but I also suggest to keep using the libpq library and bindings where they are available, due to their maturity.

4

Thanks for the work. :slight_smile: Is there any plan to support miou in caqti ? We are writing a web framework based on miou and httpcats and we really would like to be able to use caqti inside it (as it was quite pleasant to use from our experience).

1 Like
5

The short answer is that I don’t have plans for miou, and I’m not familiar with it.

I think the Async/Lwt divide is already unfortunate, though there may be good reasons for it. I hope that OCaml 5 will unify and modularise OS and scheduling abstractions, rather than further fragment, though this will require some coordination between implementers.

6

My plan is to proceed with the release, but omit the caqti-tls-async, caqti-tls-eio, caqti-tls-lwt packages from the OPAM release for now. This means Caqti will have TLS-support under MirageOS (since it’s included in caqti-mirage) and for other platforms caqti-driver-postgresql is available with OpenSSL support, as always. (You can still pin the omitted packages to the Git repo, but I am not sure yet whether they will be replaced by direct use of Tls.Engine.)

There will also be a new sublibrary caqti.plugin which I think will eventually replace the caqti-dynload package, though I’m not deprecating the latter yet. (caqti.plugin uses dune-site.plugin while caqti-dynload uses findlib.)

7

Caqti 2.1.1 has now been released to OPAM. The following packages were included:

Package 2.1.1
caqti yes
caqti-async yes
caqti-eio yes
caqti-lwt yes
caqti-mirage yes
caqti-tls-async no, omitted for now
caqti-tls-eio no, omitted for now
caqti-tls-lwt no, omitted for now
caqti-tls yes
caqti-driver-mariadb yes
caqti-driver-pgx yes
caqti-driver-postgresql yes
caqti-driver-sqlite3 yes
caqti-dynload no, unchanged
caqti-type-calendar no, unchanged

Details can be found in the release notes for version 2.1.0 since 2.1.1 include build and packaging fixes only.

1 Like