[ANN] Caqti 2.2.4 Release and Plans

Community
,
1

The Release

I am pleased to announce the release of Caqti 2.2.4, after stumbling through a few minor releases starting at 2.2.0. These are the combined release notes since the previous OPAM release, omitting intermediate regressions:

Improvements

  • The sqlite3 driver now supports the refined error causes (Caqti_error.cause) for integrity constraint violations.
  • There is now experimental support for Miou (#117 by Calascibetta Romain).
  • Make the pool implementation shared-memory safe.
  • The new library caqti.template provides a preview of a interface for creating and working with request templates, with a few new features and, I think, a tidier design. This is not yet suitable for production code, since it will change before the final version. Feedback is welcome.

Fixes

  • Fixed a memory leak in the fall-back implementation of the populate connection method which affects all except the postgresql drivers.

Deprecations

  • Caqti_request.query_id is deprecated and will be removed.
  • Constructors of Caqti_type.t are now fully private and will be moved away and likely defined differently in the next major release.

Dependency Updates

  • Prepare for upcoming mirage (#124 by Hannes Mehnert).

Upcoming Work on caqti.template

As mentioned, this library is experimental for now, yet it contains the real implementation of request templates, while the stable API is a backwards compatible wrapper. I already have plans for revising the new API both due to hard requirements and feedback about usability and preferences, esp. for the most commonly used part of the API.

Dynamic Prepare Policy and Parametric Types

One thing I am excited about with the caqti.template library is the support for prepared queries for dynamically generated request templates.

To motivate this, note that each prepared query retains resources on both the server side and locally, both associated with an open database connection, which can be long-lived. So, if the application generates queries based on e.g. search expressions like (author:Plato or author:Socrates) and topic:epistemology, which cannot be prepared statically due to the boolean algebra over search terms, then users of the current API must use one-shot (non-prepared) queries to avoid unbounded retention of resources over time. The new API provides a so-called dynamic prepare policy, which uses an LRU cache of prepared queries internally to limit the resource usage while providing a heuristic for re-using common prepared queries.

There is, however, a missing piece in order for this to be practically efficient. Type descriptors like Caqti_type.t2, Caqti_type.t3, etc. which represent parametrically polymorphic types, are currently generative with respect to the equality used by the LRU cache, meaning that the type expression would need to be lifted out of the function which generates the request template in order to avoid consistent cache-misses.

My plan is first to make a major release which moves the concrete type representation away from the public modules, and then revise it to properly encode parametric types. Apart from changing the constructors, this involves adding an extra phantom type parameter, but the parameter will be universally qualified in the type exposed to the public API, so I expect to retain backwards compatibility for typical usage.

The Main Request Template API and Bring-Your-Own-Paint

For most use cases, it is sufficient to be able to construct request templates, which consists of type descriptors and a possibly dialect-dependent query template. Everything needed for this is bundled into the module Caqti_template.Create. I may have developed a bit colourblindness after walking around this bikeshed, so let me show you how the current stable API looks like, how the current iteration of the new API looks like.

Here is a request template using the stable API:

let select_owner =
  let open Caqti_request.Infix in
  let open Caqti_type.Std in
  (string ->? string)
  "SELECT owner FROM bikereg WHERE frameno = ?"

In the current iteration of the new API, this looks like:

let select_owner =
  let open Caqti_template.Create in
  static T.(string -->? string)
  "SELECT owner FROM bikereg WHERE frameno = ?"

What changed?

  • There is now only a single open. That can only be good.
  • The type descriptors have been moved under a T sub-module. This isn’t strictly necessary, but I though it was tidier, esp. since there are clashing names (like int) under a parallel Q sub-module which is used for dynamically generated query templates.
  • The arrow was previously the main function, while in the new API it only constructs the type and multiplicity representation (Caqti_template.Request_type.t).
  • Instead, the main function now indicates the policy for whether to use a prepared query and, if so, the life-time of the request template. The options are direct, static, and dynamic (as explained above).

The latter adds verbosity, but I think it is good to be explicit about static life-time, since using it for generated request templates would typically lead to a resource leakage.

It’s getting late, so I will not write about dynamic and dialect-dependent request templates, but in the current revision, they are constructed with a “general” version of the above functions, direct_gen, static_gen, and dynamic_gen which takes a function receiving a dialect descriptor and returns a Caqti_template.Query.t instead of a string. The stable API used combinator operators for this.

Thanks to contributors and to OCaml Software Foundation for funding my work towards this and past releases.

4 Likes