ANN: Major releases of Cohttp, Conduit, Dns, Tcpip

This is a headsup about some major package reorganisation for some MirageOS packages that are used by other projects as well. The Cohttp, Conduit, Dns, TCP/IP libraries have all been repackaged to do three things:

What has improved

Removed optional dependencies from the opam packaging that would activate the compilation of features. While this was a good idea in the early days of opam, it now leads to having to specify a long set of packages to make a particular backend available (e.g. for dns.lwt). In MirageOS, we are moving towards explicit opam packages for extra functionality, and deprecating the use of depopts where possible.

Ported to Jbuilder to speed up builds, and use modern OCaml features. This means that the use of pack is gone in TCP/IP in favour of module aliases, and that build times have sped up by roughly 4-5x for many of the libraries and applications using them. They are also more easily embeddable in third-party projects using the Jbuilder workspace support.

Camlp4 entirely eliminated, ppx_driver now used: the move away from camlp4 is now complete (we even have a no-camlp4 package to warn if it returns to our dependency chain). The modern use of ppx enforces the use of ocaml migrate-parsetree so that they can be built in the latest releases of OCaml more easily as well.

What is incompatible

The primary change you will observe is that an ocamlfind package that was formerly in the subpackage format (e.g. cohttp.lwt) is now using a dashed format and is a separate opam package (e.g. cohttp-lwt). This requires you to port your libraries to adjust the build to:

  • set a minimum opam constraint on the version of library to >="0.99" to exclude older releases.
  • add an opam dependency on the feature required (e.g. cohttp-lwt-unix)
  • adjust the ocamlfind dependencies to replace dots with dashes (e.g. cohttp.lwt -> cohttp-lwt).
  • release a new version of the package to the stable opam-repository

Here is the table of old/new opam packages and old/new ocamlfind packages, along with the opam version constraint for the new opam package.

old opam packages new opam package old ocamlfind new ocamlfind
conduit conduit >=1.0.0 conduit conduit
conduit lwt lwt_ssl conduit-lwt >=1.0.0 conduit-lwt conduit-lwt
conduit mirage-types-lwt mirage-flow-lwt dns [and more...] mirage-conduit >=3.0.0 conduit.mirage mirage-conduit
conduit async async_ssl conduit-async >=1.0.0 conduit.async conduit-async
old opam packages new opam package old ocamlfind new ocamlfind
cohttp cohttp >=1.0.0 cohttp cohttp
cohttp lwt cohttp-lwt >=1.0.0 cohttp.lwt-core cohttp-lwt
cohttp lwt base-unix cohttp-lwt-unix >=1.0.0 cohttp.lwt cohttp-lwt-unix
cohttp js_of_ocaml cohttp-lwt-jsoo >=1.0.0 cohttp.js cohttp-lwt-jsoo
cohttp async cohttp-async >=1.0.0 cohttp.async cohttp-async
cohttp mirage-http mirage-http >=3.2.0 cohttp.mirage mirage-http
tcpip tcpip tcpip >=3.2.0 tcpip
old opam packages new opam package old ocamlfind new ocamlfind
dns dns >=1.0.0 dns dns
dns lwt mirage-profile cmdliner dns-lwt >=1.0.0 dns.lwt-core dns-lwt
dns lwt mirage-profile cmdliner dns-lwt-unix >=1.0.0 dns.lwt dns-lwt-unix
dns mirage-time-lwt mirage-stack-lwt mirage-kv-lwt lwt duration mirage-profile mirage-dns >=3.0.0 dns.mirage mirage-dns
dns async dns-async >=1.0.0 dns.async dns-async

Where the work is

There have also been around 30 or 40 upper bounds added to existing users of Cohttp. We would really appreciate your help with updating your own code to the newer version so that users aren’t held back by these constraints. Please reply to this post with any questions or difficulties you have with moving to the newest version of Cohttp.

We also realise how confusing the old package structure was, as it grew very fast out of an initially simple set of clean-slate network protocol implementations for the MirageOS project. We are now steadily working on restructuring the repositories to make them easier to use and integrate your feature additions, so thanks for all the patience so far :slight_smile:

Many thanks to @rgrinberg @samoht and @djs55 who did a vast amount of work to make this particular restructuring possible, and the many other contributors who are noted in the changelogs of the libraries.

8 Likes

This is great – thanks for writing it up. I wonder if we should extend the Using jbuilder and topkg with Mirage page to include conventions like avoiding camlp4 and using ocaml-migrate-parsetree? Or perhaps we could create an “anatomy of a Mirage package” page using one of these new packages as an example?

2 Likes

Ah yes, that’d be great! I’ll send a PR to the Mirage-www repository with the latest learnings from the Jbuilder porting.

Headsup: after a few iterations on the releases, the hopefully final pushes are in https://github.com/ocaml/opam-repository/pull/9824 and will be merged today or tomorrow if the tests all look green. Please let me know if you have any outstanding issues or questions.

1 Like