Experience report switching to jbuilder and topkg

Ecosystem
, , , ,
1

Having just updated all my projects to jbuilder and topkg, I’m happy to report that this is a very much worthwhile step to take and encourage everyone to consider doing the same.

My previous approach employed Oasis and ocamlbuild. This mostly worked fine, but the development process, e.g. ease of specification and configuration, compilation speed, complexity of the release process, etc., clearly had room for improvement.

Neither topkg nor jbuilder initially seemed compelling enough to warrant rewriting my build and packaging process, which I had already mostly automated by shell scripts. But their integration with topkg-jbuilder provides such strong synergistic effects, it would be hard to argue for any other approach right now.

Among the biggest advantages:

  • Considerably improved compilation / package installation speed
  • Much lower effort specifying the build process and packages due to elimination of redundancies
  • Much shorter and less error-prone release process. Releasing a new package (including API documentation) and upload to GitHub and OPAM, now typically takes me around 2 minutes.

I usually take great care to make my packages as uniform as possible in the way they are structured for building and packaging and have liberally copied the best ideas from other users. Feel free to do the same by using my projects as a template for your own.

Here are the steps I usually take when making a new release:

  1. Make sure everything is committed and pushed to GitHub, including the definitive version of the CHANGES.md file, from which the version information of the release is derived automatically.
  2. topkg tag -s to tag and GPG sign the release
  3. git push --tags to push the new tag to GitHub
  4. CONDUIT_TLS=native topkg bistro
    • See this opam-publish issue whether CONDUIT_TLS=native is still necessary.
    • The above topkg bistro command is equivalent to:
      1. topkg distrib builds the distribution
      2. topkg publish publishes it and its API documentation on GitHub by combining the following sub-commands:
        1. topkg publish distrib
        2. topkg publish doc
      3. topkg opam pkg creates an OPAM package
      4. topkg opam submit submits it to the public repository, opening a browser window with the pull request
  5. make clean to remove the _build directory created by the steps above.
  6. git checkout gh-pages for updating the README.md used by GitHub Pages (more fancy looking project page).
  7. git pull in case new API documentation was submitted earlier.
  8. sync_github_readme.sh - this script contains:

    #!/bin/bash

    git show master:README.md > index.md
    git commit -m "Synced master README.md to index.md" index.md

  9. git push in case the last step introduced changes.
  10. git checkout master to switch back to the master branch.

These ten steps are obviously trivial and could in principle be automated further. But I generally prefer some caution before e.g. committing changes or pushing things to remote repositories. Please feel free to suggest further improvements to the packaging approach and release process!

Thanks to Daniel Bünzli and Jeremie Dimino and their contributors for Topkg and Jbuilder!

14 Likes
2

I’m noticing that you’re using a gh-pages branch for your Github pages. Since about a year ago, Github lets us put the root of site under /docs in the master branch, which I find it much easier.

I just wanted to share the info in case you didn’t know.

2 Likes
3

configurator deserves a mention as well. It’s not really jbuilder specific, but it is utterly trivial to use with jbuilder and it sets the bar higher for packaging C bindings. I know you’re aware of it as you’ve switched over lacaml to it for example. Would be nice if everyone else switched

4

Topkg does that automatically for you, it’s as simple as topkg distrib && topkg publish doc. Personally I wouldn’t store generated documentation in my master branch.

3 Likes
5

This is only tangentially related, but as someone that sometime looks at pull requests against the opam-repository, I find it very convenient that the pull request (PR) comes with a changelog, or at least some clear information of what has changed / justifies the release. Having a feeling for the content of the release helps make decision about merging the PR, can help understand continuous integration failures, and in generate makes opam-repository maintenance more pleasant.

So, thanks for having this as a part of your process, and hopefully the people reading this topic will think of doing the same (automatically or manually).

3 Likes
6

I agree with Daniel that there are downsides to storing documentation in the master repository. Though one could probably exclude it and thus prevent it from increasing the package size, I’d rather keep it in a separate branch. topkg will happily push the updated API-documentation to your gh-pages branch on GitHub anyway without requiring any user interaction. I only update the index.md file in the gh-pages branch in a short separate step of my release cycle. I guess that, too, could be added as an option to topkg.

7

It’s indeed a nice feature that topkg prepares the OPAM pull request with a (well-formatted) changelog entry.

8

Especially Lacaml configuration and file autogeneration was a considerably more messy issue with oasis and ocamlbuild before I switched to jbuilder and configurator. It often required doing a make clean (and then ridiculous rebuild times) to make sure that all needed dependencies were properly built. This has bitten me on multiple occasions during development when I wondered why my seemingly correct (but incorrectly built) code wasn’t working.

Note that Pcre, Sqlite3, Postgresql, and Gsl have also been switched to the configurator. I hope it will eventually have even more functionality for discovering system features and taking into account user configuration options.

2 Likes