Wouldn’t it be nice if every OCaml package had examples as well as documentation? As part of a pilot programme funded by the OCaml Software Foundation, I’ve been looking into the feasibility of such an idea.
What do we mean by examples, and what distinguishes them from documentation and from tests?
What an example is
Examples are independent of the library they explain. They do not require the source of the library, or any built artefacts.
Examples are self-contained. They require only OCaml, a build system, and the library in question to be installed.
Examples are easy to build. They are built in a single command, and do not depend on environment.
Examples are easy to edit and play with. They are of a reasonable size, split into chunks, and are commented liberally.
Examples use standard techniques. Both in how the library is used, and in how the OCaml code is written.
Examples are open licensed. Users should be able to copy & paste code from the examples without care.
What an example is not
Examples are not tests. Unlike tests, examples do not care about code coverage, cannot be automatically generated, and need not necessarily be tightly integrated into the source repository.
Examples are not in the API documentation. Examples need to be buildable, and separate from the API documentation. This is not to say that they might then not be automatically imported into the API documentation one day.
Examples need not be comprehensive. Better a small example than no example at all. So long as the basics of an API are introduced, the cliff is climbed and the API documentation should thereafter suffice.
Pilot project plan
The plan was to build small examples for about twenty packages, put together a place for them to live, and then try to upstream them. The examples’ home, prior to upstreaming, is the OCaml Nursery:
Most of these little examples have been submitted to upstream - you may have noticed the pull requests on your repositories - with varying degrees of interest / success.
Opinions requested, please!
Are you interested in adding examples for your package or someone else’s package? To the nursery or to upstream? What do you think of the definition of example I gave above? Do you think examples should sit in a separate space like the nursery or be upstreamed or both? Opinions requested on all those topics, please!
I’d add: it shows the full “lifecycle” for using the library. E.g. you may need to perform some initial setup, some final cleanup, and to compose various library functions together in order to do something useful. If the library has a vast and flexible collection of functions (which in itself is desirable!), you may initially be overwhelmed in how to get started.
An example for this is Cmdliner: although the documentation for the individual functions is detailed, without the “Basics” and “Examples” section, that shows how to put it all together, it’d be a lot more difficult to get started.
They should be easily discoverable from a package’s ocaml.org page https://ocaml.org/p//latest/doc/index.html
Whether that is a link on the Docs page, another tab next to Docs, or tightly woven into the module/package documentation itself may depend on the package (and whether it makes sense to upstream the examples).
I agree that keeping them buildable is useful (otherwise they get stale, fail to build upon breaking API changes, leaving users confused), however that doesn’t exclude having them in the API docs. There are ways to automatically insert or extract snippets from docs with mdx.
I’d say that is is not required to have them in the API docs.
In fact I think most packages’ API docs would benefit from having a very minimal example that just shows how to get started with the package. It doesn’t have to cover every feature the package has.
Of course it is also fine to have them as separate files, although perhaps odoc should eventually support this, and automatically generate a page containing the examples, where function calls are clickable and take you to the documentation of the function. I think some of this exists already (it supports rendering docs for implementations too?).
There could be some metadata in the dune / opam package that tells the tooling where to find the examples.
If reasonably sized I think that they really should be there, that is not in the API documentation but in the package documentation that gets rendered on ocaml.org.
Or extracted from it. That’s already a reality. Since odoc 3.0.0, you can extract code from .mld and .cmti files with odoc extract-code.
Also, the PHP’s comment on document pages is spiritually similar. (e.g. PHP: String - Manual). Various other documentation site do the same now. (e.g. Godot, vulkan-tutorial…)
“Examples are not tests.” And yet tests are examples. I’d be happy with a complete suite of documented tests. In my humble experience that’s the way I usually learn how to use an API, by looking at the tests.
One thing I think people forget about, is the toplevel. I’ve started writing more tests in the toplevel, using mdx as the test-runner. It’s useful because it forces you to know (and show) how to get your package working usefully in the toplevel, and that’s so, so, so useful for debugging, and also for exploratory programming.
I honestly think mdx is just amazingly useful, and more people should use it.
One thing I’ve long wanted to do is to build js_of_ocaml toplevels for every package to have alongside the docs. This is quite feasible, and I had a brief go some years ago.
I’m betting that a lot of packages could have example code that worked well in the browser, though obviously many packages couldn’t. However, we could certainly provide a decent merlin-powered editor experience right there next to the docs on ocaml.org.
My more recent musings on this subject look more like Jupyter-style notebooks - I’ve got some examples on my blog. The code that powers that has been written with the use-case of ocaml.org in mind, so adapting it to run in the docs-ci should be relatively straightforward.