I’ve reached a certain point in my personal project where I regret not having built unit tests right away. There’s too much state for me to easily test each path in the program and I want to know I’m not breaking basic things beyond the protection afforded to me by the typechecker.
Unfortunately I have very little experience in unit testing in OCaml. I’d love to hear about framework recommendations but more importantly, I’d love to see examples of how you set them up in your projects. I don’t mind placing unit tests in every file, but I’d prefer not to build them all the time if possible.
I’m a beginner when it comes to unit testing OCaml code too (have a little bit of experience using OUnit2 and Alcotest and preferred the latter), but I might have a small suggestion to make test-writing more a tiny bit more convenient.
I think the reason OUnit2 and Alcotest need to have a list of tests at the end (unlike JS/.NET and some others which auto-discover tests) is because OCaml lacks runtime reflection, but I might be wrong about that.
I think ppx_expect and ppx_inline_test are the easiest way to add unit tests. You can include them side-by-side with the code they test, and you can include tests of functions that are not exposed in the mli.
If you don’t want to compile the tests along with the code, extract the tests into a “test only” library that depends on your main library. The one downside of doing this is you can only test your public interfaces.
I think Alcotest is fine, but not as simple as I’d like. And it doesn’t do very much for me other than report the totals. So for my current experiment I’m just constructing my own list of (test_name, test_fun) and have a very small expect_eq function and test runner. The expect_eq takes a pair of (equal, pp) functions, just like Alcotest.check. Unlike Alcotest, I’m just using a tuple of functions instead of a first class module, which I think is overkill for this.
The largest amount of code (edit: in the test itself) is constructing the pp (formatter) that you need for every data type you’re comparing. This is true for Alcotest and ppx_expect, and I suspect for all unit testing. It is the Fmt module combinators, not the test framework, that you really need to leverage. That’s why I don’t think I really need to depend on a test framework.
I tried ppx_expect and liked it at first because it is so easy. But I quickly became disenchanted with having to manually print a string for every simple value I want to check. I prefer using checks that make use of the equal function for the data type and call the pp function for me.
I didn’t try ppx_inline_test because I don’t see a need for a ppx just to add a named test to a list. If I did want to explore the use of a ppx further, I would look at ppx_assert and specifically its test_eq to see if it made the formatter construction any easier. But I prefer not to depend on the Core/Base libraries if I don’t have to.
For my projects, I previously used Alcotest but switched to just using Dune’s built-in testing that works by diffing each test executable’s output with a .expected file. If you don’t mind using your pp function’s output as an equality check, then it works great. The end result is basically the same for me, but it’s simpler and I get access to nice features like dune promote.
I haven’t done a lot of real “unit” testing though, at least not by testing individual functions in each module. This may or may not be ideal for unit testing specifically, but IMO it’s nicer to use than Alcotest in general.
I agree that dune promote is really nice, with regular test executables like you’re using and with ppx_expect (works the same way there). If I had only a couple hours to write some tests, I would do this.
It “just works” with dune+inline_tests. But I expect you’re not a base/core user, so you’d have to derive your “show” functions some other way than using their sexp machinery.
Edit: and also bisect_ppx for test coverage is nice. Especially if you need some help in covering already written code. (Even works with dune cram tests as well.)
I use ounit for something that I’m guessing are not really unit tests, so you may say I abuse it But it works nicely together with dune test support and with the rather specific structure of my project (lexing / parsing).
Thanks for mentioning this. I hadn’t tried it until just now when I saw this. It is so easy to do and the reports are great. Nice work, @antron, thank you!
My general habit is not to consider logic drafted until I’ve achieved good test coverage with unit tests. It’s a habit I picked up writing code for robot taxis.
I used ounit to achieve pretty good test coverage in Orsetto, but I’ve since switched to alcotest for my current projects. I don’t have something to publish that illustrates how I’m using it, but it’s more or less the same (except I’m building with dune instead of omake now).
I find it easier to write test cases with alcotest than ounit and troubleshooting test case failures is much less unpleasant.
I use ounit a ton. I mean a ton. And it’s great. But sometimes, I want to test things in the toplevel – for instance, to know that something compiles, or that something does not compile (and if so, with what error). For PPX rewriters, that’s really useful. For that, the mdx project is … wow, sooooo great.
I am giving mdx a massive plug here, b/c I really love it. Bit-by-bit, I’m going to be doing all my examples in MDX, b/c they’ll also be basic unit-tests as a result. Again: just wonderful stuff.
At the whole-library level one thing you can do is use dune’s wrapped library mechanism. You can bypass it in your tests but it’s reasonable to expect your downstream users not to do so.
Here’s a library called blah. It’s made of a single module Foo.
$ cat src/foo.ml
let y = 12
let x = y + 1
$ cat src/blah.ml
module Foo : sig val x : int end = Foo
$ cat src/dune
(library
(name blah))
Here’s a test for this library which accesses the y value. It does so by by-passing the wrapped library and accessing the internal name Blah__Foo which contains two underscores and is not meant for external use.
The main idea is that the internal module has a different interface than the module inside the wrapped library. (E.g., Blah.Foo.y gives you Unbound value error.)
You may end up needing a lot of duplication of interface depending how you set things up. You might need to reorganise some stuff and explicitely declare module types to avoid this.
But it works nicely together with dune test support and with the rather specific structure of my project (lexing / parsing).