Ppx_inline_test: how to get bettor error messages?

Hello,

I’m reading the book “Ocaml from the very beginning”, and using inline tests to validate my implementations + to explore different ways of structuring the code.

Overall, I very much enjoy the ability to declare inline tests, as such:

let rec insert a lst =
  match lst with
  | [] -> [ a ]
  | x :: xs -> if   a <= x
               then a :: x :: xs
               else x :: insert a xs
               [@@ocamlformat "disable"]

let%test _ = insert 1 [ 1; 2 ] = [ 1; 1; 2 ]
let%test _ = insert 2 [ 1; 2 ] = [ 1; 2; 2 ]
let%test _ = insert 3 [ 1; 2 ] = [ 1; 2; 99 ]

I thing that’s bugging me though, is that the failure message doesn’t really tell me what went wrong. I introduced an obvious typo here to illustrate.

$ dune runtest 
File "lib/chapter05.ml", line 11, characters 0-45: <<(insert 3 [1; 2]) = [1; 2; 99]>> is false.

FAILED 1 / 107 tests

I’d like to get something resembling the kind of output below, here’s an example of a similar feature in Elixir:

defmodule Example do
  @doc """

  ## Examples

      iex> Example.insert(1, [])
      [1]

      iex> Example.insert(1, [1])
      [1, 1]

      iex> Example.insert(2, [1, 2])
      [1, 2, 2]

      iex> Example.insert(3, [1, 2])
      [1, 2, 99]


  """
  def insert(h, []), do: [h]

  def insert(h1, t1) do
    [h2 | t2] = t1

    if h1 <= h2 do
      [h1 | insert(h2, t2)]
    else
      [h2 | insert(h1, t2)]
    end
  end
end

Can I get close to something like this with ppx_inline_test or any other tool?

I’m not looking at creating a “proper” test file, I’d like to keep my “test examples” close to the implementation.

Prior to finding ppx_inline_test, I played with the qtest library but found it didn’t work too well with dune out of the box, I would often need to clear dune’s cache to ensure a proper test run.

Thanks

1 Like

It looks like you are looking for expectation tests, which are well integrated in dune with the dune promote command. See Writing and Running Tests — Dune documentation

The fact that tests are not run when they have already run and nothing has changed is a feature of dune (test runs is a build target like any other). You can still force them to run with dune runtest --force.

2 Likes

I second the recommendation for expect tests. But, if you want to stick with assertion style testing, but just want better error messages, you could try ppx_assert.

1 Like

That probably explains my initial confusion :slight_smile:

Thanks for pointing me in the right direction @bnguyenvanyen @mooreryan.

ppx_assert looks more like what I want, although ppx_expect has a nicer output, interesting tool!

Here’s an example of a bogus failure, for reference:

let print_nums nums =
  nums |> List.map string_of_int |> String.concat ", " |> print_string

[@@@ocamlformat "disable"]

(* ppx_inline_test *)
let%test _ = ins_rev 3 []       = [ 3 ]
let%test _ = ins_rev 3 [ 1 ]    = [ 3; 1 ]
let%test _ = ins_rev 3 [ 1; 2 ] = [ 9; 2; 1 ]
(* File "lib/chapter05.ml", line 107, characters 0-45: <<(ins_rev 3 [1; 2]) = [9; 2; 1]>> is false. *)

open Base

(* ppx_assert *)
let%test_unit _ = [%test_eq: int list] []          @@ rev_inssort []
let%test_unit _ = [%test_eq: int list] [ 1 ]       @@ rev_inssort [ 1 ]
let%test_unit _ = [%test_eq: int list] [ 2; 1 ]    @@ rev_inssort [ 1; 2 ]
let%test_unit _ = [%test_eq: int list] [ 9; 2; 1 ] @@ rev_inssort [ 1; 2; 3 ]
(*
File "lib/chapter05.ml", line 116, characters 0-77: <<(([%test_eq : int list]) [9; 2; 1]) @@ (rev_i[...]>> threw
(runtime-lib/runtime.ml.E "comparison failed"
  ((9 2 1) vs (3 2 1) (Loc lib/chapter05.ml:116:29))).
...
*)

(* ppx_expect *)
let%expect_test _ = print_nums @@ rev_inssort [];          [%expect {||}]
let%expect_test _ = print_nums @@ rev_inssort [ 1 ];       [%expect {|1|}]
let%expect_test _ = print_nums @@ rev_inssort [ 1; 2 ];    [%expect {|2, 1|}]
let%expect_test _ = print_nums @@ rev_inssort [ 1; 2; 3 ]; [%expect {|9, 2, 1|}]
(*
-|let%expect_test _ = print_nums @@ rev_inssort [ 1; 2; 3 ]; [%expect {|9, 2, 1|}]
+|let%expect_test _ = print_nums @@ rev_inssort [ 1; 2; 3 ]; [%expect {|3, 2, 1|}]
*)

ppx_inline_test was short and sweet though. If you anyone knows a technique or two to make my test calls terser, please let me know :slight_smile:

1 Like

Answering my own question…

Improvement n°1:

(* ile => int list eq *)
let ile =
  let open Base in
  [%test_eq: int list]

let%test_unit _ = ile []          @@ rev_inssort []
let%test_unit _ = ile [ 1 ]       @@ rev_inssort [ 1 ]
let%test_unit _ = ile [ 2; 1 ]    @@ rev_inssort [ 1; 2 ]
let%test_unit _ = ile [ 3; 2; 1 ] @@ rev_inssort [ 1; 2; 3 ]

Improvement n°2:

(* WOW *)
let ( == ) = ile

let%test_unit _ = rev_inssort []          == []
let%test_unit _ = rev_inssort [ 1 ]       == [ 1 ]
let%test_unit _ = rev_inssort [ 1; 2 ]    == [ 2; 1 ]
let%test_unit _ = rev_inssort [ 1; 2; 3 ] == [ 3; 2; 1 ]

Probably not conventional but I’m amazed right there :exploding_head:

It’s not just nicer output. ppx_expect also has the advantage of not requiring you to write the expected result by hand in the first place; you can just run the tests and let dune promote fill in the results, and do a quick skim to see if they’re what you expect.

1 Like

Thanks, I did see that :slight_smile:

I could definitely see myself using this cool feature, very handy by the looks of it.