Jbuilder: Inline tests

jeremiedimino · March 18, 2018, 11:19pm

Hi,

Jbuilder 1.0+beta19 has just been released in opam. There are several exciting new features in this release, but one I think deserves special attention is the newly added support of inline tests.

Inline tests allow to embed tests directly in the library code, in order to have the tests close to the tested functions or simply has a general framework to write unit tests. Setting up such tests usually require to write some tedious build system boilerplate, which has been implmented correctly once and for all in Jbuilder. All you need to do get jbuilder to build and run your tests is write (inline_tests) in your jbuild file.

The support in Jbuilder is completely generic, and can be used with various backends such as
ppx_inline_test or qtest. Defining a new backend is very easy and the manual explains how to do it.

Example 1: using ppx_inline_test

Ppx_inline_test is an inline tests framework developped by Jane Street. As its name suggest, it relies on a ppx rewriter. Starting from version v0.10.1, ppx_inline_test works smoothly with Jbuilder by simply adding (inline_tests) to the jbuild file. For instance:

$ cat jbuild
(library
 ((name tests)
  (preprocess (pps (ppx_inline_test)))
  (inline_tests)))

$ cat foo.ml
let%test _ = 6 * 7 = 0

$ jbuilder runtest
(cd _build/default && ./.tests.inline-tests/run.exe inline-test-runner tests -source-tree-root . -diff-cmd -)
File "foo.ml", line 1, characters 0-22: <<(6 * 7) = 0>> is false.

FAILED 1 / 1 tests

Example 2: using ppx_expect

Inline expectation tests are supported as well, for instance using ppx_expect >= v0.10.1:

$ cat jbuild
(library
 ((name tests)
  (preprocess (pps (ppx_expect)))
  (inline_tests)))

$ cat foo.ml
let%expect_test _ =
  Printf.printf "%d" (6 * 7);
  [%expect]

$ jbuilder runtest --diff-command "diff -u"
          sh (internal) (exit 1)
(cd _build/default && /usr/bin/sh -c 'diff -u foo.ml foo.ml.corrected')
--- foo.ml	2018-03-18 19:05:38.843682465 -0400
+++ foo.ml.corrected	2018-03-18 19:06:02.305686745 -0400
@@ -1,3 +1,3 @@
 let%expect_test _ =
   Printf.printf "%d" (6 * 7);
-  [%expect]
+  [%expect{| 42 |}]

$ jbuilder promote
Promoting _build/default/foo.ml.corrected to foo.ml.
$ cat foo.ml
let%expect_test _ =
  Printf.printf "%d" (6 * 7);
  [%expect{| 42 |}]

$ jbuilder runtest

You can check this blog post for more information about testing with expectations.

rdavison · March 21, 2018, 4:04pm

This is awesome, however, for the life of me I can’t figure out how to get the (glob) and (regexp) features to work. Are they broken in this release?

I’m using:

jbuilder.1.0+beta19.1
ppx_expect.v0.10.1

$ cat jbuild
(library
  ((name example)
   (inline_tests)
   (libraries (core))
   (preprocess (pps (ppx_jane)))))

$ cat foo.ml
open Core

let rec fact = function
  | 1 -> 1
  | n -> n * fact (n - 1)

let%test _ = fact 5 = 120

let%expect_test _ =
  printf "foo";
  [%expect {| foo\|bar (regexp) |}]

let%expect_test _ =
  printf "foo";
  [%expect {| \(foo\|bar\) (regexp) |}]

let%expect_test _ =
  printf "foo";
  [%expect {| * (glob) |}]

$ jbuilder runtest
Done: 92/94 (jobs: 1)File "foo.ml", line 1, characters 0-0:
        diff (internal) (exit 1)
(cd _build/default && /usr/bin/diff -u foo.ml foo.ml.corrected)
--- foo.ml	2018-03-21 12:01:43.000000000 -0400
+++ foo.ml.corrected	2018-03-21 12:01:45.000000000 -0400
@@ -8,12 +8,12 @@

 let%expect_test _ =
   printf "foo";
-  [%expect {| foo\|bar (regexp) |}]
+  [%expect {| foo |}]

 let%expect_test _ =
   printf "foo";
-  [%expect {| \(foo\|bar\) (regexp) |}]
+  [%expect {| foo |}]

 let%expect_test _ =
   printf "foo";
-  [%expect {| * (glob) |}]
+  [%expect {| foo |}]

yoriyuki · March 21, 2018, 4:52pm

This is a great new and would drastically simplify testing. But just for showing interest not request, I want to see easy integration of quick check style test (property based test with random test data generation). Sometimes testing on several hand-crafted values are not enough.

jeremiedimino · March 21, 2018, 5:12pm

@rdavison, we consider (glob) and (regexp) to be a design mistake, as they break the normal workflow: when you accept the correction you have to go back and edit the expectation. In practice this is a pain, it is much better to process the output to get rid of noisy parts.

There is a flag to allow support for (glob) and (regexp), but we are planning to get rid of output patterns entirely soon, so I don’t recommend using it.

We have an expect_test_helpers_kernel library that helps writing expect tests and process the output, for instance to hide file positions.

jeremiedimino · March 21, 2018, 5:15pm

@yoriyuki this feature is mainly about the workflow, then you are free to use any testing library to write the tests themselves. I know that in Core_kernel and Core we support quick check and every module has generators.

kevinqiu · March 27, 2018, 6:00pm

I am having an issue where inline tests depend upon static files located outside the src directory since jbuilder runs tests from within that directory. Take for example the following:

inline test located in foo/src/foo.ml
depends upon foo/static/file.file

The solution I have come upon is to add a deps clause to the the inline_tests stanza that copies the static folder. This creates a directory structure in the _build that mirrors what exists in the repo. So, in my tests, I would access that file relatively via ../static/file.file.

However, it seems wrong to have to do this for every static directory being used, is there a recommended way to do this?

Should I move static into src? I’m not sure that would solve the issue of having to specify every directory to be copied.

Or is there a way to specify which directory the tests should be run from? Perhaps a setting to run the tests from the workspace root? I see that jbuilder does:

cd _build/default/foo/src && ./.foo.inline-tests/run.exe

Having it run in the root would also solve my problems since no copying is needed and foo/static can be referenced directly.

jeremiedimino · March 27, 2018, 6:32pm

However, it seems wrong to have to do this for every static directory being used, is there a recommended way to do this?

Why does it seem wrong? Jbuilder needs to know what files every command will read in order to schedule commands and avoid re-running commands in incremental builds, so you do need to specify dependencies for tests.

kevinqiu · March 27, 2018, 7:17pm

Perhaps wrong is the incorrect word, I was wondering if there was a better way to have the behavior I want. Your explanation makes sense for the case where static needs to be copied into the build.

Is there a way to accomplish the latter(have inline tests run from SCOPE_ROOT)? Although I anticipate the answer is that I shouldn’t use (inline_tests) since the default behavior works for the majority of people and this is a an unusual case and I probably shouldn’t be depending on static files in inline tests in the first place.

EDIT:
The solution that seemed to be the best for my situation ended up being utilizing (inline_tests) to build the tests and then defining a custom alias that does chdir before running the test binary rather than using runtest. jbuilder is able to see that .foo.inline-tests/run.exe depends upon the inline tests.

jeremiedimino · March 27, 2018, 8:03pm

There is no way at the moment. In general running things for the directory where they are defined is more modular: for instance you can do mv foo bar/foo and everything will still work as it should without having to modify the tests.

kevinqiu · March 27, 2018, 8:12pm

Thanks for the quick replies. Yes, it is definitely better to not have these inter-directory dependencies but it is unfortunately unavoidable for myself at the present. The workaround above is sufficient though, all in all jbuilder is still a 100x improvement over the build system that was in use previously.

grayswandyr · April 2, 2018, 12:58pm

Hi Jérémie,
great! is there a way to have this work with a preprocess action (not pps) section? (The following example is artificial, as ppx_deriving would pose no problem, but I use visitors which, AFAIK, cannot appear in a pps clause)

$ cat jbuild 
(library
 ((name tests)
	 (preprocess (action
		(run ppxfind -legacy ppx_deriving.std,ppx_inline_text,ppx_expect --as-pp ${<})))
  (inline_tests)))
$ cat foo.ml 
type t = C of int [@@deriving eq]
let%test _ = 6 * 7 = 42
let%expect_test _ =
  Printf.printf "%d" (6 * 7);
  [%expect {| 42 |}]
$ jbuilder runtest 
File "jbuild", line 5, characters 2-16:
Error: No inline tests backend found.

(BTW is it possible to have both a pps and an action section as preprocess clauses?)

rgrinberg · April 3, 2018, 4:16am

It’s not a problem of having inter-directory dependencies. It’s just that you should make your build system aware of them for the reasons explained by Jeremie. The usual reason why tests are hard coded to run in a certain directory is that they expect fixtures to exist in a certain place. If you want to specify those places relative to your project root, you should simply make your tests aware of what the project root is. E.g. PROJECT_ROOT=$(git root) jbuilder runtest.

Note that you should still specify the dependencies using relative paths.

rgrinberg · April 3, 2018, 4:20am

Probably not possible in the current version of dune. Dune discovers inline tests (and other) backends from the dependencies you list as preprocessors and libraries. What you can try and do is use the per_module specification for preprocess to limit your use of classical ppx to a single module. Elsewhere, you can use ppx_inline_test as usual.

Only for different modules. See per_module in the documentation.

grayswandyr · April 3, 2018, 9:24am

Ah cool. Now, is there a way in a per_module to say “all modules except X and Y”? Like the :standard \ ... clause for library modules? I couldn’t find way to handle this.

Aurelie_H · April 24, 2018, 3:15pm

Hi,

I have troubles with jbuilder and inline tests.
I have :

jbuilder 1.0+beta20
ppx_inline_test.v0.11.0

When I try to run tests, I got the following message : " Error: No inline tests backend found."

Here is my project
./src:
_build fact.ml fact.mli jbuild
./test:
_build jbuild test.ml

And my jbuild files.

the one in src
(jbuild_version 1)

(library
((name fact)
(inline_tests)
(preprocess (pps (ppx_inline_test)))))
the one in test
(jbuild_version 1)

(executable
((name test)
(libraries (fact))))

(alias
((name runtest)
(action (run ./test.exe))))

What am I doing wrong?

Thanks,
Aurélie

jeremiedimino · April 24, 2018, 4:35pm

@grayswandyr, there is no way at the moment but this could be added. Feel free to open a ticket about it

@Aurelie_H, try reinstalling ppx_inline_test. If it was installed with an older version of jbuilder and then jbuilder was upgraded with ppx_inline_test being reinstalled it won’t work

Aurelie_H · April 25, 2018, 7:06am

@jeremiedimino, I’ve done that (several times! with reinstall and with remove and reinstall) and I still have the same error.

jeremiedimino · April 25, 2018, 7:44am

from where do you try to run the tests? Also, what is the contents of ppx_inline_test.dune in the directory reported by ocanlfind query ppx_inline_test?

Aurelie_H · April 25, 2018, 7:58am

I try to run the tests in the directory on top of the test directory. The one from whom I’ve shown the “ls”.

The content of the file ppx_inline_test.dune is “(dune 1 ())”

Aurelie_H · April 25, 2018, 8:05am

My computer just crashed, I had to reboot it, and now it works!
Sorry for the troubles for nothing!

Topic		Replies	Views
Building a test runner with ppx_inline_test Ecosystem jbuilder , testing , ppx	5	2690	March 18, 2018
Unit testing with jbuilder Ecosystem jbuilder , testing	12	3329	July 30, 2017
ANN: jbuilder 1.0+beta12 Ecosystem announce , jbuilder	2	1451	August 25, 2017
Any interest in inline test alcotest backend? Community	3	755	August 25, 2021
Some feedback on teaching with jbuilder as a build tool Ecosystem jbuilder	11	1515	December 16, 2018

Jbuilder: Inline tests

Example 1: using ppx_inline_test

Example 2: using ppx_expect

Related topics