I’m happy to share release 0.2 of qcheck-lin
and qcheck-stm
for black-box property-based testing.
-
qcheck-lin
requires little more than an interface description. It allows to test a library for sequential consistency, that is, whether results obtained from using it in parallel agree with some linear, single domain execution. -
qcheck-stm
is a model-based, state-machine framework for both sequential and parallel testing. It allows to test an imperative interface against a pure model description, and thereby allows to express and test intended behaviour beyond a signature description.
For example, here’s a minimal qcheck-lin
test of a selection of the Stdlib
Hashtbl
interface:
module HashtblSig =
struct
type t = (char, int) Hashtbl.t
let init () = Hashtbl.create ~random:false 42
let cleanup _ = ()
open Lin
let a,b = char_printable,nat_small
let api =
[ val_ "Hashtbl.add" Hashtbl.add (t @-> a @-> b @-> returning unit);
val_ "Hashtbl.remove" Hashtbl.remove (t @-> a @-> returning unit);
val_ "Hashtbl.find" Hashtbl.find (t @-> a @-> returning_or_exc b);
val_ "Hashtbl.mem" Hashtbl.mem (t @-> a @-> returning bool);
val_ "Hashtbl.length" Hashtbl.length (t @-> returning int); ]
end
module HT = Lin_domain.Make(HashtblSig)
;;
QCheck_base_runner.run_tests_main [
HT.lin_test ~count:1000 ~name:"Hashtbl DSL test";
]
Running this test quickly finds a minimal counterexample to illustrate that Hashtbl
is not safe to use in parallel:
Messages for test Hashtbl DSL test:
Results incompatible with sequential execution
|
Hashtbl.add t '<' 0 : ()
|
.------------------------------------.
| |
Hashtbl.add t 'a' 0 : () Hashtbl.remove t '<' : ()
Hashtbl.length t : 0
We presented preliminary work on both these libraries at the OCaml Workshop 2022. The libraries furthermore underlie our continuing effort to test the multicore runtime of OCaml 5.x, and have helped identify several issues.
The 0.2 release adds a range of features and bugfixes, including support for OCaml 4.12.x, 4.13.x and 4.14.x without the Domain
and Effect
modes.
Detailed release notes and more information is available from the GitHub repository:
GitHub - ocaml-multicore/multicoretests: PBT testsuite and libraries for testing multicore OCaml
Happy testing!