[ANN] First release of monolith

Community
1

It is my pleasure to announce the first release of Monolith.

Monolith offers facilities for testing an OCaml library (for instance, a data
structure implementation) by comparing it against a reference implementation.
It uses a form of black-box testing, and relies on afl-fuzz for efficiency.

The user must describe what types and operations the library provides. Under
the best circumstances, this requires 2-3 lines of code per type or operation.
The user must also provide a reference implementation of the library.

Then, like a monkey typing on a keyboard, Monolith attempts to exercise the
library in every possible way, in the hope of discovering a scenario where the
library behaves incorrectly. If such a scenario is discovered, it is printed
in the form of an OCaml program, so as to help the user reproduce the problem.

At this time, a tutorial is not yet available. There is however an API
documentation and a number of demos.

Repository:
https://gitlab.inria.fr/fpottier/monolith

API Documentation:
http://cambium.inria.fr/~fpottier/monolith/doc/monolith/Monolith/index.html

Installation:

  opam update
  opam install monolith

Happy testing,

François Pottier.

13 Likes
2

Looks interesting. From a very cursory glance it seems quite similar to crowbar. Can anyone more knowledgeable give a summary of the differences?

3

I also noticed the similarity. From the glance I had:

  • crowbar has a more complete support for built-in types (e.g., result) and more flexibility for custom types (e.g., via map or bind)
  • monolith has a meta-programming-y aspect (see code) that is completely absent from crowbar

I have used crowbar, but I haven’t used monolith so I can’t comment much more than that.

1 Like
4

Some aspects are common with crowbar: e.g., the source of random bits can be either a pseudo-random generator or controlled by afl-fuzz; and there are generators of values of concrete types (integers, lists, etc.).

I believe that the two approaches differ when testing an abstract data type (say, a balanced binary tree, for example). crowbar seems to follow a QuickCheck-like approach, where you generate balanced binary trees and you use them as inputs to test operations (or laws), one operation (or one law) at a time. monolith follows a different approach, where you never attempt to generate a value of an abstract data type; instead, you generate a scenario (a client program that uses the abstract data type) and you check that the observable results are correct.

It seems to me that monolith's black-box approach is the only one that is applicable when dealing with complex mutable data structures that involve sharing. In that situation, it doesn’t make sense to try to artificially generate valid data structures; instead, using the operations provided by the library to construct valid data structures is the only viable approach.

monolith attempts to make it easy for the programmer to describe the operations of the library (including their type and their specification). Ideally, a few lines of code per operation are enough, so the code required to fully test a small library of (say) 10 operations can be written in under an hour.