Boltzgen a test generator for teachers

Community
,
1

Boltzgen a test generator for teachers.

I am pleased to introduce Benoit Barbot / BoltzGen · GitLab ; Boltzgen
: Boltzmann sampler test generator is a tool to generate tests. Given a function signature it generates a random set of call of this function on generated random input. Inputs are generated using a Boltzmann sampler for unlabeled structure. This project has the following objectives:

  • The primary goal was developing automatic grading system for purely functional programming exercises.
  • Generate values with as little information as possible in most case only its type, the input by the user should be as small as possible;
  • Strong probabilistic guarantee on the uniformity of the sample process thanks to Boltzmann sampler theory.

Given a function specification as a type signature :

type 'a tree =   Empty  | Node of 'a tree * 'a * 'a tree
val sum: int list tree -> int

Boltzgen returns calls to this function :

sum (Node((Empty,[],(Node((Empty,[(-4); (-2); 6],Empty))))))
sum (Node(((Node((Empty,[],Empty))),[13],Empty)))
sum (Node(((Node((Empty,[57],Empty))),[],(Node((Empty,[],(Node((Empty,[],(Node((Empty,[],Empty))))))))))))
...

You can try it with a buggy web interface here OCaml test gen.
Or install it with opam and test locally.

This tool was quickly developed during the first COVID 19 lockdown to quickly generate tests for programming exercise, many bugs remains, comments and contributions are welcome.

3 Likes
2

Hello,
It seems we need an account to access the git.

3

No, but there was a typo in the URL, thanks for reporting.

4

Is there a reason that the utility of such a tool should be restricted to teachers? Property based randomized test systems also need tools like this for real world testing systems, do they not?

5

No, they should not be restricted to teachers,
There are some tool in Haskell using the same technique for test generations:

and some project with similar objectives :

.

But this utility in particular was built with teaching and automatic grading in mind. When teaching the base of functional programming, most exercises are writing a well define standalone pure function.

When writing real world code, you often (depending on programming style) have a mutable custom type and function which interact with this type (ex HashTbl). Generating good test in this setting require additional work (but I have a few ideas).

For this tool additional developments are required to use it in a testing framework :

  • The tool was written in haste, it probably should be entirely rewritten.
  • Make it more reliable, right now I look at the generated result to make sure everything went right, more users are required !
  • It should be integrated in a testing framework which requires, I think, moving the generation at compile time and not at the execution time, for each type several functions are generated, this generation should be done with ppx_deriving, I need help to do that.
  • The ToDo list should be emptied, in particular I need help for the runtime generation of Ocaml value which requires knowledge in the internal Ocaml representation of value.
  • One goal of this project could be to work with https://opam.ocaml.org/packages/qcheck/ as a generator generator.

The theory of Boltzmann sampling is really beautiful in the fact that abstract math (namely formal power series) are used in practice to generate quite efficiently discrete value. I believe it could be used for efficient testing in general.

1 Like