[ANN] First release of Popper

Community
,
1

I am happy to announce a first release of Popper — an OCaml testing library for writing simple unit tests as well as property-based tests, using a uniform API.

A few things that distinguishes Popper from other property-based testing libraries:

  1. Generic shrinking — when a property-based test fails, the test function attempts to shrink the sample data. This works for any data type.
  2. Any invariants used when constructing samples for property-based tests are also respected when shrinking.
  3. There is a ppx for automatically deriving sample functions (generators in QuickCheck style libs) for custom data types.

The library also supports bundling and nesting tests arbitrarily, colorful output inspired by Alcotest, logging facilities and timing info.

Here are some starting points:

Below is an example of what Popper output looks like. It was generated by this code.

image
image1001×544 43.8 KB

This is a first release and there are few work-in-progress items. For instance:

  • A CLI for filtering and sorting tests.
  • An API for better blending with Lwt code bases.
  • A better way for limiting shrinking times for failing test samples.

Any suggestions or other feedback is much appreciated!

20 Likes
2

That seems quite impressive, congratulations. A suggestion would be to take some existing works based on property-based testing (for example, Jan’s work) and adapt it to your framework to serve as examples/show cases.

At a glance I didn’t see any public shrinking API, what do you use underneath? Seq.t? Also, is there a way to make a custom Sample.t from a generator and a shrinker?

In QCheck we’ve been working on a similar track with these PRs, so it’s interesting to see the convergence.

2 Likes
3

This is a nice looking tool! The interface looks really slick.

It’d be nice to add base_quickcheck to your comparison table. It also offers ppx and shrinking.

4

Thanks, I can take a look at these examples here. Or, did you have some specific repo in mind?

At a glance I didn’t see any public shrinking API, what do you use underneath? Seq.t ? Also, is there a way to make a custom Sample.t from a generator and a shrinker?

I don’t (yet) expose an API for obtaining a sequence of shrunken values. In general the design is different from QuickCheck style libraries in that shrinking is only performed on the input sequence fed to the generators/samplers. This strategy is inspired by the rather popular Python library Hypothesis. Here are some more details on how it work in Popper.

There is no coupling of generators/shrinkers in the same data-structure. So far I’ve been focusing on getting generic shrinking to work reasonably well (and fast) but it would be straight forward to extend the library to also allow overriding with custom shrinkers when need be.

In QCheck we’ve been working on a similar track with these PRs, so it’s interesting to see the convergence.

Nice, noticed you landed that for QCheck2. I’m also curious to compare the strategies since your underlying design is different.

2 Likes
5

This is a nice looking tool! The interface looks really slick.

Thanks, been trying to make the public API as intuitive as possible.

It’d be nice to add base_quickcheck to your comparison table. It also offers ppx and shrinking.

Sure, will take a look. Couldn’t find much information on that library tbh. Most likely missing some others as well.

6

There’s some documentation here Base_quickcheck.

1 Like