Alcotest testable of `module CharMap = Map.Make(Char)`

Learning
1

Hi,

I am self-learning the CS3110 (OCaml) course and need some help, because my head is getting tired/exploding after reading and playing with Fmt and Alcotest sources. Pretty different beasts compared to most Xunit frameworks outside FP.

I have 2 questions

  1. How can I make an Alcotest testable of CharMap defined below (using existing combinators)?
  2. More general question: Am I testing ‘not equals’ in an idiomatic way (the last check) below? Also are the Alcotest API always checking a testable (a “sig” and/or module type" with equal and pp function) such that other kind of assertions (greater than, not equal, text starts with etc…) are using this testable equal abstraction. Seems impossible…so are there other “check-abstractions” in Alcotest?
module CharMap = Map.Make(Char)

let test_bindings() =
  (* TODO: How to create testable for stdlib CharMap above *)
  (* let charStringMap = Alcotest.testable (Fmt.hashtbl (Fmt.pair Fmt.char Fmt.string)) (CharMap.equal Char.compare) in *)
  (* testables *)
  let eq = Alcotest.(list (pair char int)) in
  let not_eq = Alcotest.neg eq in
  (* values to test *)
  let b1 = CharMap.(empty |> add 'x' 0 |> add 'y' 1 |> bindings) in
  let b2 = CharMap.(empty |> add 'y' 1 |> add 'x' 0 |> bindings) in
  let cm3 = CharMap.(empty |> add 'x' 2 |> add 'y' 1 |> remove 'x' |> add 'x' 0) in
  let b3 = CharMap.(cm3 |> bindings) in
  let b4 = CharMap.(cm3 |> add 'z' 5 |> bindings) in
  (* asserts/checks for eq and not_eq *)
  Alcotest.(check eq "b1 = b2" b1 b2);
  Alcotest.(check eq "bi = b3" b1 b3);
  Alcotest.(check eq "b2 = b3" b2 b3);
  Alcotest.(check not_eq "b3 <> b4" b3 b4)
1 Like
2

Hi. Welcome to OCaml! :slight_smile:

You’re running into the fact that combinator libraries like Fmt and Alcotest struggle to interact with datastructures provided only as functors in the standard library.

  • For the pp component, you can use something like iter_bindings to define a printer from an iterator. Another option is to use Fmt.using to convert to a different datastructure (like a list of bindings), and then supply the pretty-printer for that instead.

  • For the equals, you need to pass an equality function for the values being stored in the map. The equality function over keys is already implied by the Char module passed to the Map.Make functor, so isn’t required.

Putting it together:

let char_int_map =
  let pp =
    Fmt.Dump.iter_bindings CharMap.iter (Fmt.any "map") Fmt.char Fmt.int
  in
  let equal = CharMap.equal Int.equal in
  Alcotest.testable pp equal

The Alcotest check functions are sadly limited by their type. What you’re doing with Alcotest.neg will work, but produces a confusing failure message. In such cases I’d normally do the check in userland and then call an assert_ function; we should almost certainly be providing that in Alcotest directly.

There are not, though there are plans to improve the situation (likely by altering the existing one but potentially by introducing something new altogether). There is a temptation to take test assertion DSLs much too far, but Alcotest should certainly be providing more than it does :slight_smile:

1 Like
3

Thanks for the pp code. I find Fmt (and Format in stdlib) very difficult to use for an OCaml beginner.

What you’re doing with Alcotest.neg will work, but produces a confusing failure message . In such cases I’d normally do the check in userland and then call an assert_ function

But the assert_ failure message also has the disadvantage of the not showing why the predicate said “fail”. But of course the predicate is outside of the Alcotest.testable bool and therefore can be custom.

   Expected: `true'
   Received: `false'

BTW I am certainly more used to the “rich” assertions API (like the scala link) in the xunit libs I use outside OCaml. I shifted to Alcotest, because I didn’t like the failure messages in ounit2. Are there any other xunit libs in OCaml.

Would it be possible to kind of forget about equal semantics, that is interpret equals as a close_enough/matches predicate with same signature) and create functors for different testable policies (probably type specific) with different equal semantics? I am not saying this approach is good. Maybe pointless if the the testable (provided by a functor) cannot control the failure message. assert_ is certainly simpler and therefore preferred if the failure message is not tailored to the testable policy.

1 Like
4

But the assert_ failure message also has the disadvantage of the not showing why the predicate said “fail”. But of course the predicate is outside of the Alcotest.testable bool and therefore can be custom.

Good point :slight_smile: I guess the only real way to fix the issue externally is to define one’s own check function that calls Alcotest.fail(f) in the failure case (e.g. here).

Are there any other xunit libs in OCaml.

You may be interested in @jobjo’s Popper library, which has a richer test proposition API (though at the moment still no not_equal, from what I can see).

Would it be possible to kind of forget about equal semantics, that is interpret equals as a close_enough/matches predicate with same signature) and create functors for different testable policies (probably type specific) with different equal semantics? I am not saying this approach is good. Maybe pointless if the the testable (provided by a functor) cannot control the failure message.

Indeed, if we’re not too concerned about the error messages being readable then it’s possible to make the equality function do whatever you’d like and return false in the failure case – Alcotest doesn’t require that the equal function actually behave like an equality function.

My general feeling is that such test assertions are only really useful in the extent to which they improve failing test messages (and perhaps also that they encourage writing more readable tests), so I wouldn’t do this myself. It’s really on Alcotest itself to provide a better story here.

5

Thanks again @CraigFe

You are probably the maintainer of Alcotest (looked at contributors on github…but for some reason beginners can only link twice from replies?!?), but I am going to ask anyway: If you were a beginner, what xunit framework would you choose in OCaml?

(I am not interested in property based testing yet…and guess qcheck has the lead there)

Do you know which has the highest usage among ounit2, alcotest, popper and crowbar?

(I have found it extremely hard to google for help in OCaml)

P.S.

I started using the REPL (liked that a lot)
Then moved to inline tests and PPX (liked that a lot)
Then moved to ounit2 (liked that a little less)
Then moved to alcotest (had to study for a day to use it…didn’t feel that great the first day)

BUT I am not just trying to pass some exam or test. I am trying to learn to get a proper workflow in a different (language, stdlib, tools, …) setup. I have that in other (language, stdlib, tools, …) combinations.
I would like to be able to build parsers and interpreters in OCaml and need to mature with “The OCaml Platform”.

PPS

It is actually Alcotest used in the best practices section…under “running tests” (no link)

PPPS

Maybe I should open new item for this. But I am going to ask anyway: Stdlib, Base, Core, LWT, Battery(?) etc is some of the most confusing parts of OCaml Platform. The “corelib” seems a bit divided (and that is an understatement). Are there any conventions in OCaml for the basic traits/type classes eq, ord, show etc. in OCaml? Think the most basic type classes from Haskell Prelude without all the category theory (monoid, applicative functor, monad, yada yada yada) based abstract types, or maybe IEquatable, IComparable, IFormattable etc from .NET, if you are acquainted with that platform (otherwise think Java). Also like the Realworld OCaml book a lot, but don’t like being forced to use Base and Core. I would rather invent my own stuff (probably coping bits and pieces from the established Stdlib alternatives)

Du you know af any good intros to the Stdlib (or Stdlib alternatives) focusing of eq, ord, show etc basic protocols?

2 Likes
6

I also had a frustrating time getting going with Fmt and Alcotest when learning OCaml. Part of this was just getting familiar with OCaml & its design patterns but a reasonable chunk was due to the Alcotest API and the lack of a corresponding manual. The situation hasn’t changed much since then, so your own experience is not too surprising :slight_smile: Hopefully it will improve shortly.

I would recommend that beginners read the Real World OCaml testing chapter and follow the approach described there. Personally I find ppx_expect and ppx_inline_test to offer the best out-of-the-box experience for writing non-property-based tests in OCaml today. Alcotest is perhaps less of a magical black box, but it’s less straightforward and more cumbersome to use even after becoming familiar with it. Once I have time to fix the more basic limitations with Alcotest, I plan to propose adding PPX features of that sort.

By the very arbitrary metric of counting Opam packages that depend on them, Alcotest comes out as the most popular:

$ pkgs=(alcotest ounit2 popper ppx_expect ppx_inline_test qcheck)

  for pkg in $pkgs; \
    do printf "%-15s " $pkg
    opam list --depends-on $pkg --with-test --short | wc -l
  done

alcotest             402
ounit2                51
popper                 2
ppx_expect           104
ppx_inline_test       85
qcheck                45

This is quite a popular topic on this forum so I won’t retread too much of it here. You may be interested in this thread, which I think is the last time it was discussed. As far as non-Base Stdlib alternatives go, you may be looking for Containers. It has the advantage that its individual modules are independent of each other, so individual snippets can be imported into your own projects quite easily. It’s also quite well-documented as these things go.

Personally, I just default to Base / Core whenever I’m not writing for a released Opam package these days. They’re much harder to consume in pieces, but make up for that in internal consistency & an opinionated approach to safety.

7

@CraigFe Thanks again. Great answers :muscle: Slightly (hmm totally) off topic. Are you running NixOS, or do you only have nix installed (maybe using home-manager) on another distro? (saw your ocaml script post with nix-shell trick). Is Nix a thing in the OCaml community?

1 Like
8

Just Nix (+ home-manager) for now :slight_smile: I’ve tried NixOS a few times and bounced off it; will probably give it another go once I understand the internals a bit better (or when the UX improves to the point that I don’t need to do that to get by).

It’s definitely a thing, though not quite as much as in the Haskell community (yet). We have opam2nix for building Nix development environments, and there are at least a few big projects that do this (coq and dune spring to mind). We also have a few people doing really cool things with e.g. Nix cross compilation, which go way above my head for now.

9

Cool. I use Nix and Home-manager myself.

But unfortunately for .NET (the platform I use the most) a nix-based dev environment does not work, because the closed source debuggers (there are 3 kinds, all closed source, MS has 2 and JetBrains 1) does not work with a nix derived .NET SDK/Runtime.

But good to know that OCaml has great Nix support. Right now I am not using OCaml with Nix. For now I just use opam and direnv with the following .envrc in the root (here from the CS3110 course folder)

opam switch cs3110-2021fa

eval $(opam env)
10

Yes, check out OCaml library : Map.OrderedType which represents ‘types which can be equated and compared’, and OCaml library : Hashtbl.HashedType which is for ‘types that can be hashed’.

Since OCaml modules are structurally typed, you can simply define the functions from those module types in your own modules, and they will then be compatible with functors which expect the above module types.

My 2c re: testing: snapshot/expect/approval testing imho offers the best dev experience, and we shouldn’t need a PPX to implement it. ‘All’ it does is read a snapshot file from a predefined location, generate the new result, and if they are different then print out a diff and fail the test. We should think about adding this into Alcotest, it would really up the game. The user would still need to define a printer though, so not much different than the current situation.

1 Like
11

@yawaramin Thanks.

I guess a functor (combined with lower level sigs…module types) is the way to enforce a “protocol” for some internal part of a structure (module). And this is how one can combine protocols (shapes such as eq, ord, monoid, monad etc) into higher kinded modules (shapes) that build upon the “bound” protocols, and achieve polymorphism/reuse/conventions.

I also guess that the protocol (‘sig’ in ocaml) are ‘nameless’. With that I mean that I could define a structurally equivalent (copy) of OrderedType with a different name, but that name would be of no importance to the ocaml compiler. All the compiler care about is the shape/structure (this is maybe evident, but coming from C# etc this is magic :grinning:)

A copy of Stdlib.OrderedType

module type MyOrderedType = sig
  type t

  val compare : t -> t -> int
end

Are there many differently named, but structurally equivalent ‘shapes’ of sig’s in Ocaml? Are there a catalog of agreed upon ‘shapes’ somewhere. Someting similar to agreed upon binary operators?

Even though the shape /structure is important, doesn’t ocaml developers care about standard names for ord, eq, show, …, all the way up to monad transformers or whatnot?

12

Right, that’s what I mean by ‘OCaml modules are structurally typed’, literally, the compiler figures out their types by their public structure.

Re: ‘standard names’, the two I shared above are in the standard library, so you may consider them standard :slightly_smiling_face: Other than that there may be a few others in other libraries; but none that are standardized to such an extent.