Strange behavior with `Scanf.scanf` and black-box testing

Learning
1

Hi folks -

After learning enough OCaml to finish Advent of Code this year, I’ve been doing some problems on https://open.kattis.com. All of the solutions I’m building are a single file executable, so I’ve been testing them in a black-box style via dune and ppx_expect. I wrote a little helper to spin up a process, send it some input via stdin, and read the output via stdout. Oh, I’m also on OCaml 4.13.1 because this is the version that Kattis supports. Here’s the helper code:

let read_all ic =
  let output = ref "" in
  (try
     while true do
       let result = input_line ic in
       output := !output ^ "\n" ^ result
     done
   with
   | End_of_file -> ());
  !output
;;

let run_solution name input =
  let bin_path = Unix.getcwd () ^ "/../bin/" ^ name ^ ".exe" in
  let ic, oc = Unix.open_process bin_path in
  output_string oc input;
  flush_all ();
  ic |> read_all
;;

This was working fine until I mixed use of read_line and Scanf.scanf in one of my executables. Suddenly my tests were hanging forever, even though running the executable directly worked fine.

Here’s a simple example test:

let%expect_test _ =
  let test_input = {|5
Hello
|} in
  Helper.run_solution "example" test_input |> print_endline;
  [%expect {| 5 Hello |}]
;;

This executable successfully passes the test just fine:

let () =
  let n = Scanf.scanf "%i\n" Fun.id in
  let s = Scanf.scanf "%s\n" Fun.id in
  Printf.printf "%i %s\n" n s
;;

However, this one hangs forever in the test, but works fine if I run the executable in the shell!

let () =
  let n = Scanf.scanf "%i\n" Fun.id in
  let s = read_line () in
  Printf.printf "%i %s\n" n s
;;

What am I doing wrong? Also, is there a way better way to accomplish black-box testing of executables?

Thanks!

2

My guess is that Scanf is using its own buffering mechanism. So, when you are piping the input from a test harness, the call to Scanf.scanf captures the whole input, which means that read_line has nothing left to read and hangs forever. On the contrary, when inputting from the shell, Scanf.scanf cannot read more than what you have typed, that is, only the first line.

3

@silene is right, Scanf.scanf performs additional buffering that can cause it to read ahead too far. You can see that by running your program under test from a shell with standard input redirected from a file containing your test input: read_line raises an End_of_file exception, because Scanf.scanf read ahead too far and left nothing to read for read_line.

Speaking of end-of-file: the flush_all () line in your test driver should be close_out oc. For one thing you don’t need to flush all the channels, just oc. But more importantly, closing the channel will send an “end of file” condition to the other end of the pipe, so that further reads will report end-of-file instead of blocking forever waiting for more input, as you observed.

At any rate, to process line-oriented input, I suggest that you do all input with read_line, then extract data from the resulting strings using Scanf.sscanf if needed, or simpler conversions like int_of_string. For example:

let () =
  let n = int_of_string (read_line()) in
  let s = read_line () in
  let coords = Scanf.sscanf (read_line()) "X=%d, Y=%d" (fun x y -> (x, y)) in
  ...
4

I changed the title of the report to point the finger to Scanf.scanf instead of Unix.open_process, since the latter is innocent. Still, there are a few things in your test driver that are not quite right:

  • As mentioned above, oc must be closed. It would be cleaner to close ic as well.
  • If the program under test interleaves reading input and producing output, and if the amount of input and output exceeds what can be held in a pipe buffer, a deadlock can occur: the program under test is blocked while writing to a pipe that is full, and the test driver is blocked writing to a pipe that is full. The solution is to use threads to do input and output concurrently:
  let t = Thread.create (fun () -> output_string oc input; close_out oc) () in
  let output = read_all ic in
  Thread.join t;
  close_in ic;
  output
  • The read_all function has quadratic complexity because of the repeated string concatenations. Also, I’m afraid you’re putting the \n at the wrong places. A simpler and more efficient solution is to accumulate input lines in a string buffer (module Buffer from the standard library).
5 Likes
5

Excellent, thanks for the clear explanation!