Running executable and inline tests in watch mode at them same time for Advent of Code

[p1xelHer0]

Hi folks! I have some problems I want to solve: I am building a little
boilerplate project for https://adventofcode.com/ and I’ve stumbled into some
problems in regards to dune exec, dune test and --watch:

Excuse my long thread but I felt it was better to write down everything I’ve tried. the project looks like this can can be found in the development branch here GitHub - p1xelHer0/advent-of-ocaml at development

.
├── \_build
├── \_opam
├── day_01..25 - self-contained library and executable
│   ├── dune
│   ├── input
│   ├── puzzle_01.ml - solution to AoC, contains inline tests
│   └── runner.ml - executable, calls the `run` function from `Puzzle_01`
├── lib
│  ├── dune
│  └── util.ml - library all other modules can use, can contain tests
├── aoc.opam
├── dune
└── dune-project

The idea here is that each day I, or some other user, would start a

dune exec ./day_01/runner.exe -w

and a

dune test ./day_01 -w

To get immediate feedback on the test of the example data and the output of the written program.

Now, this does not work since dune locks the _build directory.
This lock CAN be disabled with setting DUNE_CONFIG__GLOBAL_LOCK=disabled but
now the processes crash instead, so this does not work in my case.

My approach to this was to create an alias:

(rule
 (alias 1)
 (deps ./day_01/input)
 (action
  (progn
   (run dune build ./day_01)
   (run ./day_01/runner.exe))))

Now, using dune build @1 -w and achieve the same thing as dune exec ./day_01/runner.exe -w. (I’m not sure if it’s the EXACT same thing but you get
idea…).

Defining this as an alias allow us to use multiple aliases for the dune build
command:

dune build @1 @runtest -w

The issue now become that the runtest alias runs ALL the tests of the project,
not just the test for day_01. So if I user leaves Day 1 of Advent of Code
uncompleted and move on to day 2 they’d have to clean up the file before moving
on. That’s not good!

I have not been able to pass parameters to the runtest alias when using the
build command above. I was thinking something in the lines of:

dune build @1 @runtest ./day_01 -w. But that does not work, it still runs my
tests in ./day_02 and lib/util.ml.

Another idea I had was to expand my alias to include the tests in the alias
itself:

(rule
 (alias 1)
 (deps ./day_01/input)
 (action
  (progn
   (run dune build ./day_01)

   ; This seems like the easiest most straightforward solution?
   ; But this does not seem to work, or any alternative for that matter, be it
   ; runtest, build @runtest etc
   (run dune test ./day_01)

   ; An idea I had was to run the inline test directly:
   (run ./day_01/.puzzle_01.inline-tests/inline_test_runner_puzzle_01.exe)

   (run ./day_01/runner.exe))))

Now ppx_inline_test specificly mentions that you can’t run the test executable
directly. Doing this results in the following:

You are doing something unexpected with the tests. No tests have
been run. You should use the inline_tests_runner script to run
tests.

The README mentions this here: GitHub - janestreet/ppx_inline_test: Syntax extension for writing in-line tests in ocaml code

But the best I could get from the test executable was:

(run /day_01/.puzzle_01.inline-tests/inline_test_runner_puzzle_01.exe inline-test-runner Puzzle_01)

This prints nothing.

I can verify by passing the -show-counts option to the test:

0 tests ran, 0 test_modules ran

Maybe I am missing something here? There also seems to be an issue open in
regards to running a single test, not sure if related to this: running a single test · Issue #34 · janestreet/ppx_inline_test · GitHub

So in the end, dune build @1 @runtest -w is the closest I’ve gotten. But I
don’t feel like it’s good enough. I can do better. I would like the runtest
alias only run the tests in the day_01 folder.

Thank you so much for taking your time to read this! I realize now that I
should’ve asked for help much earlier but on the flip side I’ve learned a lot
about dune! Am I even approaching this problem from the right angle?

[Release-Candidate]

That’s because the name of the library is wrong, it doesn’t contain tests. You can list the tests by passing -list-test-names to the runner. And you always want to pass -verbose to the runners.
What does

dune exec INLINE_TEST_RUNNER_PATH/inline_test_runner_puzzle_01.exe inline-test-runner Puzzle_01 -verbose

do? And why isn’t your runner in _build/?

And dune test -- ARGS only passes ARGS to the “normal” test runners, not the inline test runners.

I personally use the Alcotest inline tests, their runner is also less, well, of a PITA. Same for Dune’s locking.

[p1xelHer0]

I believe my test runner IS in the build directory, it’s just not very obvious when reading the dune action

From User Actions — Dune documentation :

User actions are always run from the same sub-directory of the current build context as the dune file they are defined in, so for instance, an action defined in src/foo/dune will be run from $build/<context>/src/foo.

Since this is a top-level dune file

(run ./day_01/.puzzle_01.inline-tests/inline_test_runner_puzzle_01.exe inline-test-runner Puzzle_01)

refers to the path

./_build/default/day_01/.puzzle_01.inline-tests/inline_test_runner_puzzle_01.exe inline-test-runner Puzzle_01

Nonetheless I took you advice and tried using Alcotest and the inline Alcotest PPX and… it works!

(rule
 ; Aliases in Dune are referred to with the @ sign.
 ; This means this alias below is `@1`.
 ; We can run this alias with `dune build @1`.
 (alias 1)
 ; Dependencies for the alias
 ; Run runner needs the input file to run the real puzzle
 ; Also the pretty printer where we strip some parts of the alcotest output
 (deps ./day_01/input ./bin/pp.exe)
 ; This alias is an action - we want to do something
 (action
  ; Progn runs multiple processes in sequence
  (progn
   ; First we build day_01
   (run dune build ./day_01)
   ; Pipe the result of the inline test to `pp.exe`
   (pipe-stdout
    (with-accepted-exit-codes
     (or 0 1)
     ; Run inline tests for day_01
     (run ./day_01/.puzzle_01.inline-tests/inline_test_runner_puzzle_01.exe))
    ; Filter some output of the test
    (run ./bin/pp.exe))
   ; Run the day_01 program itself
   (run ./day_01/runner.exe))))

This way I can run dune build @1 -fw and continously see the inline test followed by the result of the actual program!

The only issue I am getting now is after doing a dune build running the said alias (which only runs things in the ./day_01 folder instead of the whole project results in

Warning: No dune-project file has been found. A default one is assumed but
the project might break when dune is upgraded. Please create a dune-project
file.
Hint: generate the project file with: $ dune init project <name>
Error: Multiple rules generated for _build/default/aoc.dune-package:
- file present in source tree
- <internal location>
Hint: rm -f aoc.dune-package

But that is a separate issue tied to the project setup I guess. Happy with any pointers and thanks for the Alcotest tip!

[benjamin-thomas]

For such simple setup, you can build yourself an adhoc build/run pipeline with an external tool, see:

Running tests when using dune build watch-mode

An alternative to running dune in watch mode is to use entr to create an ad hoc builder/watcher. Something like this: find ./lib/ -name "*.ml" | entr -c bash -c 'dune build && echo OK' This won’t lock any files, so you can reload your dune utop process more easily that way.

Hope that’s clear enough

[emillon]

@p1xelHer0

The idea here is that each day I, or some other user, would start a

dune exec ./day_01/runner.exe -w

[…]

To get immediate feedback on the test of the example data and the output of the written program.

I think that this what’s causing this difficulty. I would recommend instead making that “high-level” check a test in the dune sense (= something attached to the runtest alias).

Here’s how I do AOC. Same kind of approach: I have inline tests in the library part of each solution, and a run executable (well it’s inlined in the dune file but that’s the same thing).

The interesting part happens here:

(rule
 (with-stdout-to
  p1.out
  (run ./run.exe %{dep:input.txt})))

(rule
 (alias runtest)
 (action
  (diff p1.expected p1.out)))

This is a useful pattern to learn with dune:

• one rule explains how to generate p1.out. This file is just going to live in the _build directory.
• one rule explains that p1.expected (in your source tree - create it empty first) should get compared to p1.out and offered for promotion. This means that whenever there’s a difference, the diff is printed and you can “accept” it (copy p1.out to p1.expected) by running dune promote. This is explained here: Diffing and Promotion — Dune documentation.

The dev process looks like the following:

• I start with an empty p1.expected and a run function that does nothing. This means that dune runtest succeeds.
• I then write the solution iteratively using inline tests. dune runtest helps me check the library, and dune promote replaces the [%expect] blocks. p1.expected stays empty.
• when it’s time to try the whole thing, I implement run, usually as just a pipeline of of read stdin |> main_logic |> printf "%d\n" . Dune will execute this with input.txt, create p1.out, and display the diff (which is just the result, since p1.expected is empty).

Using this, you can keep the short iteration loop that dune runtest -w provides, while keeping operating at two levels at once: your unit tests with inline tests, and your integration tests for part 1 and part 2 with dune rules. You never have to interrupt the dune process or tinker with low-level aspects of how inline tests are executed.

[p1xelHer0]

Thank you, this is great feedback! I’ll take a look at your repo.