Large repository of real dune files, to grep for examples

Ecosystem
1

Is there a place in which a large number of real dune files are easily accessible, so that one can grep them for examples? Namely for details which are not necessarily written out in the docs.

For instance, opam-repository is ideal for grepping opam files and finding out how real people solve some issues, which syntax they use, etc. I often feel the need for a similar thing for dune files, so that I can quickly check if a given expression is meaningful (or better: finding a similar, but actually valid, expression).

2

Iā€™ve used Github search for this in the past: Code search results Ā· GitHub (But Iā€™m not sure how stable/reliable it is anymore).

https://sherlocode.com is a good alternative for OCaml, but it does not index dune files, as far as Iā€™m aware.

2 Likes
3

https://sherlocode.com is a good alternative for OCaml, but it does not index dune files, as far as Iā€™m aware.

This is impressive because it is extremely fast. But how are people using it? It implements string search and that is often a bit too granular for conceptual problems.

4

You could also use opam-grep. It downloads every package in OPAM and searches within that set.

1 Like
5

opam-grep is quite limited (it only says ā€œpackage foo matchesā€ or "package foo doesnā€™t match), and slow. At least assuming it hasnā€™t changed in recent times. I had to write another tool instead. E.g. to check how people use the preprocess field in the opam packages of the current switch:

$ time _build/default/bin/opam_search.exe -x 1 $(opam list --short | while read p; do echo -p $p; done) -- grep -r --include 'dune' 'preprocess' -A 1 | head
alcotest:src/alcotest-stdlib-ext/dune: (preprocess future_syntax))
alcotest:--
alcotest:src/alcotest-engine/dune: (preprocess future_syntax))
alcotest:--
alcotest:src/alcotest/dune: (preprocess future_syntax))
angstrom:lib_test/dune: (preprocess
angstrom:lib_test/dune-  (per_module
angstrom:--
angstrom:lib/dune: (preprocess future_syntax))
base:generate/dune: (preprocess no_preprocessing))

real	0m0,849s
user	0m0,937s
sys	0m0,655s
6

Thanks! I didnā€™t know about opam grep.

I tried a bit, and I must agree with @v-gb that it is somewhat slow, and the matching is not very informative, but just to clarify, it does search every file in the package, right? And thereā€™s no option to filter by file name (e.g. rg --glob)? Because then I could include only dune and dune-project files and it would be perfect!

7

I just cloned opam-search and ran dune build, and after 3 minutes compiling (I thought it was looping, since there was no specific output during this time), it produced some warnings:

File "time_float_unix/time_unix/time_unix.ml", line 1, characters 4-14:
1 | [@@@deprecated "[since 2022-04] Use [Time_float_unix] instead"]
        ^^^^^^^^^^
Warning 53 [misplaced-attribute]: the "deprecated" attribute cannot appear in this context
File "bigstring_unix/src/dune", line 4, characters 9-29:
4 |   (names bigstring_unix_stubs recvmmsg))
             ^^^^^^^^^^^^^^^^^^^^

and some errors:

/home/user/.cache/dune/toolchains/ocaml-compiler.5.3.0-ca92397a8f2c8e38e16694f64da85045/target/lib/ocaml/caml/platform.h: In function ā€˜caml_plat_latch_is_releasedā€™:
/home/user/.cache/dune/toolchains/ocaml-compiler.5.3.0-ca92397a8f2c8e38e16694f64da85045/target/lib/ocaml/caml/platform.h:222:10: error: implicit declaration of function ā€˜atomic_load_acquireā€™ [-Wimplicit-function-declaration]
  222 |   return atomic_load_acquire(&latch->value) == Latch_released;
...
error: implicit declaration of function ā€˜atomic_store_releaseā€™
...

Should I report these as issues in Github?

Otherwise it seems very useful for what I want, thanks!

8

It downloads and unpacks the whole repository, you can take that location and use it with e.g. ripgrep

2 Likes
9

Just the dune files from everything in Opam is probably not a lot of data. So a script that looks just at code hosted on GitHub (for simplicity) and downloads all dune files into a hierarchy in the local file system is probably not out of reach. Sounds like a job for an AI.

10

The first dune build would build dependencies, so yeah, itā€™d take a bit of time. But dune provides progress, so not sure why you werenā€™t seeing any.
For the build errors, thatā€™s surprising considering the lock file. Letā€™s discuss this in an issue, yes. Thanks!

1 Like
11

The regular releases only do so when started with --display short (starting with 3.21). The preview used to display this by default, the nightly preview does not.

1 Like
12

In a nearby future, Software Heritage should hopefully allow such kinds of tasks to be performed without need for an AI. And with the advantage of having many more repositories than Github (e.g. self-hosted Gitlab instances).

1 Like
13

Indeed, and it has a copy of everything on the opam repository.