[ANN] opam-grep: search through the sources of all the packages in opam-repository

Ecosystem
,
1

Hi!

opam-grep is an opam plugin that greps through the sources of all opam packages.

To install it, simply call:

$ opam install opam-grep

Then to use it, simply call:

$ opam grep "your regexp"

Side note: currently opam-grep will cache the sources in your cache directory (e.g. ~/.cache/opam-grep), so a few GB of available disk space is most likely required.

I hope this can help core compiler and community library devs alike, to know which part of their software is used in the wild.

20 Likes
2

Wow! Looking at what sort of code is used in the real world can be useful, for instance there were some surprises about the amount of code that uses asynchronous exceptions in the wild (and more surprises in a couple of other occasions). I can definitely recommend looking at real-world code, and I hope this will make it easier for compiler/library developers.

In my experience it was necessary to prune repetitive bits of source packages (e.g. vendored copies of ocaml headers or ocaml runtime) to eliminate noise, and I also like to have a stable copy to keep as a reference (later on this will also allow me to compare evolution over time). For the record one can download the sources as follows:

for pkg in $(opam list -s --all); do opam source $pkg; done

But I wonder if the plug-in could do it better in some way (especially in omitting irrelevant parts).

3

I’ve just released opam-grep.0.2.0 with quite a bit of change compared to the previous version. Here is the highlight:

  • Complete rewrite from shell script to OCaml, making it more portable
  • Use the faster ripgrep and ugrep over grep when available (suggestion by @Engil)
  • Use the progress library to show progress instead of a non-portable/DIY spinner

See the changelog for the full list of relevant changes.

Big thanks to @CraigFe for the progress library (such a treat!) and to @dbuenzli for bos and cmdliner in particular, making it easy to do such rewrite :relaxed:

10 Likes
4

One interesting thing I noticed about this opam plugin is that there is no opam integration code anywhere! All that I could find is a find is a flags: plugin line in the opam-grep.opam file.

Is this how opam knows that you need to invoke this program when you type opam grep at the command line?

5

Yes exactly. flags: plugin basically makes opam symlink the opam-<pkg> binary installed by the package to $OPAMROOT/plugins/bin.

In turn, when opam <unknown-subcommand> is called, opam will look in this directory and call the corresponding plugin with the same parameters, if it exists.

3 Likes