How to avoid having a bunch of opam files at the project root?

Ecosystem
,
1

Hello,

I must admit I haven’t looked deeply into the issue but I didn’t find an obvious solution, so hopefully, a solution will be posted here.

Problem: Our growing OCaml project has many opam files at the project root, and most of them are empty. This clutters the view in GitHub and elsewhere.

~/semgrep $ wc -l *.opam
  47 ast_generic.opam
  35 commons.opam
   0 graph_code.opam
  27 lib_parsing.opam
   0 lib_parsing_tree_sitter.opam
   0 ograph.opam
   0 ojsonnet.opam
   0 osemgrep_targeting.opam
   0 otarzan.opam
   0 parser_bash.opam
   0 parser_c.opam
   0 parser_cpp.opam
   0 parser_csharp.opam
   0 parser_dockerfile.opam
   0 parser_elixir.opam
   0 parser_go.opam
   0 parser_java.opam
   0 parser_javascript.opam
   0 parser_jsonnet.opam
   0 parser_json.opam
   0 parser_julia.opam
   0 parser_kotlin.opam
   0 parser_lisp.opam
   0 parser_lua.opam
   0 parser_ocaml.opam
   0 parser_php.opam
   0 parser_python.opam
   0 parser_regexp.opam
   0 parser_r.opam
   0 parser_ruby.opam
   0 parser_rust.opam
   0 parser_scala.opam
   0 parser_solidity.opam
   0 parser_swift.opam
   0 parser_terraform.opam
   0 parser_typescript.opam
   0 pfff-lang_GENERIC-analyze.opam
   0 pfff-lang_GENERIC-naming.opam
  24 process_limits.opam
  25 profiling.opam
  65 semgrep.opam
  29 spacegrep.opam
   0 tree-sitter-lang.opam
   0 tree-sitter-to-generic.opam
 252 total

The opam files were created because dune requires such a file for any local library that’s declared as public. Having an empty file is sufficient for the project to build. Maybe we shouldn’t leave them empty, but this doesn’t solve the problem. We’d like to hide them from view in one way or another. Being able to keep them in an opam/ folder or similar would be great. Is there such a solution already? Is it more of an opam issue or a dune issue?

2 Likes
2

It’s more a dune issue, opam supports lookups in an opam/ directory at the root of your project, see Pinning.

1 Like
3

Thanks, @dbuenzli. I thought I remembered something about an opam/ folder. I opened a dune issue: Support opam/ folder to declutter root folder from *.opam files · Issue #7329 · ocaml/dune · GitHub

1 Like
4

You can also declare the opam packagea in the dune-project file. See what done for tezos dune-project · master · Tezos / tezos · GitLab.

3 Likes
5

@hhugo is right. To expand a bit:

That part is not quite right. The mechanism is:

  • public libraries and executables need to be attached to a package
  • packages are declared either by the presence of a .opam file in the root, or by (package) in dune-project
  • (this does not apply if you’re not interested in opam) if you’re using the latter, you can in addition set (generate_opam_files) in dune-project and the opam files will be generated from the metadata in dune-project

In a nutshell, you can delete parser_rust.opam, add (package (name parser_rust)) in dune-project and you’ll be done.

To go further: you don’t have to map libraries 1:1 to packages. For example, if you set (public_name parser.rust), this will go in the parser package. If you do the same for parser.swift, this means you’ll have a parser package that contains the parser.rust and parser.swift libraries.
It can be a good idea or not depending on the situation.

8 Likes
6

To develop emillon’s last point (which is the solution I use):

  • Make each of your libraries a sub-library:
;; This is the dune file of library solidity
(library
 (name solidity)
 (public_name parser.solidity)
 (...)
)

  • Then, you only need a parser.opam file at the project root.

  • In the outside world, use

open Parser
open Solidity
3 Likes
7

Thanks @hhugo and @emillon - sorry I just saw your replies today. Here’s my translation work: Move the contents of the *.opam files to dune-project. by mjambon · Pull Request #7651 · semgrep/semgrep · GitHub

The remaining issue is of installing the opam dependencies by extracting them from the package definitions. We’ve been using:

opam install --deps-only .

How to achieve the same effect without having semgrep.opam and other opam files in git? The documentation says dune build will create the opam files, but we don’t want and we can’t build the project before the dependencies are installed. Is there a dune subcommand that will only generate the opam files?

Eventually, we want to install specific versions of packages as produced by opam lock (creates foo.opam.locked from foo.opam, both being in the same format). According to @Khady 's post from 2018, it’s done with

opam install --deps-only --locked .

and it works nicely. The issue with lock files is that they must be kept under version control. Opam format, dune format, or something else doesn’t matter to me, of course, but we want either just one such lock file or have many lock files buried in a subfolder out of sight.

8

Hello Didier! Long time no see.
Did you also write a Solidity parser?
We have one in Semgrep too.

9

Absolutely not. This was only an example. (Although, I am interested in how to write compact user-friendly parsers).

10

There is one there too:

11

Ah nice Fabrice. We should have maybe used it instead of using tree-sitter-solidity.
We should join forced to check solidity! People have written a few semgrep rules for it: