Local open seems to confuse Dune's dependency cycle detector

Learning
1

Note: Seeing as people have complained about this before, just to be clear, I’m not trying to suggest or accuse any software in the OCaml ecosystem of being implemented incorrectly, or doing something wrong. I have just run into a problem, and am asking for advice.

When using a local open that introduces a module with the same name as a module in the project, dune complains of a non-existent dependency cycle.

Example, consider a project with 3 files:

  • database.ml
module  Actor = struct 
    let version = 10
end
  • helper.ml (depends on database.ml):
let project_version = Database.(Actor.version + 10)
  • actor.ml (depends on helper.ml):
let actor_version = Helper.project_version + 20

When I try to build this project using dune, it complains about a dependency cycle:

Error: Dependency cycle between:
   _build/default/failure.ml/.main.eobjs/dune__exe__Helper.impl.all-deps
-> _build/default/failure.ml/.main.eobjs/dune__exe__Actor.impl.all-deps
-> _build/default/failure.ml/.main.eobjs/dune__exe__Helper.impl.all-deps

If, for example, I replace the contents of helper.ml with the following (removing the parenthesis):

let project_version = Database.Actor.version + 10

The code compiles correctly.

Similarly, if I modify the contents of actor.ml as follows:

let actor_version = 20 (* no dependency on helper *)

The project compiles again, so the issue (again, not trying to claim that any software is implemented incorrectly, just having difficulty compiling my project) really does seem to be due to dune, not some peculiarity of the semantics of local opens.

I was thinking of submitting an issue (not suggesting that it has been implemented incorrectly, just if it turns out to indeed be a bug (which to be clear, I am not claiming)) to the dune git forge, but wasn’t sure if this was my own mistake.

Does anyone know if this is indeed a bug, or a mistake on my part?

1 Like
2

The issue is not really local open but the interaction between open, module name collision and the local nature of ocamldep analysis. Your example can be simplified to

open Database
open Actor

The issue is that without knowing the module type of Database, open Actor can either refer to the compilation unit Actor or an eventual Database.Actor submodule. And when computing dependency, we don’t know yet the module type of Database since we don’t know yet how to compile Database. The standard dependency ocamldep tool resolves this issue by computing an over-approximation of the dependencies of a compilation unit using only local information.
In our example, from the information at hand locally, the module Actor in

open Database
open Actor

might be the compilation unit Actor, thus ocamldep counts this open Actor as an Actor dependency.

The advantage of this approximation is that ocamldep might compute some spurious additional dependency but it is not missing dependencies (excepted in some known corner case).

It is possible to resolve (nearly) exactly module dependency with a project-wide analysis, this is the aim of my codept tool. But in practice, there has been limited interest in integrating codept to dune.

7 Likes
3

To emphasize: all build systems building on top of ocamldep (a simple, heuristic dependency analyzer) have this issue. @octachron’s codept project solves it nicely, but build system authors so far have not bothered looking into this, because it’s work and ocamldep works most of the time. You should ping your build system developers to encourage them!

1 Like
4

Alternatively we could ping upstream developers to encourage them to integrate the better dependency analyser in ocamldep :–)

5

Would that be the right move?

I think technically it would doable. codept depends on (dune, menhir, ocaml), we could write a Makefile and use the same store-generated-parser trick as with the ocaml parser. But in general we are trying to move stuff out of the compiler distribution, not add more stuff to maintain.

Besides, while codept can work as a drop-in replacement for ocamldep, if I remember correctly it also provides additional usage modes that require explicit actions from the build system maintainers. So (again iirc.) we need build-system support anyway to get the full benefits.

6

It depends a lot of what is hidden behind the term drop-in replacement of ocamldep. For a single file, codept and ocamldep mostly have the same output, codept will only emit warning message as soon as it needs to make an approximation for an unknown module.

The core issue is that in order to compute accurate dependency in OCaml, it is necessary to admit that dependency resolution depends on “module-level” outline in OCaml. And that property breaks the simple parallel model of computing one dependency file by source file. Layering the computation of dependencies in the right way to reach fully accurate dependency requires a non-trivial amount of work at the build system level.

7

For the end users yes.

Consider that if your project only builds with codept then you have to instruct users that they need to install/depend on that third party tool.

Consider that people will rather use what is here by default and minimize their dependencies, i.e. they will use ocamldep. Then when it fails in confusing ways instruct them to use codept ? Great user experience.

Multiplying the paths in build procedures also opens to more build variability and general build mess.

I think it would be nice if upstream could also consider moving stuff in the compiler distribution. Seing that only from the “stuff to maintain” perspective is a bit shortsighted in my opinion. Thinking more about the user experience of the language would be a good thing.

The general tendency of upstream to offload problems on build systems is problematic (e.g. the terrible namespacing hack implemented by dune which then creeps everywhere down to the documentation tool. When you add up all the work that is needed to support all this I’m not sure it would have cost much more to have a good solution provided by upstream).

Would you expect a language to provide a reasonable source-level dependency discovery tool ? A reasonable namespace and library management story ? A reasonable API documentation tool ? My answer to these questions it three time yes and by reasonable I mean that is able to handle and evolve with the language it is supposed to support.

I recently onboarded a newcomer to the eco-system. She just had tried a few other alternative languages out there just before, she told me it was from far the worst experience she had – and don’t tell me dune is the solution to these problems, that’s part of what she was using and dune’s experience itself is utterly confusing in many aspects.

4 Likes
8

Off-topic, but I couldn’t find this information in the project README: how dependant is codept on a particular OCaml version?

Also, have there been any concrete experiments or work on dune integration?

Cheers,
Nicolas

9

Essentially, the main dependency is on the parsetree : the first step of codept’s pipeline is to reduce the OCaml parsetree into its own very simplified AST. This ast converter needs to be slightly tweaked for each OCaml version with a few conditional macros.

However, once this conversion is done, codept is mostly independent of the OCaml version.

Infrequently (once every few version?), a new module level feature of OCaml needs to be ported to codept too. The last features at this level were the with module type constraint and the generalized open struct ... end for instance. However, since features are monotonic on the OCaml side, codept can just support a superset of all module-level feature of all known OCaml version.

None, that I am aware of? I haven’t found the continuous block of time to truly look at this integration (note that I do include myself in my “limited interest” remark).