Help with cryptic dependency cycle compile errors using `dune build`

Learning
, ,
1

Hi all,
I am working on writing a library for large chunked & compressed multidimensional arrays. I was working on a adding a feature to support sharding of array “chunks”. The way this works is that sharding further splits array chunks into sub-chunks that can be stored in a single “shard”. These sub-chunks have a set of codecs that are applied to them to compress them before storing them in a shard. The way the sharding process is defined requires one to implement the functionality in a recursive manner: see here. This image kind of illustrates what is required .

I tried to implement this feature using recursive modules here: ... · zoj613/zarr-ml@9cd411d · GitHub but this failed to compile due to cyclic dependencies. I tried to debug but came up short. So as a second attempt I removed the recursive module definitions and relied on mutually recursive functions instead: see: Blaming zarr-ml/lib/codecs/array_to_bytes.ml at 1176628bd4f48c1aad5b06b2c74472ce9de49800 · zoj613/zarr-ml · GitHub , but this too wouldn’t compile due to some cyclic dependencies. I am still not able to pinpoint where this occurs given the error message

$ dune build
Error: dependency cycle between modules in _build/default/lib:
   Storage
-> Array_to_bytes
-> Array_to_bytes
-> required by _build/default/lib/zarr.a
-> required by alias lib/all
-> required by alias default

I spoke to @shonfeder and they recommended I open a github issue on the dune project since it is not clear why I keep getting this compile error; but I wanted to read what others think before I do.

1 Like
2

Indeed! First, I cannot figure out why Array_to_bytes is determined to have a cyclical dependency on itself: nothing in the modules itself should entail this afaict. Second, as a usability issue, I would expect that any detected dependency cycle should come with clear, legible instructions that indicate how to resolve it. That is not the case here, so this looks to me like a UI but at least.

That said, I expect more some with deeper dune and compiler expertise should be able to help diagnose here.

@zoj613 if you get a chance, trying to reduce your example to a minimal reproduction, by systematically cutting out parts of the project (e.g., removing nested dirs, removing modules that aren’t involved in the cycle) should help narrow down the problem.

3

If we look at _build/default/lib/.zarr.objs/zarr__Array_to_bytes.impl.all-deps, we can see that zarr__Array_to_bytes is in there; that’s not quite right.

After digging a bit, the cycle seems to be (or rather one cycle is):

  • in array_to_bytes.ml, Array_to_bytes depends on Metadata
  • in metadata.mli, Metadata depends on Codecs
  • in codecs.mli, Codecs depends on Array_to_bytes

This kind of cycle is not supported.
I think that’s equivalent to a library with modules A and B arranged like this:

(* a.mli *)
type t = int
val x:t
(* a.ml *)
type t = int
let (x:B.t) = 0
(* b.mli *)
type t = A.t
(* b.ml *)
type t = A.t
1 Like
4

This is not supported by dune, but the compiler can compile this kind of spurious cycle. Indeed, what is needed for the compiler is that:

  • there are no dependency cycles between interfaces (to compile cmi files)
  • there are no dependency cycles between implementations (to compile cm{o,x] files).

Cycles that appear when merging interfaces and implementations as a single node in the dependency digraph are not a problem.

1 Like
5

Should dune be able to report these actual steps to the user? If so, I can open an issue to request more detail in the cycle detection output.

6

It’s a bit difficult to solve. We’re aware of that, see for example Cycles reported by dune are cryptic · Issue #2818 · ocaml/dune · GitHub. codept might help; adding locations to ocamldep info would be useful as well.

2 Likes
7

Ah, great. Sorry I missed that issue when searching before, I think I was too narrowly focused on the specific error message. Thanks!

8

Note that even in codept this specific case would require a new warning for transitive dependencies in interfaces that don’t exist at the implementation level.

1 Like
9

Thank you for this reply. This actually helped me realize that the cycle is caused by using a sub-module of Metadata in both Metadata and in Array_to_bytes. Moving the re-usable pieces of code inside the Metadata module into their own standalone module got rid of the dependancy cycle. See this commit: ... · zoj613/zarr-ml@8980748 · GitHub

2 Likes