Dune: serious bug found in the design of library variants

If we move to a proposal where the set of variants of a virtual library is static and declared upfront, we don’t have such a problem. It seems that this is what we are converging towards. Once this feature is working well and er have feedback from usage, we can think whether it is worth extending it to support an open scheme.

I understand, and it seems that this is the fix and the solution you described in your original post : one can only define a variant in the same dune project than the virtual library.

What I was trying to say is that if you extend variant to an open scheme then we will need cooperation from opam and that, in this case, the dependency solver will solve the conflict problem that you describe : if two packages provide the same variant then they conflict.

If we compare with modules and functors, when you use variant you end up with implicit arguments for the functor and you need a unique concrete implementation that is tagged as implicit to avoid ambiguity. As long as these implicits are defined locally (in the same project), dune can ensure the unicity but as soon as you’ll try to define them globally (in all the opam universe) I guess that you will need help from opam.

By the way, this is not clear to me if this open scheme for variant has some interest : if I understand correctly, variants are use to avoid boilerplate in the dune file; but with a hypothetically open scheme this boilerplate should be put in the .opam file. This is not clear to me if there is some gain compare to mention explicitly the concrete implementation we rely on.

That’s a good point.

Please consider the use case in UI/Application development where people will wish to implement a set of core application functionality which must be implemented in a “backend” for each platform. In that case, you cannot anticipate the number of backends, and the people adding backends for one platform will not necessarily be coordinating with the people who add backends for another platform (because it’s rare that people have expertise across all the platforms, or sufficient time to be able to implement all of them).

So we do want the ability for library authors to implement a virtual library, and then at the top level executable/package, select the concrete implementations for the platform they are targeting, and I was convinced that this does become burdensome to list out all of the concrete implementations for a given backend. Those concrete implementations even for a given platform/backend may be implemented by several different people in different packages. Variants supposedly addressed the burden of selecting them all because it allowed your dependencies to tag concrete implementations and the top level app can just select whole swaths of them by specifying tags.

Is this use case still being considered for variants?

It still is the final goal I believe where, in the other side, MirageOS has the same require: provide different implementations (for different systems) under the same interface. I just would like to say that we already have many ways to do that (linking trick, functor or abstraction by values).

As Rudi said:

The issue discussed in this thread is only relevant to variants. There are no issues with implementations and virtual libraries spanning multiple packages.

This is all a bit of our fault for choosing such confusing terminology, but I want to re-iterate that the issues are only relevant to the tagging system we call “variants”.

dune will add a proper way specially about the linking trick (currently used by ptime or digestif) where it will compute the default implementation at the linking time.Finally, as the last goal, dune will compute again this default implementation across several packages - which it is not the case currently.

2 Likes

Just to conclude this discussion: library variants are a new thing that didn’t really exist before and we don’t know yet how they are going to be used or as @dinosaure puts it, their impact on the OCaml ecosystem. At this point, we simply need more data point.

However, as @rgrinberg rightly pointed out, variants are nothing more than a convenience layer on top of virtual libraries. Virtual libraries are the real basis of this system, they are fully supported by Dune and are not going away anytime soon.

So please try virtual libraries, use them in real projects and report your experience here or on the Dune github project as @Drup did. This will make this discussion a whole lot easier. Until then, library variants will remain an experimental 0.x feature, meaning that we are not committed to support it long term and might drop support for it without bumping the major version of dune.

Thanks you for your cooperation

5 Likes

Sure thing! I have been thoroughly enjoying virtual libraries.

1 Like