Upcoming Dune Feature: (include_subdirs qualified)

The trouble is that mod.ml is a valid module name already and we don’t want to take that away from people.

There’s some explorations but I believe it’s still in research. Rehabilitating Packs using Functors and Recursivity, part 1. | OCamlPro

that’s why I said “file modules” (not sure if there’s a better name)

I’m not sure I understand the issue there. You mean it wouldn’t be backward compatible with projects that already have files called mod.ml? Or it would prevent people from calling their files mod.ml? If the latter, then I think it’s a negligible issue (at least Rust has proven that this is not an issue in practice).

The issue is that it would prevent users naming their modules Mod. Yes, it’s negligible, but what’s the point of imposing this pointless inconvenience when there’s other options (such as _module_) without this downside. Copying rust is not a convincing reason to me.

1 Like

What about being able to write ocaml if it’s an .ml file? In order to export modules part of this folder module, or even have code

Thanks to everyone who participated in the discussion. I’ve decided on the following course of action:

  1. Use the convention that the interface module must be inside the directory and should be named after the directory to be the interface module. E.g. foo/foo.ml is the interface module for Foo. This isn’t the greatest, but it’s the path of least resistance for now. In dune file, specifying the interface module will use the module path of the group rather than the interface module. E.g. (private_modules foo) rather than (private_modules foo.foo).

  2. We’ll add a mode that allows users to switch to a convention based naming scheme. For example, _module_.ml would correspond to the interface module for both (include_subdirs qualified) and wrapped libraries.

  3. If we end up liking the above convention, we’ll switch it to be the default in 4.0.

5 Likes

Once again, I find that @dbuenzli has asked the first question that sprang to my mind when I just read the proposal a moment ago.

My additional observation is that— if the Entmoot upstream hasn’t figured out how to do this cleanly yet and they are still deliberating over proposals (as the Ent folk do)— maybe “solving” the problem with a hasty patch to the dominant build tool is more likely to entail long term commitments to investments in regrettable decisions than not.

As a parallel, consider the mess we are currently in with Findlib packages, which were similarly introduced by a community that wanted to move fast and break things while the upstream compiler team balked at proposals to add a new level of encapsulation above the module level while they were still trying to sort out the theoretical implications of modules as first class values.

When the Entmoot upstream introduced the -no-alias-deps flags In June of 2015, surely they intended us mortals to use it?

In addition, the feature has been first proposed by somebody in fairly close contact with the Entmoot around 5 years ago, and also independently proposed and implemented by others around that time. Hastily designed it was not.

Although I admit that my patch might have been hastily written over the course of a few months, I still promise not to break anything for anyone deciding to opt out of using this feature (exactly as we’ve done for our existing uses of -no-alias-deps). If the Entmoot ever give me something superior to -no-alias-deps, I will surely add support for it and encourage everyone to migrate to whatever the blessed solution is.

Tbh, I’m of a different opinion about this story. Findlib was the innovator that showed us that the concept of a library is valuable and deserves to live in the compiler. I’m of the impression that the Ent folks are dragging their feet with catching up the compiler to the rest of the ecosystem in this case.

2 Likes

My impression is that it was a case of the compiler team introducing a minimal surface change to the toolchain to promote experimentation with various workarounds to the lack of a proper theory about library packaging. I think they expected the community to experiment with it, in the hopes of mitigating the risk of the community developing even worse workarounds, and having those metastasize into an even deeper tarpit.

I’m not worried about you breaking anything. I’m worried about you building further obstacles to rolling out a proper fix once we have well considered theory of library packaging for the compiler team to implement.

Once again, I probably have a less charitable view of Findlib here than you do. Probably also a more forgiving view of the compiler team. I don’t think they needed to see Findlib happen in order to recognize that the concept of a library packaging system is valuable and the language and the compiler should offer some support for it. They’re very smart people. I think they understood well that the problem is a pain point for practitioners. I think they just had other more pressing problems to solve at the time, and they prioritized accordingly.

I think the parallel with Findlib is apt, because what we really needed at the time was not ocamlfind but rather the combination of a decent application packager, i.e. opam, and some vastly improved build tooling, i.e. … well, I wish I could say dune is the tool we needed, but I’ll admit it’s the best tool we have at the moment, although, as I’ve said, I do not very much love it. Instead of that, we got ocamlfind and the conceptual mess that came with it, and here we are struggling to explain it to language newcomers who don’t have to put up with that kind of nonsense in competing languages like Rust (because their compiler developers made other, different, more awful mistakes, rather than this one).

Circling back to Daniel’s question, my worry here is that by putting more stones down on the board to reinforce the -no-alias-deps position, we may be building more under-planned infrastructure that will make the practical problems of adopting a properly intergrated system of library packaging directly into the compiler more difficult, not less.

Don’t get me wrong. I love Rome. There’s a reason that city is in the background of the image that I use here on the forums. Having family there and visiting them often has taught me to be mindful of the costs of not investing enough in the planning of your civil environment. That’s the feedback I’m trying to offer here.

If we wait for the compiler to solve all the problems, we’ll never get them solved. It moves far too slowly.

In this case, dune is simply following the precedent set by other languages. That’s a good place to be – there’s a lot of experience in this domain already and there’s no reason to lag so far behind other communities. Every modern language has headed in the direction of opinionated file layouts because it saves a lot of programmer time and headaches. Every modern language has realized that directory structure is a natural expression for module hierarchy.

3 Likes

This sounds a lot like the rather cynical things I used to say about OCaml not having a multicore runtime environment. I don’t say those things anymore, because the team has now delivered a multicore runtime environment that I firmly believe was worth the wait to get it done right.

1 Like

And if you never try to solve them in the compiler, they will never get solved.

Even worse, these eco-system matters have latency measured in years. When I wrote the library linking proposal in 2019 I expected that it would take between 5 to 10 years for the eco-system to fully pick it up and liberate Gerd from maintaining findlib – and thus for people to have an easier, cruft free, system to comprehend and work with[1].

It would also be interesting at some point to consider the derived cost that half baked solutions like -no-alias-deps have had on the whole system (short paths, dubious convention like __, special support needed in tooling like odoc, etc.)

That’s not true anymore, we get something like two releases per year often accompanied with non-trivial features and enhancements. A lot of old timers have never recovered from the culture of workaround that was the only solution when compiler development was almost brought to a halt at the end of the 2000 decade (under the fair response from people maintaining it that this work couldn’t be justified to their employer).

Rather it’s a matter of putting the energy at the right place regardless of political goals.

A valid criticism could be that it’s much harder. But that’s because upstream is, luckily for us, mindful of quality and long term maintenance.


  1. dune people try to make you think you don’t need to know about findlib. But it’s quick to come back e.g. as the toplevel loading mechanism. So eventually the mess has to be discovered by users. ↩︎

4 Likes

I think the core team is doing great work given the resources they have, but even they admit the compiler advances slowly due to a lack of resources. Anything theoretical in particular can (and does) get stuck in limbo for many years.

The multicore work involves some theory, but it’s largely an engineering advance, and even accomplishing it required the team obtaining a specialist in multithreading for compilers (@kayceesrk). Notice though, that while we have multicore, we do not have typed effects, which are a theoretical construct.

Theory-wise, the core team is strapped for able bodies who can develop things further. Observing the team over time, we see that the main theoretical advances happen close to ICFP, with papers being published and ideas being put out, only to languish underdeveloped in theory limbo. By the time of the next ICFP, old ideas are often forgotten.

I am reminded again of the old joke about the differences in technology culture on both sides of the Atlantic ocean: the left side always pushes against you with “I see that the theory seems sound, but how does it hold up in practice?” and the right side retorts with “I see that it seems to work in practice, but is the theory really sound?”

Being personally torn apart by both sides of the Atlantic cultural divide, this is endlessly amusing to me. On the whole, my impression is that the OCaml community really does manage to straddle the divide exceedingly well. Compared to others, which I will not name out of politeness.

2 Likes

I agree that dune is making decisions that are much closer to “designing new surface features for the language” than most other tools in the ecosystem and other build systems in the past. In practice, today there is little discussion between dune developers and OCaml developers on this front, probably because everybody is already so busy dealing with their own problems.

(To give a concrete example, I find it slightly strange that Dune unilaterally made decisions about which groups of warnings to enable by default and turn into errors, basically erasing dozens of warning-specific discussions and debate by the compiler folks upstream, without to my knowledge even considering to ask upstream for an opinion. But I’m not sure it is bad; in many cases the choices made were more daring than the compatibility-focused consensus built by the upstream developers, and we are probably better for it. On the other hand, some choices are suspicious, for example the way basically every warning that is enabled is turned into an error. That choice made it much more difficult for upstream to find a gradual deprecation policy that does not piss user off – good luck deprecating things gradually when the warning designed for this immediately breaks the build by default for all your users.)

(I worked on namespaces in 2011, see remnants of the proposal, I refer to it from time to time when relevant. Maybe I would revisit someday, but there is much to do in other parts of the language.)

One underlying, low-level aspect of namespace proposals that I would really like to have is the ability to have different compilation units with the same name, and use them both in my program. I worked on preparatory changes for this ( https://github.com/ocaml/ocaml/pull/2228 ), I believe that Leo White and Mark Shinwell looked at it as well, but no one pushed the work to completion to have something upstream that allows this in practice. I think that it would avoid a lot of unnecessary complexity around name-mangling in the current namespace proposals, whether upstream or tooling-based.

One aspect of dune experiments that I find very interesting, and certainly deserves discussion with upstream, is the work on hiding transitive dependencies to avoid “implicit” dependencies. @bobot turned this into an RFC for upstream inclusion, Add -Ihidden in addition to -I for avoiding transitive dependencies in the initial scope, and I think that this is nice work going in the right direction, I hope we can turn this into an actual upstream improvement in the upcoming months.

4 Likes

Dune unilaterally made decisions … [about warning and errors]

For that topic I believe Dune is not confortable with the place it got itself into. One possibility discussed, but not implemented because too many things, is to allow packages to define compilation profile that could be used by other packages. So one package would be able to say something like (env (profile (of-package janestreet))) or (env (profile (of-package ocaml-devel))). The goal would be to let the community make those choices.

(PS: There is a page where we try to gather ideas for upstream OCamlFeatureRequest · ocaml/dune Wiki · GitHub )

1 Like

Never heard of it before, thanks!

It would be great it you could play the intermediary between dune and the compiler, as you did for the -Ihidden RFC. I think it needs people with their foot in both groups to continue making progress.

Imho dune turning on a wide range of warnings and making them errors has been overall a win for the ecosystem. It forces us to be more disciplined in dealing with code quality issues. I would push back on the idea that dune builds break by themselves. If a build breaks without an explicit upgrade, that is giving us valuable feedback that the build is not reproducible. If it breaks after an explicit upgrade, then the human who did the upgrade should figure out how to fix it (or suppress the warning if not).

For production codebases that must be maintained over time, strict warnings as errors is a godsend. Just look at Go, this is one of the things they got very right.

5 Likes

The issue with warnings isn’t a good example to generalize. We sleep walked into this situation while we were busy frantically turning jbuilder into a build system usable by most OCaml projects from its humble start as a build system for JST’s public projects. After the work was done, the mistake was realized. Though it was already too late to change things without causing a lot of breakage.

Thanks for the link regarding namespaces. We intend to add whatever build system support is necessary if you ever get around to implementing it.