Streamlining the OCaml build ecosystem

build

#41

About this, I saw (I’m not an user of jbuilder, @rgrinberg tried in the last hackathon to convert me to this build system but I’m lazy to switch and the couple topkg/ocamlbuild is enough for me now) than jbuilder put a X__ in front of the name of each module. It seems to be close to a namespace and, I think, it’s why jbuilder can build many projects.

My question is: it’s better to still in this situation and simulate the namespace by the name of the module or try to integrate something directly to the compiler - however, this question already appeared and we already some old implementations about that.


#42

I don’t see the connection with the ability to “build many projects”, but yes it seems jbuilder implements a form of namespacing via module aliases (that you can disable via (wrapped false)) which allows you to use short names in your projects while making sure those will not clash at the toplevel with other packages and without using -pack (which yields bloated executables).

This is still an ugly hack in my opinion, your source files get renamed behind the scene, the prefixed names still leak at the toplevel and worse are exposed to the end-user of the API (i.e. the same module members can be accessed via different paths) leading to a distatefull orgy of names.

Using module aliases also makes us loose a good property that we failed to exploit for many years (odig tries to go in that direction) which is that compilation units have their module dependencies fully described in the object files via interface digests (yes 98% of these META files can be inferred from the actual objects).

There are plans to do this. After all these half-baked solutions over the years, I hope we can eventually get one to this problem that is not a hack so that the linking model coincides with the name hiding capabilities of language’s module system. The current situation is embarrasing.


#43

This is still an ugly hack in my opinion, your source files get renamed behind the scene, the prefixed names still leak at the toplevel and worse are exposed to the end-user of the API (i.e. the same module members can be accessed via different paths) leading to a distatefull orgy of names.

IMO this is a gross exaggeration. I’ve yet to see anybody access anybody the double under scored names. So while name mangling is indeed a hacky solution to the problem, it’s extremely effective in practice. Therefore my expectation is that once we have something better, jbuilder can be simply updated to support it and we’ll get free support for it everywhere with hopefully minimal breakage.

@dbuenzli I’m curious, is this the only reason that’s preventing you from moving to jbuilder? Would you switch if jbuilder supported your current cmi hiding convention?


#44

It may be if you selectively quote me. Juste note that it has effects in many other parts of the system, the name mangling leaks everywhere, makes API docs completely messy and less accessible (e.g. the URI structure doesn’t map to the actual naming structure), and by dropping the dependency tracking we had before in compilation objects relegates strong and sound inferences we could do on compilation objects to ad-hoc heurisitics and conventions.


#45

@dbuenzli is of course right that jbuilder’s (and Jenga’s) namespacing tricks would ideally be replaced by something better integrated into the language.

But the downside is relatively modest. With short-paths on (which is desirable for other reasons), you effectively never see the double-underscore names in practice. Leo has a new version of short-paths in development which should be fast enough that I’m hopeful it will allow short-paths to be turned on by default. That should fix most of the problem in practice. There are some related issues that need to be solved in odoc, at which point I think sightings of double-underscore names will effectively drop to zero.

y


#46

(I also agree that module aliases are not a very pleasant solution to the namespace problem, and that we should try to get something better in the language – see namespace archeology for my own past work on the topic. Happy to hear that people are still working on that.)

There is something about jbuilder that is implicit in my review above. One can conceptually a build system in (at least) two parts: (1) a “core build system” part that defines concepts of artifacts, rules, actions, and their scheduling, and (2) a “surface build logic” part that decides what rules to use and how users describe and parametrize them. The strengths and weaknesses of the current jbuilder implementation (a very solid core build system, and a convenient but ad-hoc surface build logic) align very well with its designer intent: the initial goal was to have a solid core build system to build Jane Street packages efficiently, but to have a “surface build” interface just expressive enough to describe the Jane Street packages. External contributions to the tool have added new features to the surface build layer, but not (yet) really helped in making it more flexible.

I wonder if it may be interesting to package the core build logic of jbuilder as a library separate from the surface logic – this is certainly something that solvuu-build, for example, could make use of. I think we may still need more exploration/iterations in the design space of surface build logics in general.


#47

@diml is really the authority here, but my guess is that some form of extensibility will be exposed eventually. But it’s a more complicated thing than just exposing the current core build system. There are many choices to be made, like how to distribute jbuilder extensions, how a stable extension API should be managed, and what should the substance of the API be.

By staying in this more declarative but limited world for now, jbuilder can provide a better experience for those whose use-cases fit the system, and buys itself more flexibility in the future as we figure out what the extensibility story should really be.

y


#48

Well I’m afraid the visibility of module aliases is pretty part of their intrisic definition and semantics.


#49

Intrinsic or not, I can report that inside of Jane Street’s walls, I almost never see these double underscore names. Getting the right tooling in place is tractable.

y


#50

Well I’m glad you are happy behind your walls, but outside these walls where some care about api documentation I’m afraid this is not the case.

But anyways, it seems we all agree that module aliases for namespacing is not a good idea. In general for orientation in a code base it’s a good idea if the mapping from source and object files to the names you get in the language is as obvious as possible, module aliases do not enable this.


#51

I care about deploying API documentation outside those walls too, for example on https://docs.mirage.io, and the tooling issue seems eminently fixable in the short term at least in odoc. I couldn’t find an issue for this upstream; is there one @lpw25 already or should I create one?


#53

I would also like to point out that having a solution to namespaces seems very high priority to me since it should make it possible along the way to streamline the eco-system by allowing us to drop a few embarrassing redundant names, indirections and noise we have in the system right now.

I do think it’s very important that we improve usability on this front and get a clear, better and understandable linking model for the language.


#54

What would the governance model be for the standard build tool?


#55

Just wanted to point that the link is broken, it should be http://docs.mirage.io (not https).