OBazl Toolsuite - tools for building OCaml with Bazel

Ecosystem
,
41

I’ll add to that Bazel’s notion of toolchain, which I think is probably unique. It’s pretty slick, and particularly apropos for OCaml. Obviously you may have multiple toolchains if you’re doing cross-platform development, but Ocaml comes with no fewer than eight toolchains right out of the box: ocamlc, ocamlopt, ocamlcp, ocamloptp, in .byte and .opt versions. Then we have flamba and who knows what else.

So how should a build language express toolchain control? Most have a build “mode” flag for bytecode v. native; I don’t know what they do about the profiling compilers. But flags don’t scale - what happens when you start adding true cross-compiling OCaml toolchains? And what about building the standard toolchain with different C compilers? (FYI Bazel toolchains for LLVM and Zig are available and I’ve used them to build the OCaml C kernel. Took about 10 minutes to get them working.)

With Bazel you can register all the toolchains you want - with selection characteristics like build/target host platform - and at build time it will select the one that best fits the “profile” of your build. You set buildhost and targethost platforms at the command line and Bazel takes care of the rest. You can also use user-defined CLI flags to control tc selection. For example you might configure things so that using Bazel’s built-in --compilation_mode=opt switch serves to select the flambda toolchain.

I’ve used toolchains instead of a ‘compile mode’ flag to support native v. bytecode compiles. Think of bytecode as what runs on a ‘vm’ platform, and native code as what runs on a ‘sys’ platform. Then our 4 basic compilers are vm>vm, sys>sys, vm>sys, and sys>vm. The default is sys>sys. If you want to emit bytecode, you pass --platforms=@ocaml//host/target:vm on the command line (this can be abbreviated). The beauty of this is that the build rules don’t have to worry about it - Bazel will select the right toolchain. So long as they all have the same tool interface, the rules don’t care. There are cases where the build rule needs to know if the target is the vm (e.g. in order to emit the right commands to configure a runtime) but they don’t need to know anything about the toolchain.

A very nifty illustration of how cool this is, is that any build tools that your project needs (e.g. some kind of preprocessing, or in fact menhir, ocamllex, etc.) are automatically switched to the appropriate toolchain, one that targets the build host, which is what you want. Also, with js_of_ocaml, you can use the sys>sys toolchain to build jsoo itself, but the rules that use it to transpile will automatically switch the toolchain so that the input to the jsoo compile is built with a vm-targeted OCaml toolchain.

It’s very OCaml-like in a way. Toolchains provide a ToolchainInfo struct that contains the tools, which the rules then use. That struct is essentially a toolchain interface. Bazel figures out at build time which implementation (specific toolchain) should be bound to that interface, so to speak.

Izzat cool or what?

Gregg

3 Likes
42

Totally concur with that. At my day job, we’re using over a half dozen different C++ compilers from various vendors, each with multiple important variants, e.g. profiling, thread sanitizer, address sanitizer, valgrind, lcov, et cetera. And that’s just for the C++, never mind the many many other languages that various nerds have forced into the system.

Absolutely could not manage the complexity without the Bazel toolchains concept.

43

Hi, Gregg! Thank you for your hard work on this, all of this seems very interesting.

Are there any plans to integrate with rules-nixpkgs? Being able to define an Opam toolchain that is pulled in by Nix would be amazing.

44

No plans at the moment, but I know the Tweag folks are interested in rules_ocaml and they’re pretty heavily involved in various nix-related Bazel things, so maybe someday. I’ve just never found the the time look closely at nix. I looked at it briefly until I discovered that it will only install at the system root with no customization allowed, which seems like a good reason to ban it with extreme prejudice, but a lot of people really like it, so I suppose I’ll take another look at it when I can find the time.

45

I tried it recently. For me, a bigger problem than the /nix directory [1] is that it insists on injecting stuff in the global shell configuration. I worked very hard to have an /etc/bashrc which does exactly the necessary things (setting up PATH, MANPATH, correct stty and similar) while avoiding the insufferable frills (colored prompt, AAARGH!) and have it propagated to all my systems seamlessly. I’m not throwing that out, even for the goodies of Nix.

[1] After all, /nix is not fundamentally different from /usr/local/homebrew

1 Like
46

Revisiting this thread…

p1. I’ve now gotten my hands sufficiently dirty with Dune that I can replicate most of the functionality that I implemented in OMake with my Conjury library. I have not warmed to it, and I’m now regretting the time I’ve invested in it. The build logic I’ve written with it works, but it makes me very sad. There are lots of places in my logic where key names are replicated in various dune files because there isn’t a reasonable way to do variable binding and expansion. I also hate that dune insists that you modify the source code for the tool to add new stanza forms. I really hate that.

p2. I haven’t started trying to write anything with b0 or brzo but I think I’ve figured out from reading the copious quantities of source code on Github that use them what are the basic ideas around them— and, I think a light bulb went on over my head when I noticed that b0 tries very hard to avoid being a package distribution system. It wants to delegate that to opam (and maybe alternatives). Moreover, I think I figured out what brzo is doing in the mix. (It’s the moral equivalent of bazel run :outcome instead of bazel build :target and I gotta say, I think I should have looked at this more closely sooner, and I shouldn’t have bothered with Dune. I’m, in fact, rethinking how much I want Bazel rules for OCaml. (I used to think it would be useful not just for my hopes of selling OCaml to my colleagues at work, but also to give me a reasonable way to avoid using Dune in my hobby projects. Now, I think Bazel is really only going to help me at work.)

I do have one worry about the B0 System.

I get that it’s intentionally not trying to be SuperEasyBarelyAnInconvenience for newbies to learn, because that would make it a lot more difficult for experienced users to use it in projects that need a full featured build system. However, it’s not easy to learn, even for somebody who’s already an OCaml expert. This is in contrast to a lot of other systems that are in wide use. For example, with Bazel (and similar tools that are derived from Python and similar languages), you can spin up pretty quickly a new engineer who knows C++ well but has never seen a line of Bazel before. The syntax is not weird, most of the unusual concepts (like the path syntax) are not that much of a departure, and it’s pretty easy to get up and running just by filling in some templates.

I suspect the B0 System, once it’s fully polished and ready for public consumption, can be presented in way that makes it much less cryptic to newcomers, maybe even more or less as simple as Bazel can be, but I’m not sure. I feel like I may need to get my hands really dirty with B0 to really know.

Another observation I would make is that Bazel is a unified tool that combines a source code package distribution and integration system, a software build configuration manager, a compiler toolchain driver, and a raft of other smaller features. These are all separated into different tools here in OCaml land, and they are developed and maintained by entirely different organzations, which do not share an architecture and design authority. This has both positive and negative aspects, and one of the negative aspects is that it puts a fairly steep learning curve in front of newbies. “I have all these tools, each for a separate aspect of the development process. Which one is which, and how do they fit together? What is the entry point to the whole system?” I’m not sure I can see how to file down the sharp edges there.

1 Like
47

Good grief, has it really been a year?

Good timing, anyway. I’m in the last mile preparing a set of tools for release. Will post a status update soon.

In the meantime, here’s some concrete good news. Yesterday I managed to install a couple of Bazel packages in a local OPAM repository. That is, the build and install: directives in the opam file run Bazel. Since I’ve already got a tool that integrates OPAM into Bazel (converts META files to BUILD.bazel files, etc.) this gives us full OPAM-Bazel integration. There are still a lot of details to address, but at least we have a working Proof of Concept.

2 Likes
48

That’s awesome. My day job devx people finally upgraded to the version of Bazel that allows experimental additions of new tool chains without fucking up the highly regimented system of integration controls meant to keep just any random engineer with access to the authoritative repository from landing whatever code they feel like onto the giant robot and turning it loose on innocent humans. So I might be able to do some demos for colleagues soon. Yay!

(Yes, none of that would be difficult in a world that does source code integration with a package manager rather than a monorepo, but you go to public roads trials with the stack you have not the stack you wish you had.)

49

Does anybody know what the current status of OBazl is? The documentation wrt. the latest release (2.0.0) seems outdated, and there don’t seem to be any working demos around. A simple “hello_world” example would be nice!

50

I’ll have a new version out next week.

3 Likes