Generating Makefiles with OCaml?

Ecosystem
1

Has anyone ever tried to generate Makefiles for OCaml projects using OCaml itself?

I understand there have been all sorts of build tools and helpers, and there is now dune of course, but I’m wondering if anyone has had the idea to just codegen Makefiles? After all, OCaml is famous for symbolic manipulation. It should be (?) easy to generate a Makefile from a succinct project description written in OCaml. The benefit would be easy bootstrapping from a single script.

E.g., input:

(* proj.ml *)

open Makefile

make begin
  exe "main" [
    cmx "config" [];
    cmx "api" [
      cmx "config" [];
    ];
  ]
end

Output:

main.exe : config.cmx api.cmx main.cmx
    ocamlopt -o main.exe config.cmx api.cmx main.cmx

main.cmx : config.mli api.mli main.ml
    ocamlopt -c main.ml

config.cmi config.cmx : config.mli config.ml
    ocamlopt -c config.mli config.ml

api.cmi api.cmx : config.mli api.mli api.ml
    ocamlopt -c api.mli api.ml

This is of course a trivial example. Point is, OCaml is a great DSL to describe a build graph and generate a lower-level build file (Makefile, or even ninja file). E.g., you might have noticed a redundancy in the above OCaml snippet that we can factor out.

2

There was GitHub - samoht/assemblage: A collection of tools to manage the configuration of OCaml projects at a time but as far as I know it never reached production quality.

Cheers,
Nicolas

1 Like
3

I don’t want to dissuade you, but … I actually doubt OCaml would be a good choice, b/c creating Makefiles involves so much string-manipulation, and OCaml isn’t good at that. If I were going to attack this problem, I’d try to build a MakeMaker (the perl equiv to what you seek) work-alike for OCaml.

The other place I might look is the OMake implementation (Ocaml implementation of make, but with added function). These days, maintained (IIRC) by Gerd Stolpmann (of findlib fame).

4

If you insist on generating some build system, consider ninja instead perhaps?
It’s faster than Make and arguably simpler, too.

2 Likes
5

I was thinking something specialized for OCaml, actually, so presumably there shouldn’t be that much string manipulation. Something much like Assemblage, actually, possibly simpler?

It’s tempting but make is just so much more easily available that it’s probably a better target.

6

Have a look at Haskell’s Shake, and Build Systems à la Carte for discuss about building systems in general.
Instead of creating a Makefile, Shake produces an executable that directly calls the compiler via command line, in this way it is more efficient and not limited by the capability of make program.

2 Likes
7

Dune has the ability to generate makefiles. dune rules -m ./your/build/target

3 Likes
8

It’s certainly doable, but what’s the point? Dune is a lighter dependency than make and only requires the OCaml compiler which is already necessary by your project. Make offers no performance advantage over dune either.

So it seems like a very challenging project (once you go beyond toy examples and start considering portability), with no benefit as far as I can see.

4 Likes
9

I agree with @rgrinberg : if you’re going to start with OCaml datastructures, then you’re almost certainly not going to have the flexibility to express the stuff that Make does, that dune cannot. Might as well just use dune.

And I’m not even a dune user.

10

If we’re going to talk about alternatives to Dune, then I’d like to put in my request for better support for OCaml in Bazel. This is one of the few remaining things that keeps me from evangelizing OCaml as a language for off-robot applications at my day job.

11

My impression is that dune is faster than bazel, while also being inspired by it. I haven’t run tests to verify this. For me, the main benefit of switching to bazel would be to keep learning a general purpose build system which I might use to compile code in other languages when necessary.

12

If you don’t already have a big pile of Bazel on your hands, then I would say “Thank your lucky stars, and just use whatever you’re already using.” If it’s straight OCaml and nothing else, Dune is probably your friend. A highly opinionated friend, but a good friend. The only reason to consider alternatives to Dune is when you have a giant monorepository full of C++, Python, Rust, Go, and who knows what else, and it all has to go through a giant multiplatform build, integration and test system. And if that is what you’re doing, then OMG I hope you have a lot of whiskey on hand if you’re trying to do it with Makefiles.

2 Likes
13

It should be possible for dune to be helpful in this situation as well. In theory, there’s nothing difficult about dune generating functional, but highly ugly, and unportable bazel build files. Do bazel builds allow for a stage where rules are generated using an external tool?

14

No, and it’s really quite opinionated on that point. Rules are written in Starlark period full stop, and Starlark isn’t Python because Python is too powerful for people entrusted to write BUILD and *.bzl files to be allowed to use it. And nobody complains to the Bazel gods about that, unless they want to be smitten down.

Bazel does allow you to call out to an external build system for constructing an imported package, and this is what jin/rules_ocaml does for building an external OPAM package, but it isn’t very robust at building an OCaml program from a list of .ml and .mli files— that will fail in mysterious ways if you blow on it gently, i.e. try to build anything moderately complicated. See below.

A reason OCaml is difficult for Bazel is that OCaml is finicky about build order in ways that all the various languages Bazel likes— because G***** likes them and them alone— are not. The ocamldep tool produces information that Bazel quite deliberately refuses to give you an automatic way to use in feeding the dependency graph. It is a matter of principled design that Bazel insists you manually and explicitly specify all the dependencies in a rule and they are not automatically discovered. It looks at our ocamldep tool and raises its holy symbol and shouts in a broken dead language until it goes away.

As you might imagine, that makes compiling the .cmi and .cmx files for a library that has a complicated internal dependency graph currently discovered quietly and automatically by ocamldep a bit of a problem. Which basically means, that if you want a decent OCaml toolchain for use with Bazel, you have to wrap it somehow so that when you write something like this…

ocaml_native_library(
    name = "my_library",
    srcs = glob(["*.ml", "*.mli"]),
)

…the code for that rule compiles all the .cmi, .cmx, .o files, and everything else, in the right order— because Bazel isn’t going to help you by slurping the output of ocamldep into its dependency graph like other tools will do.

An additional complication is that when you provide a list of source files in a srcs=[] parameter to a rule, the convention is that the list is unordered, and conventionally many shops uses a Starlark formatter that automatically alphabetizes such lists and a commit guard on the repository that requires the BUILD files to be fed through the auto-formatter. Which means you can’t require the srcs=[] list to be specified in the correct build order for OCaml.

To do this right, you need to be very clever in the Starlark code that compiles a library or a program, more clever than anybody has yet tried to be. I don’t think I would try unless I was getting paid.

1 Like
15

Writing OCaml build rules in crippled DSLs is something I’m very experienced in (Make, OMake), but I will have to pass on this opportunity. Somehow, reimplementing dune in skylark doesn’t sound very appealing.

I guess I was mostly inspired by blog posts like this one: https://www.tweag.io/blog/2019-10-09-bazel-cabal-stack/. Substitute haskell for OCaml, and cabal for dune, and I thought we’d be in business.

1 Like
16

I think the key concern from OP is about using plain OCaml code to configure the project. So neither dune or OMake has provided this kind of possibilities yet. However providing a combinator style interface to these build systems seems appareling.

1 Like
17

You might be able to do that in some shops that use Bazel, i.e. substitute a) OCaml for Haskell, and b) Dune and OPAM for Cabal and Stack. In a shop like the one used at my day job, calling out to an external build tool for first-party code is just not done.

We have a lot of processes that are built around fine-grained analysis and management of the Bazel dependency graph. Bazel really shines for this, and it pushes you very hard to translate everything into Starlark so you can get those abilities. Anything that requires calling out to an external build system is a pain in the neck unless you’re okay with treating it like a black box.

And even then, did you look at the bazel-tools directory in that DAML repository? There’s a lot of clever logic going on in there, and it still walls giant masses of the dependency graph off where Bazel can’t see it except as big blobs dependent on other big blobs.

18

pds generates makefiles but not in this way, it just uses a pretty simple templating language to spit them out.

19

I don’t want to flog a dead ungulate, but I found this page about Bazel Rules for Haskell Is Bazel right for me? — rules_haskell documentation and I found it really informative and wanted to add it here in case anyone cares. I looked at the code this is documenting, and it’s basically what I wish existed for OCaml. Not enough to write it myself today, but enough to hope someone else does before I have a real need for it.

20

Dune started as a makefile generator.

I wrote an OCaml program that was generating a Makefile for the Base library about 5 years ago. I then extended it to build more packages and after several months of work it became Jbuilder. It was a standalone tool that could build all Jane Street packages. Jbuilder then became Dune with the help of the community.

The original code of this Makefile generator can be found here. The Dune repository still has a file called gen_rules.ml today.

4 Likes