Project structure and build system for a project with OCaml native on the backend and Bucklescript on the frontend

Ecosystem
1

Hi,

I’m starting a new project where I’ll have

  1. Some heavy lifting via OCaml native on the backend
  2. Erlang/Elixir in the middle for communication, orchestration, websocket handlers etc. (I am coming from this background so it’s really because of it being easier to me right now).
  3. Bucklescript on the frontend.

Obviously I’d like to share code between 1. and 3. but I also want to hide most of my business logic (so I don’t want to have everything compiled to JavaScript).

I already know how to integrate Bucklescript with Phoenix and I have tested it: http://blog.overminddl1.com/posts/setting-up-bucklescript-with-phoenix/

The native OCaml part will be an Erlang port and I don’t think I’ll have huge trouble in doing that.: http://erlang.org/doc/tutorial/c_port.html

I imagine the port will be a separate OCaml project, but I’m not sure how to share code between it and the Bucklescript project? Is it a very nasty trick if I just symlink modules between the two projects? I think it won’t be a huge problem, as the two will rest in the same git repo (under an Eerlang/Elixir project umbrella). But I’ll have to very carefully decide on what code to put in which module (so that I link it).

In general, can someone advise me on something smarter?

1 Like
2

You could extract the common modules in an ocaml library that you then import and use in the native ocaml component and in the bucklescript component. I think the only constraint you have is to keep the code 4.02.3 comparible for bucklescript, but if you need some more recent feature you could use cppo to rewrite the files depending on the compiler.

1 Like
3

I’d do more or less what @mseri said:

  • Split the OCaml parts into three subprojects: OCaml native, BuckleScript, and shared
  • The OCaml native part is built by jbuilder, the BuckleScript part by bsb, the shared part by both
  • Since the shared part needs to be both native and BuckleScript-compatible, it has to use only 4.02.3-current language and library features as well as not use anything in any of the standard modules that BuckleScript ships with: https://bucklescript.github.io/bucklescript/api/index.html (e.g., this means no JS objects, HTML elements, React components, etc.)

Please take my advice with a grain of salt though. I’ve used this layout in a Scala/Scala.js project before, but not in an OCaml project.

1 Like
4

After all I went js_of_ocaml way for the “common” library (to be used on both ends). It seems way easier to make it compile and to be able to use modern OCaml tools (in my case I need to write a parser).

1 Like
5

It might sadly sounds strange to many, but I’d recommend you stick to good old Makefiles to build your project if it’s more complicated than a narrowly focused project. Indeed, building any real world large enough project will involve many different languages and tools, pre and post processors, etc, that Makefiles are designed to handle and that any language specific builder makes either very hard or merely impossible.

I don’t know much about your actual situation, but I would judge poorly a technology choice based on how easy it is to build or how well it integrates with any piece of tooling.

FWIW, the fact that a tool has been designed more recently than another one does not imply that it’s more desirable. Actually, I would argue that, at the contrary, it’s more likely that mastering a tool that’s been around for a long time will be more useful to you in your future career than mastering a more recent tool that’s less likely to be of any use for your next project.

2 Likes
6

that Makefiles are designed to handle and that any language specific builder makes either very hard or merely impossible.

Writing a fully correct Makefile for OCaml compilation process is really difficult. I would never recommend it to anyone. Even more so if you want to write an out-of-tree Makefile for any complex OCaml project (with automatic dependencies detection). And this is really the kind of common problem of which the solutions ought to be shared by using a common build system.

4 Likes
7

I’d say sure, use a Makefile to orchestrate builds between different systems. But don’t re-implement specific build logic (like compiling interfaces, then implementations, etc.) in Makefiles. Let the build systems handle that, because that’s what they’re best at, and let make handle pulling everything together across the project.

1 Like
8

I’ve heard that several times but it does not match my observations.
I’d say writing any complex out-of-tree recursive makefiles is hard, with or without OCaml, but that writing simple Makefiles for not so complex cases is rather easy to do (and easy to fix in case something goes wrong).

I’d go as far as saying that ocaml compiler mimic some C compiler behavior in order to be more easily amenable to C build workflows. This is obviously true for the ocamldep tool but there is also something about how the compiler picks which compilation units to include etc. Despite Xavier said recently in a nearby thread that they drew inspiration from Modula-2, maybe Modula-2 itself drew inspiration from C:

So I’d like to know what are you referring to when you say that writing correct Makefiles for OCaml is hard. Maybe something can be done about it, that would be better than introducing a new build system specific to OCaml every couple of months?

9

First, OCaml compiler breaks the Makefile’s model where one command must produces one file. In other words, Makefile do not handle gracefully directed hypergraph. This requires workarounds. Then, you should add the fact that the exact dependency graph depends on the compiler backend and flags used: the dependency graph varies if -no-alias-deps or -opaque are used. Similarly, ml-only or mli-only compilation units require different dependencies; even more if one need to parallelize bytecode and native build.

Sure it is easy to write a Makefile that mostly works in a well-controlled environment but it is not exactly a robust solution. The fact that you mention that your own makefile goes wrong:

(and easy to fix in case something goes wrong).

adds weight to my feeling that Makefile brings a lot of friction and requires a lot of boilerplate to get things right even for relatively simple projects. Boilerplate that you should compare to the few lines required by jbuilder or ocamlbuild in order to setup a project.

3 Likes
10

I hope you guys forgive the joke, but if I was choosing based on that, I sure wouldn’t pick OCaml :rofl:

11

I believe this is only going to be a problem if you have distinct rules for the various files produced by the compiler, one rule for the .cmi and another one for the .cmx, to take the more realistic example, and that you have several rules producing the same .cmi file (because you have some .mli files - I’m leaving aside the case where you want to compile both in native and bytecode form). Then if you ever only mention in the Makefile the .cmi files corresponding to the .mli files and explicit their dependencies, I believe you are perfectly fine (there is a simple dependency tree then). And since you should not have more than a very few .mli files that’s not very verbose to declare those relationships explicitly.

I do not see what no-alias-deps or opaque options can break in the model I just described.

It feels quite strange to have to prove that one can compile a project with make, when most of projects have been compiled with make just fine before ocaml-specific build tools become fashionable. Especially when some of them can’t handle parallel compilation at all or require complex setup as soon as one venture outside of the realm of student projects (see for yourself the doc about integration with the basic pkg-config).

I totally believe that ocaml-specific builder are better at compiling ocaml libs and programs, but that’s not the only thing a build system have to do. First it has to build other things, it has to link the result of compilation together, sometime in non-trivial ways, and it has to use ue parts of the outside system as well (libraries, tools). This is where makefiles shines.

Of course makefiles will break at time. But so do any build system, even the most theoretically robust one, since what breaks them is usually the interaction with the rest of the system not the compilation per se. For instance, it is to find the proper version of a library, or to run the proper script to generate some code file, or to compile some source code twice, once with some preprocessor and another one without, etc… The few I know about OCaml specific build tools I’ve learn by debugging others build script, so I know they can break too :slight_smile:

In a typical project, one will want to compile some OCaml to machine code, some C, some OCaml to JS, will want to preprocess source files with some program beforehand in some cases but not all, generate some data files using some scripts, maybe also generate some source code files, and so on. No build tools will make this simple or robust, but at least makefiles will make it possible in a way that will be rather easy to adapt later on.

Let’s be honest: ocaml-specific build tools are great in a teaching environment, as they allow to cut down on system administration and file types shenanigans and go straight to the business of teaching computer science. This is a noble purpose. But the same I wouldn’t recommend to use ocaml-top as an editor outside of education, I wouldn’t recommend using an ocaml specific build tool.

But anyway, there is no point to argue because the trend is clear. Soon nobody will ever remember that there was a day when software had to be build at all, as this will only occur in some deep datacenter our dumb terminals will be sending all source code to. :slight_smile:

12

Just for comparison: OCamlMakefile is a Makefile for OCaml projects from a respected OCaml developer. Most people will prefer jbuilder/dune because it is simpler and faster but it is certainly not the only choice.

2 Likes
13

But anyway, there is no point to argue because the trend is clear. Soon nobody will ever remember that there was a day when software had to be build at all, as this will only occur in some deep datacenter our dumb terminals will be sending all source code to

This is a strawman. I do want to build OCaml software myself, thank you; I just don’t want to write big, fragile makefiles. Dune (or oasis) is neat because it’s tailored for OCaml projects and compiles them cleanly, generates .merlin files, takes care of installation, recompilation, etc. You can still use a makefile to start dune and deal with a few other chores (like uploading docs).

7 Likes