This is “less than announcement”, because this tool is so small, but I thought I should write a little descriptive post about it anyway. This post consists in two parts: 1. about debugging PPX rewriters, and 2. about building, Makefiles, and “composability”. I guess they’re not connected, but this little tool doesn’t seem worth two separate posts, so … this post is to explain how to use a new opam package, not-ocamlfind.
Two problems have bedeviled me (and maybe others) in using Ocaml for a while now.
Debugging PPX Rewriters
When using or writing PPX rewriters, it’s pretty bloody difficult to dagnose errors, especially when using multiple PPX rewriters. I’ve been writing my own using camlp5, and both when trying to figure out what existing rewriters do (so I could reverse-engineer them) and debugging my own, it was pretty painful. ocamlc -dsource really isn’t enough. So with not-ocamlfind, you can take any ocamlfind ocamlc invocation, viz.
ocamlfind ocamlc -package ppx_deriving.show simple_show.ml
and replace ocamlfind ocamlc with not-ocamlfind preprocess:
not-ocamlfind preprocess -package ppx_deriving.show simple_show.ml
and get the preprocessed output (along with a list of the commands used to produce that output), so you can run it yourself. In addition, there’s an executable papr_official.exe installed in the not-ocamlfind package-directory, that you can invoke to convert text to binary-AST and back. Notice how the steps are i. convert text to binary AST; ii. run PPX rewriter; iii. convert binary AST to text. Steps i and iii are of course not executed by ocamlc: they’re there so we can run this outside of ocamlc and see the result.
ppx_execute: ocamlfind not-ocamlfind/papr_official.exe -binary-output -impl simple_show.ml /tmp/simple_showa44366
ppx_execute: /home/chetsky/Hack/Ocaml/GENERIC/4.10.0/lib/ppx_deriving/./ppx_deriving package:ppx_deriving.show /tmp/simple_showa44366 /tmp/simple_show2225a0
format output file: ocamlfind not-ocamlfind/papr_official.exe -binary-input -impl /tmp/simple_show2225a0
N.B. If you use this with camlp5, you need to add a “pr_o.cmo” (or the equivalent" in order to get human-readable output.
That’s all: just a way to more transparently debug PPX rewriter invocations.
Makefiles and composable builds
When building moderate-to-large projects, none of the build-tools are very suitable:
a. Makefiles quickly grow unwieldy, esp. when .cmo files from one directory are used as inputs in another; it gets difficult to ensure minimal-work rebuilds (“change file f1 in directory a, and everything downstream has to get rebuilt”) while still actually rebuilding what does need to be rebuilt.
b. ocamlbuild was … impenetrable, and completely inimical to C/C++.
c. ditto Oasis. I used oasis a lot, and it was always difficult.
d. dune is pretty opaque too.
Of all these tools, Makefiles come closest, except for the complete lack of “composability”. So let’s fix that!
a. Each subdirectory of a large build, as its last step, installs a package into a “local install” repository, and other subdirectories used those packages instead of directly referring to files from other directories
b. In the “depend” section of each subdirectory’s Makefile, it lists the package-directories it depends upon, thus:
EXTERNAL := $(shell $(OCAMLFIND) query -predicates byte $(PACKAGES))
$(CMO): $(EXTERNAL)
So all the .cmo files in the directory get rebuilt, every time any of the package-directories used in this build are changed.
c. In that last step, where the subdirectory’s build installs an ocamlfind package, the problem is that that install will update last-modified times. So we need a new command:
$(NOT_OCAMLFIND) reinstall-if-diff pa_ppx_expect_test -destdir $(DESTDIR)/lib META $(TARGET) $(TARGET:.cma=.a) $(TARGET:.cma=.cmxa) $(wildcard *.cmt*)
reinstall-if-diff compares the files-to-be-installed, with the files that are already in the package directory, and if there are any differences, invokes ocamlfind remove followed by ocamlfind install; if the files are identical, nothing is done. And we need a “local-install” target:
local-install::
$(MAKE) DESTDIR=$(WD)/$(TOP)/local-install/ install
With these changes, a large project’s toplevel Makefile can simply say:
all:
set -e; for i in $(SYSDIRS) $(TESTDIRS); do cd $$i; $(MAKE) all; cd ..; done
and we can be assured that no superfluous build-steps will be run.
There is one little problem left, which I haven’t cleaned-up, mostly because it isn’t that troublesome: if the directories in that list $(SYSDIRS) aren’t in topological order, then “make all” might fail. But this is, I suspect, an easily solvable problem. Just not high on the list of priorities right now.
All of the little snippets in this post are taken from the pa_ppx project: https://github.com/chetmurthy/pa_ppx
and the not-ocamlfind project is available both from opam and on github: https://github.com/chetmurthy/not-ocamlfind