Advanced C binding using ocaml-ctypes and dune

Learning
, , ,
#1

Hi there!

I worked on a socket.h binding last summer and had a great experience integrating ocaml-ctypes with dune, I thought that might be of interest to other developers so I wrote about it: https://medium.com/@romain.beauxis/advanced-c-binding-using-ocaml-ctypes-and-dune-cc3f4cbab302

Romain

9 Likes
#2

This is a good article. I encourage anyone who writes C bindings with ctypes to study it carefully.

A little bit of advice to shorten your dune files:

 (deps    (:gen ./gen_constants_c.exe))

This line isn’t necessary. Dune is smart enough to know that running a binary in a rule incurs a dependency on it.

dune has a truly amazing support for cross-compiling, which we do not cover here, but, unfortunately, its primitives for building and executing binaries do not yet cover this use case.

Indeed, we don’t have any primitives for running binaries on the target platform. Perhaps we should add some. However, we do in fact have some features in dune to solve this concrete cross compilation problem. As far as I understand, the goal is to obtain some compile time values such as #define constants and field offsets for the target platform. This does not in fact require to run anything on the cross compilation target. In configurator, we have a primitive C_define.import to extract this information. The end result is that these configurator scripts are completely compatible with cross compilation.

Perhaps this could be generalized to work with ctypes generators as well?

Funny bit of trivia: The hack in configurator required to do this is in fact something I extracted from ctypes itself. The original author is whitequark, who in turn wrote it to make ctypes itself amendable to cross compilation.

2 Likes
#3

If anybody wants to know more about this bit, I wrote an article about this last year:

4 Likes
#4

Thanks for the feedback y’all!

@rgrinberg A quick glance at the code for C_define seems to indicate that it can only handle static #define of type string, int or bool. Can it also handle cases such as:

#define SOCKLEN sizeof(socklen_t)

?

Romain

#5

Yeah, this case should work as well. I think we even some tests for it in configurator.

#6

Oh, sweet. I’ll give it a try!

#7

Hi! Just getting back to this.

Unfortunately, I don’t think that configurator will be enough. The build system also uses ctypes support for structures, which needs to compile and run using the cross-compiler in order to calculate the right sizes and offsets for the structures and their fields. As a user of the module, I cannot tap into the internals for this process via configurator, my only use is via compiler and executables…

#8

Ideally, bindings could be generated completely automatically, by parsing C headers with clang and generating appropriate interfaces. Languages that have support for this:

See, for example, how I generate bindings automatically for radare2 C headers using those in genbind.py script. It is very simple and straightforward. I wish something like that existed for OCaml.

2 Likes
[ANN] clangml 4.2.0: OCaml bindings for Clang API (for C and C++ parsing)
#9

Yes, this is a good point. I was also wondering if there was a tool that did the same in OCaml.

#10

Having dealt with multiple intricate arch-specific situations, I do not believe that a fully automated solution would work for all cases. It would be a great tool for some easy bindings but as soon as you get into some nitty gritty ones, I would be curious to see how that would be possible.

Couple of examples:

  • clock_t and time_t in POSIX are arithmetic types, they can be either float or int. How would you generate an automated OCaml API that offers a unified type on all platforms simply based on the C headers? Likewise, although perhaps easier, with int types of indeterminate size such as size_t and etc.
  • flac API is based on callbacks. How do you define the relationship between the callbacks and the garbage collector?
#11

The best scenario is that automated binding tools can do 90% of the boring binding work. The rest of the intricate work can be done manually. I’m sure that a lot of tools in the other languages also allow some customisations.

#12

This is something that we should be able to fix however. There’s no reason ctypes needs to run any code to calculate sizes and offsets. Unfortunately, our backlog for ctypes + dune is quite large:

  • We want to port ctypes to dune
  • We want to write a dune extension to simplify the dune boilerplate required

Once we have the above, we could indeed look into fixing cross compilation in ctypes.

2 Likes
#13

That all sounds great. Cross-compilation is not too important for me as long as we have a workaround like I described.

On a similar topic, after taking to Jeremy Yallop in ocaml-ctypes#620 I started what I hope could be a nice common repository of posix bindings, build on dune and ctypes with shared APIs, patterns and building blocks: savonet/ocaml-posix

The idea would be to propose to join all existing posix-related bindings under the same umbrella with both ctypes primitives for low-level bindings and high-level APIs for end-user consumption.

Some bindings would not be moved there, in particular those with a greater scope or additional complexity like unix-errno but for bindings that stick mostly to the posix specs, I think this would be a great step forward to cleanup the existing set of modules.

2 Likes
#14

There is an interesting project inspired by haskell-gi to automate OCaml bindings generation for GTK+ - ocaml-gi-gtk.

See more context here. The only downside (for OCaml developers) is that it’s written in Haskell.

2 Likes