Interim report on experience with ocamlnat

Ecosystem
1

Here’s an interim report on my experience with ocamlnat, which is an OCaml toplevel that runs native code rather than bytecode.

I’m very grateful to @dbuenzli for getting ocamlnat working and for explaining how to use packages with it in this thread and in individual help that he’s given to me. dbuenzli was kind enough to file an issue for the Owl package to help get it running with ocamlanat. (I am almost able to get Owl fully loaded into ocamlnat, but couldn’t have identified what was more was needed as well as dbuenzli did.)

I’ve worked on getting Core loaded into ocamlnat, but I’m giving up for now. It might be doable, but it’s a pretty big job–for me, at least. I’ve made some progress, but there are many components and many dependencies. One has to load each by hand as a cmxs file, in the right order of course. Not all of the module names map in obvious ways to filenames and directories. I have figured out some of the mappings, but have failed with others. I don’t think that all of the modules referenced are part of Core; that’s another complication. This detective job might be easier for more experienced OCaml folks, but still would require a bit of work. I think that there are cmxs files that one would need but that don’t actually exist, but I am not sure! (In any event, the fact that I could try to do this at all is due to dbuenzli’s help.)

Batteries doesn’t automatically build any cmxs files, but the list of its components in lib/batteries/META is a lot shorter than the list for Core, so it might be easier to get it working with ocamlnat once one has cmxs files.

Summary: At the moment, one can’t use Batteries or Core in ocamlnat, but the traditional Standard library is available. An option would be to rewrite bits of code or add one’s own definitions for functions provided by Batteries or Core that the Standard library doesn’t provide, obviously.

I’ll investigate making cmxs files for Batteries myself, and will submit issues for Batteries and Core to ask for better support for ocamlnat. I dont’ think this is a high priority for the community, but I do think that in the long run better support for a native-code toplevel would a good thing, particularly for scientific work. Thanks again to dbuenzli and everyone who commented on the original thread.

1 Like
2

If you have just a tiny additional amount of time, could you please try to load containers in ocamlnat? It should not have complex dependencies (compatibility packages such as result only) and the .cmxs should be installed.

1 Like
3

Sure–happy to do that. I won’t get to it until later, but I’ll let you know what happens.

4

Most of the time there’s a one to one map between cmxa and cmxs, if the cmxs is missing it’s likely that it was not build by the package which you should report upstream.

Regarding the manual loading it shouldn’t be too hard to reuse the analysis done by odig’s data-driven (in the sense not using META files) toplevel loaders on cmxa’s and assume they apply to cmxs to load them, my guess it that this will work most of the time (though the current loader have a few issues I should solve at some point). IIRC the infrastructure for ocamlnat is mostly there but it’s currently stubbed. I may give it a shot a some point but I’m a bit busy with other things right now.

5

I just quickly tried since a lot of the infrastructure is in fact already in place. However I’m also hiting the fact that some of the libraries odig depends on are missing .cmxs (and notably ocaml’s own compiler-libs which I need to do the dep analysis).

6

It also might be worth giving Base a try, since it doesn’t have nearly the
same kind of dependencies as Core does.

7

You can use Findlib (ocamlfind) to find the mapping between packages and cmxs files, as well as to get all the dependencies in the topological order.

Concerning packages that lack the cmxs files, we use the following trick that is an ocamlbuild rule, that builds a cmxs file from a cmxa.

Basically the command is ocamlopt -shared -linkall xxx.cmxa -o xxx.cmxs

1 Like
8

That make sense. I had forgotten about Base.

9

That’s all very helpful. Thanks.

10

The idea of managing the load dependencies automatically is obviously very appealing. Thanks for working on it.

11

I was able to load most of the components of Containers into ocamlnat. Most of the files had no external dependencies. containers.cmxs needed result.cmxs in the result package, but loaded once that file was loaded. (I sent @c-cube a private message about the other components with which I had trouble: containers_thread.cmxs and containers_top.cmxs.)

1 Like
12

It seem OCaml, like its compiler libs does not install threads.cmxs. I have updated MPR7625.

13

Note that oasis builds (and installs) .cmxs files automatically, if I remember correctly. I think jbuilder does the same.

1 Like
14

Thanks for the experiment! It seems that ocamlnat is not that far from being usable in practice for me, then (even though native utop + automatic loading of deps would be quite helpful).

15

That sounds like a useful thread.

Would you mind expanding on your reasons for using ocamlnat?

It is currently an experimental tool (with no guarantee that it works properly from the maintainers of the OCaml compiler distribution), the main reason being that there are essentially no regular users of it – otherwise we would know when this stuff breaks. I thought that the HOL community would be eager users of ocamlnat, given that some tools work directly from the ocaml toplevel, but it seems that this never happened.

16

when it becomes manageable i will definitely use ocamlnat. use case: interactive explorative numerical work. i have a custom library for statistical inference which i would drive from utop. waiting for results for minutes vs. seconds makes a big difference there. larger calculations would still be run non-interactively but to find out the right parameters at which to run the long calculation, better speed in the toplevel is definitely helpful.

1 Like
17

there are essentially no regular users of it

We use it quite a bit at Jane Street in the form of a native code utop. That’s why we submit patches to fix it when it breaks.

2 Likes
18

@gasche it’s a bit of a chicken and egg problem, isn’t it? Since it’s not really supported, and not compiled in official releases (afaik), people don’t use it. But if it was supported and more widely advertised, a significant fraction of people who currently use the bytecode toplevel would instead use the native one, because of the performance improvements.

2 Likes
19

Well obviously few people were motivated in the past to put in the work to make ocamlnat work reliably (after the initial development by Alain Frisch) – with the exception of regular fixes from Thomas Refis and Leo White. So some people may be interested, but until recently their interest level was “not enough” to put work in it – which is completely fine. In any case, it helps for motivation to get some assurance that there may be users, and @n4323’s explanation were helpful in that regard.

20

We will definitely switch to a native toplevel in BAP as soon as possible. Analyzing even a small binary in a bytecode takes so long, that it effectively renders the interactive mode non-useful.

2 Likes