Help needed for integrating OCaml in the HPC ecosystem

Ecosystem
, ,
1

Dear OCaml community,

I work in high performance computing (HPC) and I really love OCaml. But users hate programs written with OCaml because they find it very hard to install on the HPC clusters they use every day. This is the problem I would like to solve.

On an HPC cluster, you have a huge amount of C/C++/Fortran codes, so you have access to tons of compilers and libraries for those languages. Now that machine learning is trendy, the HPC systems have adapted to Python. But when you use something else, you are alone.

The good news is that there exists some package managers designed for HPC clusters, that will solve the dependencies and compile everything from source using adapted libraries and compilers. One of those is package managers is Spack. Honestly, I find this package manager very helpful for C/C+/Fortran/Python packages.

So I decided to change the world, and became a maintainer of the OCaml package for Spack. I tried to do my best, but I really need some help now!

I think the problems I am facing have already been solved by some of you, because they are the same as the “OPAM vs Linux distro packaging” questions.
For the moment, the OCaml Spack package can be installed from source, as a regular program, and the OCaml libraries need to be installed also as Spack packages. I think it is a bit stupid, because this means that the developers have to re-package everything from OPAM to Spack. So I made a Spack package for OPAM, but now I am facing other difficulties.

I am now trying to modify the (Python) source code of Spack in such a way that OCaml becomes a language known by Spack, and so that Spack defers the packaging to OPAM.

I am not a developer of Spack nor of OPAM, but just an isolated OCaml programmer who wants to help the community. I would like to initiate some short-term collaboration between OPAM/OCaml-aware people with the Spack developers to sort out the big design problems, and then things will run smoothly and I can get back to my role of ocaml package maintainer.

I started a fork of Spack here a few months ago, if you want to have a look:

Please contact me if you are willing to participate!

Anthony

12 Likes
2

I’m not knowledgeable about spack but have used ocaml on a (RHEL) linux compute cluster. I resorted to installing opam on my user account and making an opam switch. This was a bit of effort but gave me full control over the versions of libraries I wanted to use. I guess if many users of a cluster were to do this one would end up with a lot of duplicated opam switches. Is that the issue here?

3

I’ve used OCaml on HPC systems for many years. I never bothered with any of the native system package managers. My solution has always been to install the opam binary into my personal home directory, and then run a couple of opam commands. Everything works and is done. When I wanted to let another user on the cluster run ocaml, I would make my ~/.opam directory readable by them, and tell them to modify their PATH to include the binaries therein. I don’t know if that meets your needs, but it sounds a lot easier than supporting another package manager.

2 Likes
4

Yes, it is one of the issues.

5

The goal of Spack is to enable the possibility to select which C/C++/Fortran compiler you use, which version on MPI, which versions of libraries etc, and recompile everything with the most appropriate options for your CPU. Once you have installed some software and its dependencies with spack, you can create easily a module file (those you load using “module load”). It works a bit like virtual environments.

To clarify, spack is a package manager that is used in the first place by the administrators of the clusters, not the users.

Also, in this community, the admins and the users are not OCaml developers, so they don’t know anything about all these exotic tools. They are used to make, configure, cmake, etc. but not dune, ocamlbuild, etc… If all this developer-oriented business can be hidden, it will make ocaml programs more adopted by this community.

1 Like
6

Look into opam-bundle (in opam):
if you want to make an ocaml program available to non ocaml users, it allows
you to create an automatic source-based installer.
If I had to distribute ocaml software, I would create a bundle for the version I want to distribute.

opam-bundle --self [SOME_PACKAGE]
3 Likes
7

Oh! I didn’t know about opam-bundle. I will look into it, it will be very helpful.
Thanks a lot!

1 Like
8

You should also look into the recent opam binary cache/repository feature.
I have not used it, but it fills a need: compiling opam packages once while being able
to install them everywhere (fast).
https://ocamlpro.github.io/opam-bin/

1 Like