[ANN] OCaml Platform Installer alpha release

Ecosystem
,
1

In anticipation of the forthcoming OCaml 5 release—and hot on the heels of its beta release—we are thrilled to announce the alpha release of the OCaml Platform Installer.

As a reminder, the OCaml Platform is the recommended toolchain for developers to work with OCaml.

The Platform Installer allows the user to easily setup OCaml’s development environment, both for a first-time installation and for any new opam switches.

You can try it now by following the installation instructions, but TL;DR, you can install it with

$ bash < <(curl -sL https://github.com/tarides/ocaml-platform-installer/releases/latest/download/installer.sh)

And run it to install the Platform in your opam switch with

$ ocaml-platform

Don’t hesitate to open an issue if you encounter any problem!

Update of the Platform State

As part of the work on the Installer, we’ve updated the state of the Platform to make it up to date and clarify the requirements to include a project in the Platform. Here are some of the notable changes:

  • odoc has been promoted from Incubate to Active
  • OCamlformat has been promoted from Incubate to Active
  • ppxlib and other metaprogramming frameworks have been removed from the Platform (and ppxlib is now documented as the official way to do metaprogramming, as part of the official OCaml documentation)

You can see the complete changes on the corresponding PR.

In parallel, we also want to make the Platform more open and transparent. To do this, we want to provide a clear governance model that can be driven by the community. The governance model should answer questions like:

  • How can I incubate my project in the Platform?
  • Who decides when a project can be promoted?
  • How to discuss changes that would impact multiple Platform projects?

We’re currently following the OCaml.org’s governance, but are exploring new ways to govern the OCaml Platform.

We are leaning toward an RFC process for this, but we are still discussing the alternatives and how this could take shape. We will continue to experiment on the best governance model for the Platform with the project maintainers in the coming months, and we will publish the resulting governance on OCaml.org.

Platform Installer

The Platform Installer provides the best way to install OCaml and the recommended development tools for both newcomers and existing users.

Simplicity. It aims at replacing the existing installation steps with a much simpler workflow.

To install the Platform Installer ocaml-platform, you can run:

$ bash < <(curl -sL https://github.com/tarides/ocaml-platform-installer/releases/latest/download/installer.sh)

This script will install opam, if not already present in the system, and the latest version of ocaml-platform.

Then, to install the Platform tools:

$ ocaml-platform

If opam is not initialised, this command will initialise it. Then it will proceed to installing the Platform tools in the current opam switch.

For first-time users, the above two lines will set them up with a working environment that’s complete enough to hack comfortably with OCaml.

Speed. In order to speed up the process of installing development tools, the Installer caches the binaries to avoid for redundant compilation. For instance, you will only need to compile ocamlformat once per version of the tool.

However, some tools such as Merlin depend on the OCaml version. For those, the Installer’s cache distinguishes the binaries, depending on the version of the tool as well the OCaml version it was compiled with.

Integration. The Installer integrates the development tools it installs as opam packages to make it fully aware of what has been installed. The binary provided by the Installer for the dune tool will allow the installation of any package with a Dune dependency without reinstalling it.

However, some development tools include libraries in their opam package that are not provided by the Installer. In this case, installing the original opam package for the tool will replace the one provided by ocaml-platform.

The opam packages installed by ocaml-platform are dependency-free. This means that installing specific versions of your development tools will never mess with the actual dependencies of your project!

The Tools Installed

The list of tools installed by ocaml-platform will ultimately be the platform tools listed in the Platform Docs as either Active and Incubate. Currently, this list is still incomplete for different reasons, such as keeping installation time short for new users, the number of dependencies, or other technical constraints.

The set of installed tools already provides a complete working environment, with:

  • A build system: Dune
  • A documentation generator: odoc
  • A code formatter: OCamlformat
  • A release helper: dune-release
  • Editor integration: ocaml-lsp and Merlin.

Next Steps

There’s still a lot to do!

Governance. As mentioned above, we should have a clear governance model for how incubation and promotions happen in the Platform, who decides, and which criterias are applied. This should obviously be an open process driven by the community; however, we want to make sure that we propose a governance model that will work well, so we’re still discussing alternatives and experimenting on some options. We’d love your input on this, so expect a Discuss post with a proposed governance model for the Platform soon. Don’t hesitate to reach out to us before then if you want to get involved!

Editor Integration. While the Installer is a step forward towards a simple way to install OCaml, the UX for newcomers can still be improved. We plan to integrate the Installer to the official VSCode plugin. This will provide a way to get a complete development environment in only a few clicks directly in the editor. We will also explore how to make the setup of Emacs and Vim more straighforward.

Remote Cache. In order to further reduce the time taken to start hacking on a project, we are working on a remote cache for the Installer. The cache will be populated by a CI and would remove the need for compiling the tools locally. The local cache and compilation mechanism would still be used as a fallback if the remote repo is down or incomplete (such as for a pinned compiler).

Acknowledgments

Thank you to the developers and alpha-testers who contributed to the Plaform Installer project, particularly the Tarides engineers who have been driving the development:

We’d also like to thank our major funder Jane Street for supporting our work to improve OCaml’s installation experience!

16 Likes
2

I think this is excellent news. Congrats to the contributors!

However, I am a bit surprised that the OCaml Platform webpage does not mention any download link or installation process. This seems to be essential for newcomers.

Do you plan to update the webpage soon? Maybe it is not on the webpage already because of the alpha status of the installer?

3

Thanks for the kind words!

Updating the documentation on OCaml.org is definitely something we want to do. In addition to the OCaml Platform page, we’re also planning on updating the Up and Running guide to add a section that uses the Installer to set up a development environment.

We wanted to hold off until the stable release (which should happen at the same time as the OCaml 5 release) to update the documentation though - stay tuned :slightly_smiling_face:

4

Great work, but I have some questions:

  • do you have machine-readable data what went into the binaries that are being downloaded (i.e. which version of what) [nowadays the buzzword is "Software Bill of Materials] – NB: and opam switch export --full --freeze is a good start for the opam packages
  • do you have security in mind? the curl | sudo bash is not very nice from a security perspective, since that may do any harm.
  • is there an update path? when I run your installer, how to upgrade to the latest version of “platform installer” (can I check whether an update is available and what will change?)?

In my opinion, the path forward is to package up system packages and provide a repository thereof (with monotonically increasing version numbers, etc.).

If you’re interested, please reach out since there’s related work done for reproducible builds (https://bullds.robur.coop) - which automatically feeds package repositories (powered by opam, orb, builder-web, and some shell scripts) and ensures that each distributed package is reproducible.

4 Likes
5

Hello Hannes and thanks for your interest!

do you have machine-readable data what went into the binaries that are being downloaded (i.e. which version of what) [nowadays the buzzword is "Software Bill of Materials] – NB: and opam switch export --full --freeze is a good start for the opam packages

In fact, currently only the ocaml-platform binary is being downloaded. The tools are then compiled in a sandbox switch on the local machine, and cached in a local opam repository. Only the version of OCaml and of the tool is kept as info in the local cache.

do you have security in mind? the curl | sudo bash is not very nice from a security perspective, since that may do any harm.

We do have security in mind! We wanted a one line copy-paste installation for new users. So, we went for an install script, although we will consider your advice on setting up a package repository!

is there an update path? when I run your installer, how to upgrade to the latest version of “platform installer” (can I check whether an update is available and what will change?)?

Currently, the update path is simply to re-run the install script:

bash < <(curl -sL https://github.com/tarides/ocaml-platform-installer/releases/latest/download/installer.sh)

But there is no way to know if there is a new release, apart from checking the github releases page. Thanks a lot for the inputs! I’ll reach out to you soon by private message, as your work and experience on reproducible builds might help for the ongoing remote repo!

1 Like
6

It might be useful to also offer the ocaml-platform installer as an opam package (bootstrapping problems aside, of course it is useful to have an installer that does not depend on opam).

It looks like the ocaml-platform tool would be beneficial even if you already have opam and a development environment, as it seems to offer a convenient way to share various developer tools between multiple opam switches (and it could perhaps build and install itself as one of such tools, so it can be conveniently be kept up to date).

It might also be interesting to keep an eye on providing a binary built using the APE format (see APE Loader) which would be a single binary that would run on Linux/Mac OS/Windows/etc. See [ANN] Esperanto, when OCaml meets Cosmopolitan, unfortunately its support for threads.cmxa is not yet complete at the moment, but seems to be a promising alternative for the future, especially with the APE loader, which doesn’t require self-modifying executables anymore.

But of course the best would be to get this installer packaged for distros (and part of distros by default like opam is), and perhaps working with a package maintainer to see what it would take for ocaml-platform to be packagable (not requiring to download binaries off the internet, but building everything from source would probably be one).

1 Like
7

It’d also be interesting to explore how to integrate this with Neovim’s LSP ecosystem.

The latest seems to be this:

This one seems to install an ocaml-lsp binary into a separate dir, separate from the opam switch, which is great … as long as you only use one version of the compiler. Otherwise it overrides the version you’d have in your local switch and LSP doesn’t work at all due to OCaml version mismatch. It works for simple cases though.

(its predecessor may be more well known: nvim-lsp-installer

If that got changed to use ocaml-platform as an alternative to opam as “package manager” for the ocaml lsp then it could be made to work more easily with multiple compiler versions (or at least I hope so, I intend to do some experiments when I have some time).

1 Like
8

Thank you for sharing this project :slight_smile: .

Indeed, we plan to continue improving Esperanto and add support for threads.cmxa. I would especially like to inform you, as suggested above, that Esperanto works well with the infrastructure for checking reproducibility and that I have been able to experiment with both projects myself to produce: a reproductible actually portable executable :grinning:

9

Dear @panglesd, thanks for your reply.

When I look into installer.sh, at least also opam is downloaded and “installed” (copied to PREFIX, there’s a lack of manpages etc.).

Could you elaborate on that?

So every new release of a tool ~installed~ compiled by opam-installer will also need a new opam-installer release (i.e. what happens when a new version of dune is released, how do I get an update using “ocaml-platform-installer”)? I’m not sure I understand your suggested update process.

10

Let me try to explain a little bit more!

So, there are two distinct process:

  1. Installing ocaml-platform. This is done by the installer script which, as you found out, also installs opam.
  2. Installing the platform tools: merlin, dune, ocamlformat, … This is done by running ocaml-platform (more explanation of what happens during a run of ocaml-platform below).

For 1., I agree that using a script is not ideal at all. We do want to have a system package, ideally as part of a distribution. However, we are really at an early stage, and first focused on developing the “core” functionality, for feedback before going further.
Note that sudo is only used when needed (for the install command), and that we encourage users to check what is inside the script: Taken from the README:

Don’t hesitate to have a look at what the script does. In a nutshell, the script will install a static binary into /usr/local/bin/ocaml-platform as well as the opam binary if it isn’t already installed.

For 2. here is what happens when you run ocaml-platform (also detailed in the README):

  • First, “driving” opam, ocaml-platform infer a version of each platform tools (dune merlin etc). It will use the best one available for your version of OCaml, except for ocamlformat where it will first search for a .ocamlformat with a specified version.
  • Then, ocaml-platform checks which of the tools.version it has in a local cache (which is initially empty).
  • Using opam, it builds in a “sandbox switch” the ones to build (the ones that are not already in the cache)
  • it populates the local cache with the newly built package
  • Using opam, it installs everything from the local cache (which is a local opam repo).

So, to answer your question:

You simply need to run ocaml-plaform, it will know that there is a new version of dune by asking opam the best version available. (I answered incorrectly the first time as I thought you were specifically speaking about a new version of ocaml-platform, my apologies!)

Yes, I forgot about opam’s installation… I meant that only ocaml-platform (and opam…) is downloaded through the install script. Other tools are downloaded through opam.

I simply meant that, when discussing how the tool would work, we tried not to miss security issues. Since most of the communication with the outside world is done through opam, we felt it was okay, but I am no security expert… and would love specific advices on the matter :slight_smile:

@hannes I hope that I answered your questions in a clearer way!

@edwin that’s a good point! There are still some things to work out: since the tool is not switch dependent, having multiple versions of it (even in multiple switches) might create problem (for instance, v2 updates the cache to the v2 version, and v1 cannot use the cache anymore. (Similar to opam 1 and 2 with the update of .opam!))

1 Like
11

Indeed, thanks for taking the time to answer my questions.

12

Can you explain to me how is this different (and if it is better) from esy? All that I need to do to bootstrap my workflow with esy is install it globally with npm (which doesn’t need any sido permissions) and it will take care of everything in a per project basis. Does this platform take into account the different versions of OCaml and libraries of each project?

13

As a package manager, Esy is an alternative to the opam client.

To get a complete development environment with Esy, you need to follow the same steps as opam (i.e. run esy install ocamlformat odoc ocaml-lsp-server ...). With Esy you have the advantage of being able to list those in the devDependencies field in your package.json, but that’s essentially the same thing: you need to tell it which tool to install.

And the pain points with this approach are the same:

  • The tools’ dependencies can conflict with your project’s dependencies
  • You need to install the tools for each switch, which takes space and time
  • You need to know exactly what to install to get a complete dev setup, which is not the case for beginners

So the OCaml Platform Installer isn’t an alternative to Esy, but rather addresses pain points that both Esy and opam users have.

Unfortunately, the Installer isn’t compatible with Esy, as it uses opam switches under the hood. So Esy users aren’t really impacted by the release of the Platform Installer.

14

Thanks for the detailed response