[ANN] opam-bin: binary packages for opam, beta release

Hi,

I am happy to announce the first public release of opam-bin, a
framework to create, use and share binary packages with opam:

https://ocamlpro.github.io/opam-bin

With opam-bin, you can :

  • build binary packages while installing their source counterpart with opam
  • automatically reuse previously created binary packages instead of
    compiling them again
  • export and share your binary packages as part of opam repositories
    for other users/computers to use

opam-bin is a framework in 3 parts :

This is the first public release, it should be considered as a “beta
release”, though we have tested it a lot in the last days.

opam-bin is a collaborative work between OCamlPro and Origin Labs.

Enjoy !

17 Likes

Somewhat related - setting up more opam mirrors:

It would be amazing to be able to mirror both opam source and binary packages.

1 Like

Little update: the binary repository for OCaml 4.10.0 has been expanded to contain a set of 1800+ binary packages:

https://www.origin-labs.com/opam-bin/debian10.4-amd64/4.10.0

4 Likes

That sounds very promising. I’ll have a look at using it for the next liquidsoap release!

opam-bin generate binary packages with no dependencies if all of them are tagged as build. That’s the case if your package only contains executables and no libraries.

For example, alt-ergo is splitted in 3 packages alt-ergo-parsers, alt-ergo-libs and alt-ergo, and the last one contains only the executables, so i modified to tag all deps as build and now, the binary package has no dep and can be installed directly in an empty switch.

You still need to provide binary packages for several architectures/libc, it can be a bit tedious.

1 Like

I wonder if I have overlooked some instructions somewhere that someone could point me to. I’m interested in adapting a workflow based on creating an archive of an opam repo using extract_mini_repository.sh to use opam-bin to avoid rebuilding the packages from source. Does anyone already have a modification of that script/workflow that can produce a standalone snapshot containing binary packages? Is this an expected/reasonable use case for opam-bin?

1 Like

Hi Josh,

I think it’s actually much simpler with opam-bin:

  1. remove ~/.opam/plugins/opam-bin/repo
  2. install a switch with all the packages you want for your repo
  3. the result is ~/.opam/plugins/opam-bin/repo

You can share it, it should contain all the packages and their dependencies.
Best,
Fabrice

4 Likes

Hi,

I just noticed that the version 0.9.4 that I uploaded on the public opam-repo has a bug and does not create binary packages if sharing is disabled in opam-bin config. I am going to create a new version to fix this stupid bug.

I’ve integrated opam-bin into our internal CI workflow. Binary packages are fetched from http based binary repository, and new packages are created by each CI job and are pushed with --merge option via rsync after the job has finished running.

YMMV but I disabled caching of _opam switch folder completely as opam-bin is working fast enough to match time spend on extracting/archiving _opam folder! And that was even without a fix of duplicated files in tarballs, I expect opam-bin to actually beat whole switch caching with the fix (already in master). Local development environments can now get a new switch with core, core_kernel under a minute! That’s amazing.

Many thanks to @lefessan for prompt responses on Github and a great project!

@avsm is opam-bin something that might be consider for inclusion into core opam infrastructure?

5 Likes

We’re evaluating it in the broader context of the opam security requirements; you can read the last meeting notes about this topic from last week.

The good news is that when sharing binaries is not involved (such as with a CI caching workflow as you are using), it is considerably easier to engage binary caching. I expect this to be the first usecase, but we do need more people to try out the opam-bin prototype. Feedback such as yours is very valuable, thank you!

2 Likes