The "forever beta" issue

opam
jbuilder
dune

#1

There is a trend among OCaml devs which is having really many pre-releases and not reaching a final release for a long time. In particular, dune and opam 2.0: Dune 1.0+beta1 was released on Mar 8, 2017, and currently it is 1.0+beta19.1. Opam 2 alpha was released on Jul 8, 2016, and it is now 2.0.0~rc.

As a package maintainer (of Debian, Fedora, openSUSE etc.), I find it difficult to decide which version should be packaged. Usually only the latest stable release should make into the distro archive, which should contain only stable code. However, there are many OCaml libraries moved to using the beta things, e.g. replaced its build system with dune. It means that those libraries in the distro cannot be updated until dune is packaged.

I wonder if there is any way to improve the situation, say:

  1. Try to release stable version sooner. Individual new feature/API can be labeled as experimental within the release.
  2. Make a big warning to the beta versions that downstream projects shouldn’t depend on it yet.

#2

Andy, firstly thank you for your packaging efforts! They are much appreciated.

Opam has maintained a nice degree of stability – once we stabilised it around the 1.2.2 release, we nominated that as the ‘LTS’ release and are ensuring that opam2 is a well-rounded release before we tag the 2.0 release. So you should continue packaging opam 1.2.2 for now until 2.0 is released.

Jbuilder and Dune have been moving very fast, and seen extremely rapid adoption. We decided last week that it’s no longer practical to call it a beta despite the developers wanting it to remain as such, and so Jbuilder 1.0 will be around the corner before we perform the name change. One area we’re putting more efforts into is testing infrastructure for larger packages (Dune, Core, Lwt, etc) so that we’ll have more visibility into breaking changes on distros. You can already see this in the container builds for many Linux distros on the opam repository CI, so that’s going to be deployed onto other projects as well.

The reason automated testing is important is to enable your first suggestion of getting stable versions out sooner to be practical. We’re basically perpetually short of developer time to triage issues, and have many many bugs on our radars. So in summary, I think we’ll have a nice summer of stable releases to look forward to as all these efforts come online. Please don’t hesitate to raise feature requests for things that will make packaging easier for you – for example, we’ve been discussing the possibility of generating RPMs and Debs directly from Dune specs, once it has enough metadata to generate the opam files.


#3

Thanks for the detailed reply.

I’m now packaging opam 1.2.2 for openSUSE. However, there is an issue that it depends on an old version of dose3, and it seems it cannot be easily upgraded. The issue was reported, but it was solved by putting a version upper bound on the dose3 dependency. Linux repositories usually only accept packages of the most recent stable version, and do not support downgrading. Right now the openSUSE dose3 package in the devel project is at 5.0.1.

Opam 1.2.2 being an LTS, would it receive updates to make it compatible with the latest versions of its dependencies? i.e. Releasing 1.2.3 that fixes the issue would be perfect. Alternatively, just an official patch would still be good.


#4

ping @samoht (the opam author)


#5

ping @AltGr (the opam maintainer :p)


#6

That specific issue seems to be just a module renaming, so patching it in opam 1.2 is fairly trivial:

From a1476be0edabf16584bc96c010f5bfe58d2950ec Mon Sep 17 00:00:00 2001
From: Louis Gesbert <louis.gesbert@ocamlpro.com>
Date: Fri, 13 Apr 2018 11:46:59 +0200
Subject: [PATCH] Fix compilation with recent versions of dose and cmdliner

---
 src/client/opamArg.ml    | 12 ++++++------
 src/core/opamCompiler.ml |  2 +-
 src/core/opamFilter.ml   |  2 +-
 src/core/opamPackage.ml  |  2 +-
 src/solver/opamCudf.ml   |  2 +-
 5 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/src/client/opamArg.ml b/src/client/opamArg.ml
index 7a8f0bdd..0c140224 100644
--- a/src/client/opamArg.ml
+++ b/src/client/opamArg.ml
@@ -379,9 +379,9 @@ let mk_flag ?section flags doc =
   let doc = Arg.info ?docs:section ~doc flags in
   Arg.(value & flag & doc)
 
-let mk_opt ?section ?vopt flags value doc conv default =
+let mk_opt ?section ?vopt flags value doc xconv default =
   let doc = Arg.info ?docs:section ~docv:value ~doc flags in
-  Arg.(value & opt ?vopt conv default & doc)
+  Arg.(value & opt ?vopt xconv default & doc)
 
 let mk_tristate_opt ?section flags value doc auto default =
   let doc = Arg.info ?docs:section ~docv:value ~doc flags in
@@ -459,13 +459,13 @@ let term_info title ~doc ~man =
   let man = man @ help_sections in
   Term.info ~sdocs:global_option_section ~docs:"COMMANDS" ~doc ~man title
 
-let arg_list name doc conv =
+let arg_list name doc xconv =
   let doc = Arg.info ~docv:name ~doc [] in
-  Arg.(value & pos_all conv [] & doc)
+  Arg.(value & pos_all xconv [] & doc)
 
-let nonempty_arg_list name doc conv =
+let nonempty_arg_list name doc xconv =
   let doc = Arg.info ~docv:name ~doc [] in
-  Arg.(non_empty & pos_all conv [] & doc)
+  Arg.(non_empty & pos_all xconv [] & doc)
 
 (* Common flags *)
 let print_short_flag =
diff --git a/src/core/opamCompiler.ml b/src/core/opamCompiler.ml
index 41783113..c7beb935 100644
--- a/src/core/opamCompiler.ml
+++ b/src/core/opamCompiler.ml
@@ -30,7 +30,7 @@ module Version = struct
 
   type constr = (OpamFormula.relop * t) OpamFormula.formula
 
-  let compare v1 v2 = Debian.Version.compare (to_string v1) (to_string v2)
+  let compare v1 v2 = Versioning.Debian.compare (to_string v1) (to_string v2)
 
   let eval_relop relop v1 v2 = OpamFormula.check_relop relop (compare v1 v2)
 
diff --git a/src/core/opamFilter.ml b/src/core/opamFilter.ml
index 0e919e74..ebce0903 100644
--- a/src/core/opamFilter.ml
+++ b/src/core/opamFilter.ml
@@ -203,7 +203,7 @@ let rec reduce_aux env = function
      | FUndef, _ | _, FUndef -> FUndef
      | e,f ->
        FBool (OpamFormula.check_relop relop
-                (Debian.Version.compare (value_string e) (value_string f))))
+                (Versioning.Debian.compare (value_string e) (value_string f))))
   | FAnd (e,f) -> logop2 (&&) false (reduce env e) (reduce env f)
   | FOr (e,f) -> logop2 (||) true (reduce env e) (reduce env f)
   | FNot e -> logop1 not (reduce env e)
diff --git a/src/core/opamPackage.ml b/src/core/opamPackage.ml
index 56605bbf..83908961 100644
--- a/src/core/opamPackage.ml
+++ b/src/core/opamPackage.ml
@@ -29,7 +29,7 @@ module Version = struct
 
   let of_string x = x
 
-  let compare = Debian.Version.compare
+  let compare = Versioning.Debian.compare
 
   let to_json x =
     `String (to_string x)
diff --git a/src/solver/opamCudf.ml b/src/solver/opamCudf.ml
index 62ea551d..f7903520 100644
--- a/src/solver/opamCudf.ml
+++ b/src/solver/opamCudf.ml
@@ -611,7 +611,7 @@ let check_cudf_version =
           `Compat
         | s::_ ->
           match OpamMisc.split s ' ' with
-          | "aspcud"::_::v::_ when Debian.Version.compare v "1.9" >= 0 ->
+          | "aspcud"::_::v::_ when Versioning.Debian.compare v "1.9" >= 0 ->
             log "Solver is aspcud > 1.9: using latest version criteria";
             `Latest
           | _ ->
-- 
2.11.0

The fix is not really needed on opam-repository, since there is the 1.3 version of opam-lib which already has it.


#7

Sorry for the possibly-silly question, but is there a reason not to advertise opam 1.3.0 and opam 1.3.1 as possible choice for users and packagers, besides their use in opam-lib? If they break compatibility with 1.2, maybe we could consider releasing an 1.2.3 with just minor fixes? (Would someone else than @AltGr be able to take care of most of the work by preparing a complete PR against the 1.2.2 tag?)


#8

Thanks for the patch! I’ve just tested it, it is missing one replace in src/core/opamVersion.ml.in. The updated patch: https://github.com/ocaml/opam/compare/1.2.2...andyli:1.2.2-fix.patch

Going to submit it to openSUSE soon :slight_smile:


#9

While there are quite a few improvements in 1.3 compared to 1.2.2, that benefit to its use as library for auxiliary tools (e.g. a reorganisation of modules and sub-libraries, support for compilers as packages), the corresponding opam version has never really been tested for release, and differs significantly. It was decided at the time to focus on 2.0, as adding another version to maintain, including upgrades and such, could have made support more difficult.


#10

Wouldnt it be easier to just package with something like AppImage?


#11

The main reason is that other packages of the same repository may depends on it. Usually packages are not allowed to depend on libraries outside of the repository. For example, if I want to package flowtype for Debian, I have to build it with the ocaml, sedlex, and other libraries in the Debian repository. This is a way to maintain the distro as a whole, to make sure all packages are built in a compatible way, and to make sure runtime libraries are shared (for file size, memory usage, and security reason).

In my case, my ultimate goal is to maintain the haxe package. Its upcoming version depends on some OCaml libs that switched to using dune, so I have to package dune in order to package those libs.


#12

AppImage can do that.

I once created a sheet to compare different cross-distro package manager, Flatpak and Habitat also support cooperation with local packages.

I think its possible to get in contact with their devs for detailed information.


#13

But that cross-distro support is not bi-directional. e.g. The ocaml-gen Debian package cannot depends on the dune AppImage package.


#14

Hnn, yes. So a complete chain can make sense.