Compiling old versions of OCaml

Investigating my problem with utop backtraces, I found that some change introduced the problem between ocaml 4.07.1 and 4.08.0.

I want to narrow this by testing versions of the compiler based on commits between those two tags.

There is a documentation about custom compilers in opam, but it seems not to be working with pre-4.08 releases.

For example, I am trying to compile commit f0a92e5aca245d80bf7749521a220e31c765c025.

In particular, opam install . will say “no package definition found”.

So I tried to compile manually by saying ./configure --prefix $(opam var prefix) then make world, but this fails with

/usr/bin/ld: libcamlrun.a(backtrace_b.o):/home/yann/Documents/Programmes/ocaml/runtime/backtrace.c:31: multiple definition of `caml_debug_info'; libcamlrun.a(backtrace_byt_b.o):/home/yann/Documents/Programmes/ocaml/runtime/backtrace_byt.c:47: first defined here
collect2: error: ld returned 1 exit status

This is strange because I would have assumed that the ocaml repo only contains compilable commits. Is this due to modern compilers being more strict ?

Gcc and glibc tends to break backward compatibility from time to time making it not that straightforward to compile old versions of the compilers. If you want to compile old branches, the tip of the branches often contains maintenance patches that restore compatibility with newer version of gcc and glibs. If you want to compile an old commit, you could try to cherry-pick those maintenance patches.

1 Like

Thank you, this makes sense.

I managed to compile 4.07.1 by installing and using gcc-6. The problem is I now want to install utop, it requires ocamlbuild and camomile and these do not build either (a syntax error is reported in ocamlbuild_config.ml. I could not find a version that builds : very old versions require ocaml < 4.06, newer versions have this error.

Are you looking at opam-repository or opam-repository-archive?

In case it helps, I have a switch with OCaml 4.07.0 and utop installed. Here are the versions (the output is a bit verbose and includes packages that are not actually required, but you can at least find the versions of camomile and ocamlbuild):

$ opam list --switch=4.07.0 --required-by=utop --recursive
# Packages matching: (installed | available) & rec-required-by(utop)
# Name                     # Installed  # Synopsis
base-bytes                 base         Bytes library distributed with the OCaml compiler
base-threads               base
base-unix                  base
bigarray-compat            1.1.0        Compatibility library to use Stdlib.Bigarray when possible
camlp-streams              --           The Stream and Genlex libraries for use with Camlp4 and Camlp5
camomile                   1.0.2        A Unicode library
charInfo_width             1.1.0        Determine column width for a character
conf-autoconf              --           Virtual package relying on autoconf installation
conf-m4                    1            Virtual package relying on m4
conf-pkg-config            1.1          Virtual package relying on pkg-config installation
conf-unwind                --           Virtual package relying on libunwind
conf-which                 1            Virtual package relying on which
cppo                       1.6.5        Equivalent of the C preprocessor for OCaml programs
csexp                      --           Parsing and printing of S-expressions in Canonical form
dkml-base-compiler         --           OCaml cross-compiler and libraries from the DKML distribution that works with at least Win32 and macOS
dkml-runtime-common        --           Common runtime code used in DKML
dune                       2.6.0        Fast, portable, and opinionated build system
dune-configurator          2.6.0        Helper library for gathering system configuration
dune-private-libs          2.6.0        Private libraries of Dune
dune-secondary             --           Fast, portable, and opinionated build system
dune-site                  --           Embed locations information inside executable and libraries
dyn                        --           Dynamic type
jbuilder                   1.0+beta20.2 Fast, portable and opinionated build system
lambda-term                3.1.0        Terminal manipulation library for OCaml
logs                       --           Logging infrastructure for OCaml
lwt                        5.5.0        Promises and event-driven I/O
lwt_log                    1.1.2        Lwt logging library (deprecated)
lwt_react                  1.1.5        Helpers for using React with Lwt
menhir-secondary           --           Adds Menhir to ocaml-secondary-compiler
mew                        0.1.0        Modal editing witch
mew_vi                     0.5.0        Modal editing witch, VI interpreter
mmap                       1.2.0        File mapping functionality
ocaml                      4.07.0       The OCaml compiler (virtual package)
ocaml-base-compiler        4.07.0       Official release 4.07.0
ocaml-compiler             --           Latest 5.5 development
ocaml-config               1            OCaml Switch Configuration
ocaml-env-mingw32          --           GCC mingw-w64 OCaml Runtime Dependency (32-bit)
ocaml-env-mingw64          --           GCC mingw-w64 OCaml Runtime Dependency (64-bit)
ocaml-option-32bit         --           Set OCaml to be compiled in 32-bit mode for 64-bit Linux and OS X hosts
ocaml-option-bytecode-only --           Compile OCaml without the native-code compiler
ocaml-option-nnp           --           Set OCaml to be compiled with --disable-naked-pointers
ocaml-secondary-compiler   --           OCaml 4.14.2 Secondary Switch Compiler
ocaml-syntax-shims         1.0.0        Backport new syntax to older OCaml versions
ocaml-system               --           The OCaml compiler (system version, from outside of opam)
ocaml-variants             --           Current trunk
ocamlbuild                 0.14.3       OCamlbuild is a build system with builtin rules to easily build most OCaml projects
ocamlfind                  1.8.0        A library manager for OCaml
ocamlfind-secondary        --           Adds support for ocaml-secondary-compiler to ocamlfind
ocplib-endian              1.2          Optimised functions to read and write int16/32/64 from strings and bigarrays
ordering                   --           Element ordering
pp                         --           Pretty-printing library
react                      1.0.0        Declarative events and signals for OCaml
result                     1.3          Compatibility Result module
seq                        base         Compatibility package for OCaml's standard iterator type starting from 4.07.
stdune                     --           Dune's unstable standard library
topkg                      --           The transitory OCaml software packager
trie                       1.0.0        Strict impure trie tree
uchar                      --           Compatibility library for OCaml's Uchar module
utop                       2.9.2        Universal toplevel for OCaml
uucp                       --           Unicode character properties for OCaml
uuseg                      --           Unicode text segmentation for OCaml
uutf                       --           Non-blocking streaming Unicode codec for OCaml
xdg                        --           XDG Base Directory Specification
zed                        3.1.0        Abstract engine for text edition in OCaml

I have a working 4.07 switch too. What does not work is compiling from the source, ie.

git clone git@github.com:ocaml/ocaml.git
cd ocaml
git checkout 4.07.1
opam switch create . --empty
eval $(opam env)
# opam install . does not work due to "No package definitions found"
# So we compile by hand
./configure --prefix $(opam var prefix) -cc gcc-6
make world  # there is a syntax error when compiling world.opt
make install
opam repository add archive git+https://github.com/ocaml/opam-repository-archive
opam install ocaml=4.07.1 ocaml-base-compiler=4.07.1 --fake
eval $(opam env)
opam install utop=2.9.2

Many package compilation and installation operations do work, but I get

=== ERROR while compiling ocamlbuild.0.14.3 ==================================#
# context     2.3.0 | linux/x86_64 |  | https://opam.ocaml.org/#e87127a46b6e17aa7aea2b54b5dac3249ff8d68f
# path        /tmp/ocaml/_opam/.opam-switch/build/ocamlbuild.0.14.3
# command     ~/.opam/opam-init/hooks/sandbox.sh build make check-if-preinstalled all opam-install
# exit-code   2
# env-file    ~/.opam/log/ocamlbuild-157460-d9d3f6.env
# output-file ~/.opam/log/ocamlbuild-157460-d9d3f6.out
### output ###
# [...]
# 251 states, 1051 transitions, table size 5710 bytes
# 4334 additional bytes used for bindings
# ocamlc -w +L -w +R -w +Z -I src -I plugin-lib -I bin -I +unix -safe-string -bin-annot -strict-sequence -c src/lexers.mli
# ocamlc -w +L -w +R -w +Z -I src -I plugin-lib -I bin -I +unix -safe-string -bin-annot -strict-sequence -c src/lexers.ml
# ocamlc -w +L -w +R -w +Z -I src -I plugin-lib -I bin -I +unix -safe-string -bin-annot -strict-sequence -c src/param_tags.mli
# ocamlc -w +L -w +R -w +Z -I src -I plugin-lib -I bin -I +unix -safe-string -bin-annot -strict-sequence -c src/param_tags.ml
# ocamlc -w +L -w +R -w +Z -I src -I plugin-lib -I bin -I +unix -safe-string -bin-annot -strict-sequence -c src/command.mli
# ocamlc -w +L -w +R -w +Z -I src -I plugin-lib -I bin -I +unix -safe-string -bin-annot -strict-sequence -c src/command.ml
# ocamlc -w +L -w +R -w +Z -I src -I plugin-lib -I bin -I +unix -safe-string -bin-annot -strict-sequence -c src/ocamlbuild_config.ml
# File "src/ocamlbuild_config.ml", line 10, characters 0-3:
# Error: Syntax error
# make: *** [Makefile:422 : src/ocamlbuild_config.cmo] Erreur 2

and

#=== ERROR while compiling camomile.1.0.2 =====================================#
# context     2.3.0 | linux/x86_64 |  | https://opam.ocaml.org/#e87127a46b6e17aa7aea2b54b5dac3249ff8d68f
# path        /tmp/ocaml/_opam/.opam-switch/build/camomile.1.0.2
# command     ~/.opam/opam-init/hooks/sandbox.sh build dune build -p camomile -j 7 @install
# exit-code   1
# env-file    ~/.opam/log/camomile-157460-350f45.env
# output-file ~/.opam/log/camomile-157460-350f45.out
### output ###
# [...]
# Warning : normalization option is not supported
# (cd _build/default/Camomile && tools/camomilelocaledef.exe --file locales/ur.txt locales)
# Warning : normalization option is not supported
# (cd _build/default/Camomile && tools/camomilelocaledef.exe --file locales/uz.txt locales)
# Warning : normalization option is not supported
# File "Camomile/locales/dune.inc", lines 1967-1970, characters 0-156:
# 1967 | (rule
# 1968 |  (targets zh__PINYIN.mar)
# 1969 |  (deps    (:x zh__PINYIN.txt) (alias database))
# 1970 |  (action  (chdir .. (run tools/camomilelocaledef.exe --file %{x} locales))))
# (cd _build/default/Camomile && tools/camomilelocaledef.exe --file locales/zh__PINYIN.txt locales)
# Fatal error: exception Stack overflow

And if I try from the 4.07 branch, as opposed to the 4.07.1 tag, I can make world.opt without specifying gcc-6, but then I get the same errors with ocamlbuild (camomile does get built, though).

The file ocamlbuild_config.ml has the following content

(* This file was generated from ../configure.make *)

let bindir = "/tmp/ocaml/_opam/bin"
let libdir = "/tmp/ocaml/_opam/lib"
let ocaml_libdir = "/tmp/ocaml/_opam/lib/ocaml"
let libdir_abs = "/tmp/ocaml/_opam/lib"
let ocaml_native = 
let ocaml_native_tools = 
let supports_shared_libraries = true
let a = "a"
let o = "o"
let so = "so"
let ext_dll = ".so"
let exe = ""
let version = "0.14.3"

Even though in this situation, the native compiler is built and installed.

The ocamlbuild error is weird. let ocaml_native = should be followed by either true or false, never the empty string.
Maybe ocamlbuild ends up built with the environment variable OCAML_NATIVE set to an empty string ? That would explain the result, although I have no idea what could be causing the variable to be set in such a way.

It is not set at all. opam env does not set it, my standard environment does not have it, and it seems nothing sets it.

But even if I set it manually to false, the ocamlbuild_config.ml remains the same (although the variable is present in the env reported by opam upon failure).

So, opam install ocaml=4.07.1 ocaml-base-compiler=4.07.1 --fake does not create .opam-switch/config/ocaml.config and /share/ocaml-config/gen_ocaml_config.ml, which may explain the problems.

I finally found a way to get it working.

To be sure to have a platform like in the old days, I created a machine with systemd-nspawn as explained there and set up a Ubuntu bionic (18.04) machine.

In that machine, I installed the latest revision of opam : I tried opam 2.0, but it could not solve the dependencies for installing utop.

I clone the ocaml repo in some folder. Therein, I prepared a base opam switch

opam init --bare  # answer yes to setup questions
opam repository add archive git+https://github.com/ocaml/opam-repository-archive --set-default
opam switch create . --empty
eval $(opam env)
opam install ocaml=4.07.1 ocaml-base-compiler=4.07.1 --fake

Since opam does not do it, I manually added a file _opam/share/ocaml-config/gen_ocaml_config.ml with content

let () =
  let ocaml_version =
    let v = Sys.ocaml_version in
    try String.sub v 0 (String.index v '+') with Not_found -> v
  in
  if ocaml_version <> Sys.argv.(1) then
    (Printf.eprintf
       "OCaml version mismatch: %s, expected %s"
       ocaml_version Sys.argv.(1);
     exit 1)
  else
  let oc = open_out (Sys.argv.(2) ^ ".config") in
  let exe = ".exe" in
  let (ocaml, suffix) =
    let s = Sys.executable_name in
    if Filename.check_suffix s exe then
      (Filename.chop_suffix s exe, exe)
    else
      (s, "")
  in
  let ocamlc = ocaml^"c"^suffix in
  let libdir =
    if Sys.command (ocamlc^" -where > where") = 0 then
      (* Must be opened in text mode for Windows *)
      let ic = open_in "where" in
      let r = input_line ic in
      close_in ic; r
    else
      failwith "Bad return from 'ocamlc -where'"
  in
  let stubsdir =
    let ic = open_in (Filename.concat libdir "ld.conf") in
    let rec r acc = try r (input_line ic::acc) with End_of_file -> acc in
    let lines = List.rev (r []) in
    close_in ic;
    String.concat ":" lines
  in
  let p fmt = Printf.fprintf oc (fmt ^^ "\n") in
  p "opam-version: \"2.0\"";
  p "variables {";
  p "  native: %b"
    (Sys.file_exists (ocaml^"opt"^suffix));
  p "  native-tools: %b"
    (Sys.file_exists (ocamlc^".opt"^suffix));
  p "  native-dynlink: %b"
    (Sys.file_exists (Filename.concat libdir "dynlink.cmxa"));
  p "  stubsdir: %S"
    stubsdir;
  p "  preinstalled: false";
  p "  compiler: \"4.07.1\"";
  p "}";
  close_out oc

and a file _opam/.opam-switch/config/ocaml.config with content (you will need to adapt the paths)

opam-version: "2.0"
variables {
  native: true
  native-tools: true
  native-dynlink: true
  stubsdir:
    "/home/yann/ocaml/_opam/lib/ocaml/stublibs:/home/yann/ocaml/_opam/lib/ocaml"
  preinstalled: false
  compiler: "4.07.1"
}

I added the things that will be needed later, and copied the whole thing to another directory to avoid doing this step each time (I had to specify the version for dune because opam would try to install a later version that uses Unix.open_process_args_out, which was added in 4.08 and does not exist for all of the commits I am interested in — I wonder why opam does that since it believes it has a 4.07.1 compiler) :

opam install ocaml-secondary-compiler
opam install utop=2.9.2 dune=2.9.3 --download-only
cp -r _opam ../baseswitch

Then we start bisecting :

git bisect start
git bisect good 4.07.1
git bisect bad 4.08.0

git will checkout intermediate commits automatically.
Not sure if this is needed, but I manually modified the VERSION file to pretend that this is version 4.07.1+dev0-2018-04-09. Since this is an outstanding change in the working copy of the repo, git will not overwrite the file when bisecting, so no need to do it each time.

Then we setup everything, in one line so we do not have to attend it :

git clean -fxd && ./configure --prefix $PWD/_opam && make world.opt -j 4 && cp -r ../baseswitch/ _opam && make install && opam install utop=2.9.2 dune=2.9.3 --yes