I’d like to continue the discussion that was started in another thread about backwards compatibility by pointing to what seems to me to be a true marvel of backwards compatibility: the Rust compiler. Please read this page, and the part I’d like to focus on is
A Rust compiler will support all editions that existed prior to the compiler’s release, and can link crates of any supported editions together.
This is an amazing property, and I’m not sure how the Rust community accomplishes it. One thing that probably helps is the fact that they use LLVM for the backend (leading to fewer changes in the backend). How, though, are they able to do things like adding new keywords to the language, while supporting all versions in the same compiler? My first guess is that every new ‘edition’ includes the previous editions’ code, but perhaps I’m wrong.
It would be really nice if we could start using some kind of more meaningful semantic versioning in OCaml, and in particular, if this Rust scheme is doable, adopt this versioning scheme for ourselves as well.
This is a pretty interesting property. Rust also apparently has a tool that will upgrade code mostly automatically from earlier to later “editions”. Both of these probably make the situation much easier to cope with.
I remember seeing a talk (maybe it was Carol Nichols’ ‘Rust: a language for the next 40 years’) in which she mentioned that they can support multiple ‘editions’ of the language due to compiling them all to the same well-defined intermediate representation, called MIR. I think their main achievement is that MIR is stable. I’m not sure if OCaml has any stable IRs.
I don’t know the details, but i don’t think there is any magic. For instance i know rust has a single (handwritten) parser than behave differently locally depending on the edition. For lifetime (mir based borrowck) the old code was kept until the new code got used also in old edition. For path resolution, it’s just difference in parser and in the code to interpret module path.
They also have an AST validation pass that checks various feature flag and could be used to check edition specific invariants. MIR is basically to trust what lambda is to ocaml, not that much difference conceptually. Rust has a lot of feature flags, they are used a lot for unstable features, and editions have similar way of handling locally different behavior. This approach probably is hard to scale but it does not seem scaling is required so far.
In the end, it’s more a developer choice to maintain old behavior at the cost of additionnal work than a technical choice.
There’s always and ever been only one way to accomplish such a feat: hard, hard, hard work. Because eventually, you find bugs that you want to fix, and yet people rely on them in previous versions. For instance, there was a bug in Java 1.2 StringTokenizer, that was fixed in Java 1.3. Of course, commercial products relied on that bug. So when one of those products upgraded their JDK, they broke, and didn’t know why (b/c sure, you can look at the list of bugfixes, but it’s pretty long, and you don’t realize your code relies on buggy behaviour). And there’s no way you can write a “code upgrade tool” for code that relies on buggy behaviour – how are you supposed to know what to look for? [e.g. in the case I cite, it was different return-results of tokenization].
And eventually, you’re forced to break the promise.
Since people mentioned tooling, note that simple formal refactoring in OCaml doesn’t seem to be entirely trivial, some work regarding this is being carried out by @reuben.rowe and his collegues.
I didn’t have the time to read them yet but this and this paper look interesting.
Suppose we had to mention the current ocaml version in a dune project file. Not the compatibility, but which version the code was written for. OPAM/dune would then be able to use stdlib-shims as necessary to make sure the code gets its correct environment. This would take us a step closer to silent stdlib backwards compatibility (ie. there’s no need to include stdlib-shims as a dependency – it’d be automatically invoked on the correct platforms).
Created an issue/feature request in Dune repo for implementing Rust-like automated conversion from deprecated features to a new ones: https://github.com/ocaml/dune/issues/2563