Does opam lock do what you want? I’ve been using it for projects and it’s worked well in most cases. It’s not a perfect solution since opam-repository does change over time, sometimes changing existing package versions in breaking ways.
There’s also opam switch export/import FWIW.
1 Like
It might do the trick, but again it’s not the default behavior. Esy could be a good solution (one “switch” per project, automatic locking, etc.) if it was the one and only default. Anyway that’s really OT now, but it’s always a pain to come back to a project that compiled 6 months ago and doesn’t anymore.
1 Like
I think maybe we should keep this thread focused on Camlp4, rather than general questions about lockfiles in opam.
8 Likes
What I agree with (from @ivg message) - defining a clear strategy for any future deprecations. Like the grace period, announcements at the OCaml site, etc. So people would see the uniform plan for any major breaking change, and will have the time to migrate. Less surprises and everyone will be happy.
BTW, general note: the OCaml community does not actually move that fast; it seems a lot slower than many “modern” language projects. Compared to something like Swift it’s very slow.
1 Like
@ivg makes important points, and if his contribution is considered off-topic then it would be good to have it in a new thread so that we can discuss it. I was tempted to make similar points when I read about how compiler plugins were removed, and I am glad that he wrote this.
1 Like
@ivg’s last message (and all messages) are related to Camlp4 and therefore on-topic. Note that there was nothing particularly wrong about the off-topic messages, they should just go somewhere else.
Given the occasion to comment on this message: I liked the user stories a lot, and I agree with a lot of what @ivg wrote.
Interestingly, I disagree with the part that @XVilka agrees on: I don’t think that discussing and agreeing on a deprecation strategy would help. I think that in the past we have deprecated too eagerly, and we should try to not deprecate as quickly (I have said this more privately before but: I think that deprecating Pervasives was a mistake), but that there is no way to make deprecation painless by talking more about them, because the point of @ivg’s user stories is precisely that the people bitten by the deprecation are usually people outside the core community, who do not actively participate in those discussions.
This (the fact that more deprecation policies won’t help) is rather clear to me with the unsafe-string effort. I think it’s a case where our deprecation approach has been super careful: we introduced safe strings in OCaml 4.02, August 2014. It was widely announced that this would become the default in the future. Then we made safe strings the default in OCaml 4.06, November 2017. Surely most packages had updated to be safe-string compatible by then, right? Wrong. Before the default changed, nobody did anything to properly support safe-string – the three-year window was essentially useless in terms of encouraging package authors to upgrade, and much more happened in the following six months of ensuing “oh crap users tell me my package is broken”. Now it’s 2019 and it was proposed to remove -unsafe-string completely for 4.09 or 4.10, but personally I think this would still break too many things to proceed right now. (Just a few packages remain uncompatible, but they are typically the kind of packages that a surprising amount of people rely on, and would have their place in similar User Stories.)
I think that the most efficient way to keep code working in the long run is to Not Break things. When we really think we have good reasons to break something, pain will ensue no matter what we do, and in my experience the most efficient thing is to just fix the broken code (and try to minimize the changes in the fix release, to avoid cascading incompatibilies). Communication is always nice, we should keep a good time window between introducing a new feature and breaking the older version (to help people maintain compatibility across versions), but I don’t think that agreeing on more processes, uniform plans, etc., would help unbreak code any faster – it certainly doesn’t seem to have worked well in the past.
(One thing that does help seeing the impact of changes and getting motivation for fixes as soon as possible is Continuous Integration, at the level of individual projects and at the global opam-repository level. Those have been very helpful and I think they will be more and more helpful as time goes on, and we get more resources to act on them efficiently.)
(Another thing that would be helpful is automated code-upgrade tools for OCaml. )
3 Likes
I would be careful in drawing conclusions from that example. Unless I’m mistaken there was no visible indication (e.g. under the form of compilation warnings) that strings were becoming immutable. Warnings are incentives to move on, most developers like their builds to be clean of them.
I think some deprecations policies might be useful upstream e.g. regarding the stdlib, everything is done a bit randomly for now (I actually had promised you a document to keep track of that at some point, I only have it… on a very long todo list). In particular the idea that before something is deprecated a workaround should exist for a few versions, aswell as more precise timings as to when the actual deprecation and removals will occur.
FWIW I also agree with a lot of what @ivg says but I also think we should not cling to brittle technology. Anything that is guaranteed to break on every compiler release and is not part of upstream is brittle technology and anything that depends on it becomes as a side effect which makes the eco-system more fragile. camlp4 is part of that.
5 Likes
Yes, from @gasche’s account of the safe strings story, I was guessing that there was no warning at compilation.
I don’t have much to add on backwards-compatibility, except to mention from a research perspective that it helps to think about it ahead of time in a project. It is not just a practical necessity, but it is also a helpful and interesting design constraint. So it’s good to recall that both industrial and academic users are attached to it.
1 Like
(Another thing that would be helpful is automated code-upgrade tools for OCaml.)
Totally agree. Programming also with Rust cargo fix --edition is a joy to work with - Transitioning an Existing Project to a New Rust Edition. And it is a part of the toolchain, so available out of the box. Providing a similar experience would be a blast.
3 Likes
It is true that ocaml the language does not move very fast, but due to pervasive usage of ppx, even a minor version bump does not guarantee effort-less upgrade.
There are two things which make ocaml unstable
- pervasive usage of ppx, i.e, compiler plugin even in base library
- lacking a standard way of doing conditional compilation, I know there is cppo, but it is better to have something baked in
1 Like
Having a PPX-based conditional compilation out of the box in OCaml would be very useful, indeed (without a dependency on third party libraries).
I agree with @ivg that more communication is important, especially as the community grows. I agree with @gasche that it will always have a limited impact, though it will at least help people who follow these communications.
I also agree that in the past this hasn’t always been done, but overall I feel like this is happening. -safe-string is a good example of it. Another good example is Dune; even though the project is moving fast, the Dune team has put a lot of effort into backward compatibility, planning changes and communication. Of course, Dune is not the compiler but it is now a Core part of the ecosystem.
Regarding Camlp4 and ppx, it is fair to say that this has been a bumpy and difficult ride for everyone involved. Sometimes, the perfect design is clear right from the start and sometimes we have to go through several iterations in order to find the right design and solve all the smaller sub-problems along the way. It seems to me that we are in the latter case for ppx and it will hopefully be over soon. And even if the system is not perfect right now, it has already helped a lot of people do what they had to do and create cool new systems. Believe it or not, in several talks from reason users I have been to ppx was presented as the coolest feature of OCaml.
Finally, when you are concerned about an aspect of a change that’s happening, such as the Pervasives deprecation or the bigarray changes, please do not hesitate to raise those concerns directly to the people involved. Personally, I would always welcome an email saying “hey, have you thought about X and how it will impact users?”.
4 Likes
Yes, and this is the main message of my stories. The OCaml System Distribution, that is the standard library together with the compiler and accompanying tools, should be super stable. Unlike other, so called modern languages, OCaml is not in an alpha-development stage. It is a mature language with a long history and a rich set of features. Adding new features that do not break stuff is good. But breaking stuff for no reason, but because it went out of hype, no longer popular, or just proved to be not very good to use, or there are better ways to do, is not a good idea, I think. Basically, any non backward compatible change to the OCaml System Distribution, should be accepted with a lot of care and weighting of cons and pros of this change, so that the negative impact of breaking the user code, could be outweighed with real and measurable improvement of the ecosystem.
For example, making strings immutable could be justified by concrete improvements to OCaml, like string interning, which will lead to faster code with less allocations. Unfortunately, nothing like this was done. So we just have lots of broken code, textbooks, tutorials, etc, with no real benefits.
Another example, deprecating Pervasives. What benefits it should bring to the community? I can see only drawbacks, such as invalidating nearly all books, that have been written so far, tutorials, stackoverflow answers, and basically all the corpus of knowledge. I don’t see anything wrong with the OCaml design. Haskell has Prelude. Rust has prelude. Basically any language has it. Is it about the name? If yes, then the time when we could choose names has long been passed, again OCaml is not in its alpha stage. Renaming things just for the purposes of renaming doesn’t contribute to the stability and the size of our user base.
No let’s go back to the camlp4 story. Yes, we now have, probably, a better infrastructure than camlp4. But it doesn’t really mean, that camlp4 should be killed. Well, if we will recall the camlp4 story itself, it was also a better solution to what is now knows as camlp5, and the latter was deprecated in favor of camlp4. An irony is that camlp4 is now again deprecated in favor of camlp5 which was deprecated by camp4. So the history teaches us, that we make errors. This is an inevitable part of the development process. And it means, that we should not do the policing and dictating people what to use or how to use. The OCaml System Distribution (along with other important parts of the ecosystem, such as dune, ocamlfind, oasis, etc) are mechanisms and we should let the users decide what they want or not want to use.
We should not also forget, that for the real world OCaml users, those who use OCaml for everyday work, it is all about the money and productivity. A typical users is either a commercial developer, who is payed for work being done, or a PhD student who needs research to be done and papers published. Nobody is payed for updating their code to the new and (now considered) modern technology. When I’m writing my quarterly report I can’t include time spent on switching from oasis to dune, or updating the code to the new compiler version, because this is not what we get funds for. From the business perspective, this activities are unnecessary and lead to money loss.
Thus, any breaking changes, incur money/productivity loss for the OCaml users, which should be really accounted when any new breaking change is considered.
4 Likes
I think your position is too extreme – and your point about safe string being unuseful so far a bit far fetched: some of my APIs actually only worked provided you’d not mutate the strings it returned, e.g. xmlm; now this is enforced by the language.
Of course pointless change should be avoided but there is also benefit from removing cruft in software and it should be done while moving the eco-system with you. The alternative being stuck in a vortex of compatility layers and technical debt which incur their own idiosyncrasies and usability issues.
At a certain point you have to decide whether you want to provide a good experience for the current and future users of the language or for the past users. There’s a balancing act to perform but it should definitively be in favour of the former; unless you want the platform to die.
If you are not able to convince your management that software actually does rot and that it needs maintenance, fire them: it should be a well known fact nowadays.
1 Like
Or simply kept till the next major release? It would be nice to have a major release not more often than, say, 5 years, and all the breaking change could happen on major upgrades only.
I mean, even Ada and C++ change and break stuff, and that’s fine. Just do the major releases, so that people could target something.
So far I don’t even understand what major OCaml release number even means.
4 Likes
I can still, more or less, compile C code written 40 years ago without much effort. I can still take a document written with TeX in 1989 and format it and get pretty much pixel for pixel the same document. It’s very hard to compile old OCaml code. I’ve noticed this from the time I joined the community, and it’s been a bit of a bummer. Normally one expects that a language will be pretty stable from release to release, and that tools will be very stable.
BTW, for a long time I worked in the NetBSD operating system community, and there was a strong commitment to never breaking backwards compatibility even for binaries. You could run decades-old binaries just fine, and this often proved to be very useful (for example, when re-bootstrapping a language written in itself that had gone dead for a long time.) Things did get deprecated a lot, but whenever they were deprecated, the back compatible version was left essentially forever; you were just encouraged to use the new thing.
Mostly I’m extremely happy with OCaml, but a thing that I do find disappointing is that not only do things break a lot, but they also move slowly. Normally one expects either “moves fast and breaks a lot” or “moves slowly and breaks nothing”.
That would absolutely be a big help. Being able to just run “update my code” is one of the reasons the go people have managed to get away with relatively big language changes without so much pain. I saw in the discussion above that apparently Rust has similar tools.
3 Likes
Very big fan of TeX here. But, do you find TeX to be a usable language and system ? Personally I don’t and lost too many frustrating hours with the system than I’m willing to acknoweldge here.
I would like people to stop propagating this misconception.
I maintain around 30 OCaml packages. The number of my packages that break on new OCaml releases has almost always been 0 (zero) since I have been doing this (i.e. more than 10 years). Besides the natural flow of releases of these packages allows me to cope with the deprecations and refreshements imposed by the evolution of upstream.
The problem seems rather to be that people make themselves slave of pointless brittle technological “solutions” (e.g. ppx_fields_conv) because they are not willing to write one or two lines of boilerplate that will make their code future proof for many generations of the language.
7 Likes
In fact, pure TeX rarely used nowadays, and higher-level abstractions like LaTeX, ConTeXt, XeTeX, LuaTeX dominate the use cases. Moreover, a complete rewrite of LaTeX (LaTeX3) is in progress, and breaks many things, while improving the experience drastically. Many packages nowadays deprecated, or can’t be used with LuaTeX for example. So I don’t think TeX system as stable as many people tend to imagine. Wear and tear exist even in the Knuth’s kingdom.
1 Like