Just a thought (probably impractical): if llvm moves to github and the tutorial to dune, it could be interesting to use the same tooling as RWO (or maybe mdx) to make sure the code snippets compile?
That would be great fun! At the very least, Iāve wanted to play with such tools a bit in the past.
Keeping Camlp4 compatible with upcoming OCaml releases is not actually that much work. If you ā or someone else ā are interested in having a look, I personally think that itās not more than a day of work per release of OCaml. (In contrast, fixing bugs within Camlp4 requires more expertise and can be painful, but thatās not required for keeping existing project compatible.)
The problem we have, collectively, is a that a fairly small set of people ended up maintaing a large portion of the building blocks of the OCaml ecosystem (Camlp4, ppx stuff, etc.), while mostly not using them themselves. Itās okay for a while, because it helps other people so we do it out of good will, but it gets old, especially if you have several āless than a day of workā projects to update on a regular basis. (In the specific case of Camlp4, JĆ©rĆ©mie was maintaining it because Jane Street used it, but they moved away in 2015 and itās been fan service since.)
If people outside that small set stepped up to maintain Camlp4, we wouldnāt need to make noise around end of life, deprecating projects that use it, the future of stream parsers, etc. But for now they donāt. (Iām not exactly sure why.) I think that some of the actions discussed here or elsewhere are fruitful (using lex+menhir in the LLVM tutorial: excellent idea), and some are probably a waste of effort compared to just keeping Camlp4 on life support, but itās a bit hard to tell which is which.
7 Likes
Well, then I believe that camlp4 should be moved to the ocaml-community where it will be maintained by the community, to the best of our effort.
I think that would still require someone to step up and offer to support camlp4. Otherwise ocaml-community risks becoming a dumping ground for abandoned projects.
3 Likes
Sure, I understand this and I will step in, if nobody will volunteer1.
1) The main reason, why Iām reluctant to volunteer myself is not because Iām afraid of burden, but due to my personal relationships with camlp4. I donāt like the guy, so probably, I wouldnāt be the best care provider for it. But I wonāt let the old guy to die, despite my personal dislike of him 
3 Likes
Iām reluctant to see ocaml-community take camlp4 precisely because I think itās something people shouldnāt use going forward. ocaml-community is really for things we think people will want and use quite actively but where the maintainer has moved on. utop is the prototypical example.
1 Like
Could you elaborate on this? The way I see it, people not using opam, and people using older OCaml versions, are not going to be upgrading any time soon. They are most likely using their distro versions, and based on past experiences theyāre not going to go past 4.08 for at least a few more years. That leaves a long time for them to become aware that Camlp4 is no longer supported. Itās quite a large deprecation window. I think thatās quite reasonable.
1 Like
Sure, the main question is whether by this time distribution maintainers and the community will be able to update the infrastructure, so that once they will finally get the 4.09 update, they wonāt notice the difference. My point is that if this wonāt happen, and by 4.09 there will be roughly still the same set of packages which need camlp4, then they wonāt have a choice. The way how OCaml infrastructure works, is that older versions of compilers as well as major packages are not getting bug fixes. So, if youāre using some library and an older version of OCaml, and this library has a bug, most likely the fixed version will require a newer version of OCaml.
Also, Iām not saying that there is anything unreasonable or that we donāt have enough time. I totally agree, that we should move away from camlp4 and the sooner we will start it the better. Iām just saying, that this package should be maintained (on a very minimal level) for at least a couple of years, before it finally reaches the state of oblivion. And currently, it is definitely far away from it (based on installation statistics).
Agreed.
In general, where to put it is not a big deal, we can always create a fresh organization, specifically for the purposes of camlp4. @jeremiedimino, whatās your plan on it? Are you going to leave camlp4 under ocaml orgranization or transfer it somewhere else? What preferences do you have?
Rather than transfer it around, it would be really nice to have another maintainer getting familiar with the release cycle (e.g. for 4.09). I agree with your assessment:
So Iād be very happy if you wanted to just join the existing ocaml/camlp4 repository as a maintainer, and guide it through a couple of releases for OCaml 4.09 and 4.10 to make life easier for distribution maintainers in particular. I can give you access to the repository if @jeremiedimino agrees.
My initial plan was not to volunteer myself and take it unless nobody will take it before the 4.09 release. But I suspect, that anyway it will be me 
So just to close this discussion, I will take the burden to keep camlp4 alive for 2 or more next releases. Then, we will probably just archive it. My github handle is ivg.
Initially, I imagined that someone might want to step up because they like the technology itself or because they need it for their day to day development. In this case it would have made sense to transfer the project to them. In the end, we are only talking about keeping Camlp4 alive for longer, so we can leave it where it is for now.
Before we go ahead, there is one aspect that hasnāt been discussed here; what is the exact value of keeping Camlp4 alive? When modernising an older project, itās unlikely that Camlp4 will be the only blocker. There are many other things that have changed in the past few years. Strings are now immutable for instance. The dependencies of a project might have changed as well. There are many reasons other than Camlp4 why bringing a project up to date might require additional work.
All in all, Iām wondering if doing new releases of Camlp4 will really help anything, or at least it seems to me that we should wait for feedback from the 4.08 upgrades in the various systems. For instance, now that we have a 4.08 Camlp4 in opam, how many packages using Camlp4 are compatible with 4.08 out of the box?
On the other hand, keeping the project alive just to keep it alive for an unbounded period of time can have a negative impact. Itās less incentive to migrate away from it and we might miss out cases that we would have discovered by really stopping Camlp4. Delaying the 4.08 release of Camlp4 was a good example of how it triggered discussions and a few projects to migrate to ppx.
What do you think?
4 Likes
I would suggest to look at the problem from a very different perspective. From a user perspective. Iām sorry for a long narration, I didnāt have time to explain this in a more concise manner.
User story #1
Consider an industrial user who has developed a solution in OCaml and is planning to deliver it for a few years. During the initial stages of the software product lifecycle, the project manager benefited from abundance of coders and the project was moving up-to-date with the OCaml infrastructure. Then the project was released, majority (if not all) coders left, and the project entered the support stage. At this stage users still expect bugs to be fixed, but the project manager no longer has the man-power to move the project as fast as the OCaml infrastructure moves.
Since the project was heavily using external dependencies (to minimize the cost of development), it occurs that a few of the bugs, discovered at the later stages of the project lifetime, actually belong to other projects, such as Batteries or Core. Those bugs were fixed, of course, in the upstream, but it requires updating to the newest version of OCaml (which no longer provides camlp4, and all the libraries are newer, so the effort is much higher than a manager can afford).
So from the perspective of this project manager, OCaml moves too fast and is not really caring about its users. What the project manager wants? The support cycle of at least 5 years, where the end-of-life announcement of a feature is issued no less than three years before its end of life.
User story #2
In this story we have a PhD student, actually two of them. One has written an excellent paper and even submitted an artifact with the paper, so that everyone can compile and reuse it, and compare with own results. The paper was submitted and accepted, and even got a reward. And the student moved forward and even graduated and now has lots of other stuff to do.
Another student, a couple of years later is developing a competitive solution and they want to at least repeat the results of the older paper using the provided artifact. Unfortunately, since the OCaml community moves so fast and doesnāt provide long term support, the artifact is no longer installable. Since the younger student is very new to OCaml, they hasnāt enough capability to revive the project, so they leave it very frustrated by the existing work (heavily suspecting that the results were forged), they are also very frustrated with OCaml (since it doesnāt work, in their opinion) and although they initially thought to learn it, the abandon it in favor of an inferior but a much more stable language. They end up spending more time on implementation, getting inferior results and as a result, both the OCaml and the science community suffers.
Conclusions
The following, as well as anything else that I say, is my personal opinion. Right now, OCaml moves really fast, which is good, but we are burning the bridges behind us, leaving those donāt match up with our speed behind without any option to catch up again. Deprecation of Pervasives, dropping Bigarray.*.map_file, dropping camlp4, making strings immutable, stopping opam 1.x repositories, switching to Dune everywhere, adding/removing compiler plugins, camlp5->camp4->ppx_driver->ppxlib story, and many more changes which are definitely good for the overall health of OCaml still hurt those who do not have enough manpower to adapt to those changes.
And although as an active OCaml programmer Iām happy with all those changes, as an active project manager and research scientist Iām a little bit frustrated. The half-life period of an average OCaml project is about half an year. It basically means, that most of the projects bit rot in a year or so. And together with those projects the knowledge and other assets invested into this project, decay as well. Which actually renders OCaml less usable for real world project.
Suggestions
I understand the burden associated with the support of older versions of any non-trivial project. But at least, we should try not to break things that we can leave intact. And we also need a much more explicit deprecation procedure, with proper announcements of the end of life (which do not coincide with the actual end-of-life of the feature, as it happened with camlp4, where it was deprecated and killed basically at the same time).
Besides, another good example, is the map_file deprecation, which was deprecated for some time with no clear end-of-life date and no clear mitigation path, until it suddenly was removed (yes the mitigation in the form of mirage/mmap library was added but without any announcements or references to it, or updated documentation (which still mentions a non-existed bigarray library) and, most importantly, all this mitigation was added not before, but at the same time as the PR which removes it was accepted). All these stories do not contribute well to the stability story of OCaml, to a story where OCaml is a serious industrial language, not a playground for academia.
11 Likes
If you see the C(89) and Python 2 stories - users will keep using the old technology if there is still at least one implementation. No matter that there are better and more modern standards - some people would never move until they are forced to. I donāt think in this case, with the deprecation warning in advance and announcement has been made already, it will be beneficial to change the mind suddenly.
1 Like
Iād say that for 1. some stdlib alternatives are pretty stable (containers, batteries) and continue to support older versions of OCaml for years. Core/Base seem to move a bit faster
For 2. itās an imho argument against opamās current default mechanism of constraint solving every time. I want a lockfile with explicit update, so that coming back to old projects always works. This is my biggest frustration with opam.
5 Likes
How is that different from an opam file with equalities in your constraints ?
Itās not the default behavior 
Also cargo, for rust, will try to upgrade in a way that satisfies the constraints if you ask it (and opam would be good at that!). Itās just that a separate lock file with hashes and exact versions helps with reproducibility, and defaults matter.
5 Likes
I donāt get it. This is not a question of defaults thatās a question of what you write in your opam files.
I donāt want to solve constraints manually and then write exact versions of all the transitive dependencies in my opam file by hand.
2 Likes
This is getting a bit OT but, assuming a healthy opam repository, if you are not using the transitive dependencies yourself in any way in your project this should not be a problem (i.e. you shouldnāt have to write them).