and apologies in advance if this is not the right place to ask such things. I have asked on email@example.com before a friend made me realise this was probably a better place. I have two questions related to what is appropriate or not to package via OPAM.
My first question is inspired by the tool Topiary. It is not an OCaml tool (it is written in Rust) but it could be part of the OCaml ecosystem, depending on the definition, as it can format it. Since it is quite new, it is also pretty unstable. This means that most package managers won’t have it but also that when it starts appearing in package managers the version discrepancies will also lead to discrepancies in the way it formats OCaml. For those reasons, it seems to me like it would make sense to package it in OPAM. This would allow OCaml projects to easily have access to it while controlling its version better.
I think I have seen this before with eg. Z3 that also changes quite a bit from version to version and is very relevant to some OCaml projects (Why3, F*, etc.) but I cannot find evidence of this anymore and it might be an exception and not something that should be more widespread than this. I suppose integrating Topiary could be done in a similar, fairly delicate manner: Topiary could provide a statically-linked binary for various architecture, and the OPAM package would consist in downloading this binary and putting it in an accessible place.
My first question is therefore: Is this a reasonable use of OPAM, in general or in this particular way?
My second question is slightly related but also not fully. I have discovered recently the conf-* packages (eg. conf-jq) and I find the idea great. I was wondering how far these were meant to go. For instance, I can see how jq is rather popular and probably used in many projects. What about tools that are much more rarely used? I have a personal project that uses sassc, LilyPond, Inkscape, TiMidity++ (with the freepats library of synthesizers) and xvfb. I guess the use of those tools in other OCaml projects must be very rare.
My second question is therefore: Does it make sense to package these in a conf-* package? Or not at all?
I wondered when you would eventually show up around here with this :–) It really like the approach you took here as I think programming language communities spend way too much time on their own bespoke tooling – package managers, build systems, editor integration (solved by lsp nowadays), etc.
I wanted to give topiary a try at some point since I could never convince ocamlformat to work the way I’d like both from a styling and editor integration perspective. But somehow not having it in opam makes the whole business inconvenient (I didn’t have the time to try it yet).
Topiary could provide a statically-linked binary for various architecture, and the OPAM package would consist in downloading this binary and putting it in an accessible place.
The tagline of opam is "opam is a source-based package manager for OCaml. " . There’s a truth and falsehood in it.
The truth is that it’s source based and I suspect many people prefer it that way. The repo maintainers will maybe contradict me but I suspect that downloading binaries is a no-go.
The falsehood is that it’s “for OCaml”. opam is fully OCaml agnostic. So you could likely just publish an opam package for your tool distribution that depends on conf-rust (which ensures that cargo is available) and whose build: and install: instruction simply invoke your build and installs your binary.
Regarding that question I think that if no package needs them in the official repository it’s a bit hard to justify. However you can simply put these depexts in the opam file of your project (if you have one).
Not a heavy user of OPAM so don’t know if a binary download would be acceptable or not, but I’m for one excited about Topiary so would love to see it get more exposure. I hope some way to make it available via OPAM will be found.
To be honest, I wanted to push for Tweag to write an OCaml formatter based on the principles Ormolu follows. However, as I was doing that, someone else (Tor Hovland, I believe) suggested the idea of designing a tree-sitter-based tool to write formatters. And I really like it too!
That makes a lot of sense. Not everyone wants to have a setup of Rust to compile Topiary by hand and Topiary isn’t present in any package repositories (apart from nixpkgs – you can try nix run nixpkgs#topiary if you’re a Nix user). That was also the idea behind my endeavour here: I have a bunch of friends/colleagues that I neither nixers nor rusters but that would like to try, and with which I would like to be able to work together with Topiary around.
That makes a lot of sense. and I don’t really like this option either.
To be fair, I might have discarded this option a bit too soon. I was afraid it would be a bit clunky; in particular, I was afraid about
not being able to express dependencies on version of Cargo properly,
having to deal with Cargo downloading dependencies and Cargo’s way of installing, two things that, I believe, would not fit very well in the sandboxing,
having the build involve building everything from scratch and taking a lot of space on disk simply for a rather tiny executable.
I will give it another thought and another try, though, but I’m not really convinced that this is really better than distributing binaries.
That is what I am doing at the moment indeed; I was just wondering if it made sense for me to keep those to myself or if those conf-* packages aimed at covering as many system packages as possible. Somehow it popped in my mind when writing my initial question. Thanks!
To be fair, I might have considered this option a bit too hastily. Of course it won’t work for the reasons you mentioned.
I still think it would be nice to find a way to interoperate between cargo and opam but I’m not familiar enough with cargo to say anything meaningful.
Basically we’d like some kind of depexts that hooks into cargo for preparing the build environment of your tool via opam. Don’t know if that’s possible given the integrated nature of cargo. This may actually be a good case that shows that it’s good to keep your package manager distinct from your build system :–)
Worth looking at in this context, just incase none of y’all happen to know — there’s prior art on integrating opam and <opinionated non-ocaml package manager>; see both esy and the way melange folks are doing npm packages right now. Hope that’s helpful.
fwiw a hybrid opam/cargo definitely sounds very cool; would love to see more collab between the rust/ocaml communities the same way we have so much collab between the js/ocaml communties.
As as user, I might try out topiary if it were distributed as a binary. But I’m reluctant to set up a Rust toolchain to build it from source. I’m sure you must have considered and discarded the idea of packaging it in a more traditional way e.g. Linux package repositories, Homebrew on macOS, WinGet/Chocolately on Windows? The nice thing about the traditional way is that the conf-topiary opam package would be just a tiny shim layer that would trigger the distro package install. This is how we get binary libraries now anyway…
We have considered it, yes, and we really haven’t discarded the idea! However:
It requires an impressive amount of work (Linux distributions alone are numerous, external contributions are not always easy and it requires to have people that are both Topiary-enthusiasts and users of the distributions in question).
It is more of a long-term project: even if we packaged Topiary for APT, for instance, users of Debian stable and Ubuntu LTS would not be likely to get Topiary on their machine in a long time.
The conf-* packages work well for stable products where it does not matter so much that a developer working on eg. Debian stable and another developer working on macOS with Homebrew have very different version of Topiary. For a formatting tool that might tweak the way it formats OCaml between versions, this could be a big issue.
So I still think it is going to happen eventually, but it will take a long while. In the meantime, the two package managers that I use are Nix and OPAM. I already put Topiary in nixpkgs and I was exploring the idea of distributing with OPAM, hence my intervention here. But the goal is not to push for something that is not the intended use of OPAM either, obviously, I was just wondering if it was a possibility!
Also, to clarify: I am not part of the team that develops Topiary. I am however pretty close to them (working in the same company) and I have been helping out sometimes and following the project closely just because I am very interested in it. Packaging Topiary in OPAM is on their agenda but this is mostly me trying to do that out of enthusiasm.
There is at least one package in the opam repository which has this build strategy: tezos-rust-libs. It depends on conf-rust-2021 which ensure that --edition 2021 is a valid option. This level of granularity in version dependency might not be sufficient for your use-case, but it’s not necessarily to be discarded immediately.
That is very interesting! And the problem we were mentioning of downloading dependencies in the sandbox is solved with vendoring; quite clever! It still does require people to have a Rust toolchain on their machine (with a recent enough version of Rust so that can be a problem maybe on some distributions) but that’s quite a nice solution. I will explore that.
I played around with the idea and it does seem to work quite well! For now, things are in a not-so-clean repository of mine. I could clean that up and open a PR on opam-repository if that made sense. Also, I would probably move this repository under Tweag’s organisation.
Sadly, one of the dependencies of Topiary requires Rust 1.65.0 which was only published in early November 2022, so most distributions don’t have it yet. Still, it seems to compile fine on Archlinux, Fedora and Ubuntu (LTS and latest). cf this GitHub Actions run. (Actually, in some cases, eg. CentOS, the error seems somewhat unrelated, but I don’t know those distributions well enough to understand what’s going on.)
I will try to add some smoke tests and some documentation explaining how this works and how to update (and credit tezos-rust-libs because this is clearly just a copy of their idea). After that, if we think it’s relevant and interesting, I’d move towards a PR on opam-repository. There’d be some things to figure out, and in particular whether it’d be interesting to have a conf-rust-1_65-or-above or something.
What you can do is have a conf-rust-2022/conf-rust-at-least-1-65 package which only has entries for supported distributions. With such a package:
a. People using the supported distributions will have a prompt to install the expected packages.
b. People using another distribution can still install your package but they’ll have to install the expected version of rust on their own, prior to running the opam install.
I don’t know what’s the best thing to do re: depending on precise rust versions vs depending on rust editions. I don’t know that we want to have a conf-rust-x.x.x package for each released version of rust (actually we’d need to add a -y prefix to track the version of the package rather than the version of rust). On the other hand, this might be needed for some projects. I can see several options:
A plethora of conf-rust packages. (That seems complicated for opam-repository because it would add so many conf- packages with different distribution support.)
A change in opam allowing to track version of other compiler, not just OCaml’s. (That seems complicated for opam because the scope is unclear and the feature complex.)
The package installation starts by checking the rust version and fails if unavailable (with a message along the lines of “topiary needs rust >= 1.65”). (That seems complicated for the users because opam install doesn’t just work.)
The package depends on a rust edition rather than a version. (That seems complicated for topiary because it might require some rewrites.)
Tiny note: the repository now moved under Tweag’s organisation, so you will find it here. Things seem to work just fine; on a machine with Cargo and Rustc (>= 1.65):
$ opam pin add topiary.dev https://github.com/tweag/topiary-opam.git
Package topiary does not exist, create as a NEW package? [Y/n] y
[topiary.dev] synchronised (git+https://github.com/tweag/topiary-opam.git)
topiary is now pinned to git+https://github.com/tweag/topiary-opam.git (version dev)
The following actions will be performed:
∗ install topiary dev*
Do you want to continue? [Y/n] y
<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
⬇ retrieved topiary.dev (no changes)
∗ installed topiary.dev
$ type topiary
topiary is ~/.opam/ocaml-system/bin/topiary
$ echo 'open Foo' | topiary -l ocaml
Now that the packaging in tweag/topiary-opam stabilised and seems pretty good, I have proceeded to move the discussion to a pull request on opam-repository. This should give us a bit more context thanks to the (very thorough) CI and the maybe wider audience. Thank you very much for your messages and ideas; let’s see where this leads us!
Thank you all very much for your help and advice; the PR above has now been merged and Topiary made it into OCaml packages!
According to my CI, it builds with the system version of Rust on Alpine, Archlinux, Fedora and Ubuntu (last and LTS). Users of CentOS, Debian (all kinds), OracleLinux and openSUSE will sadly have to wait a little bit. (Unless they provide Rust themselves with eg. rustup.)
Now there seems to be a tiny delay between the PR being merged and the OPAM CLI showing Topiary. Also, I have it on good authority that the Topiary team is preparing a new release, so unless you’re extremely enthusiastic about Topiary, I suggest you wait a bit before trying!
Well, this isn’t helped by the fact that OPAM literally stands for “OCaml package manager”. Granted, this could be meant to imply that it’s simply written in OCaml, but the OPAM homepage states
opam is a source-based package manager for OCaml
Having said that, I too would like to see OPAM used outside of the “OCaml ecosystem”. I doubt that will happen, though, until someone actually does use it for their C/C++/etc. project(s). Are there any examples of this out there?
Okay then. Let’s change that to: opam stands for “the Other package manager” :–)
That being said I’m not especially attached to opam. Realistically no one will likely want to have an obscure language in the dependency cone of their infrastructure. But I’d really like a single tool that offers the same kind of functionality to interact with software written in other languages.
The proliferation of language specific package managers is ridiculous, a waste of our time and a hindrance for nice language interactions. A few people pointed me to guix as being a nix without the odd configuration language (can’t tell, did not try to use nix, only read the excellent phd thesis) but sadly is not even multiplatform.