Two questions about what is appropriate to package with OPAM

Regarding that question I think that if no package needs them in the official repository it’s a bit hard to justify. However you can simply put these depexts in the opam file of your project (if you have one).

2 Likes

Not a heavy user of OPAM so don’t know if a binary download would be acceptable or not, but I’m for one excited about Topiary so would love to see it get more exposure. I hope some way to make it available via OPAM will be found.

Further off-topic comment: I already starting playing with Topiary and I like its approach a lot. I have wanted to auto-format our codebase (at LexiFi) for a long time but ocamlformat just doesn’t cut it for us. As soon as Reduce need for query file parsing, to improve performance · Issue #443 · tweag/topiary · GitHub gets resolved I plan to start using it more seriously. Keep up the good work!

Cheers,
Nicolas

To be honest, I wanted to push for Tweag to write an OCaml formatter based on the principles Ormolu follows. However, as I was doing that, someone else (Tor Hovland, I believe) suggested the idea of designing a tree-sitter-based tool to write formatters. And I really like it too!

That makes a lot of sense. Not everyone wants to have a setup of Rust to compile Topiary by hand and Topiary isn’t present in any package repositories (apart from nixpkgs – you can try nix run nixpkgs#topiary if you’re a Nix user). That was also the idea behind my endeavour here: I have a bunch of friends/colleagues that I neither nixers nor rusters but that would like to try, and with which I would like to be able to work together with Topiary around.

That makes a lot of sense. and I don’t really like this option either.

To be fair, I might have discarded this option a bit too soon. I was afraid it would be a bit clunky; in particular, I was afraid about

  • not being able to express dependencies on version of Cargo properly,
  • having to deal with Cargo downloading dependencies and Cargo’s way of installing, two things that, I believe, would not fit very well in the sandboxing,
  • having the build involve building everything from scratch and taking a lot of space on disk simply for a rather tiny executable.

I will give it another thought and another try, though, but I’m not really convinced that this is really better than distributing binaries.

That is what I am doing at the moment indeed; I was just wondering if it made sense for me to keep those to myself or if those conf-* packages aimed at covering as many system packages as possible. Somehow it popped in my mind when writing my initial question. Thanks!

To be fair, I might have considered this option a bit too hastily. Of course it won’t work for the reasons you mentioned.

I still think it would be nice to find a way to interoperate between cargo and opam but I’m not familiar enough with cargo to say anything meaningful.

Basically we’d like some kind of depexts that hooks into cargo for preparing the build environment of your tool via opam. Don’t know if that’s possible given the integrated nature of cargo. This may actually be a good case that shows that it’s good to keep your package manager distinct from your build system :–)

2 Likes

Worth looking at in this context, just incase none of y’all happen to know — there’s prior art on integrating opam and <opinionated non-ocaml package manager>; see both esy and the way melange folks are doing npm packages right now. Hope that’s helpful.

fwiw a hybrid opam/cargo definitely sounds very cool; would love to see more collab between the rust/ocaml communities the same way we have so much collab between the js/ocaml communties. :heart:

1 Like

I just had to chime in and second this! I’ll definitely be keeping an eye on topiary as it develops. :star_struck:

As as user, I might try out topiary if it were distributed as a binary. But I’m reluctant to set up a Rust toolchain to build it from source. I’m sure you must have considered and discarded the idea of packaging it in a more traditional way e.g. Linux package repositories, Homebrew on macOS, WinGet/Chocolately on Windows? The nice thing about the traditional way is that the conf-topiary opam package would be just a tiny shim layer that would trigger the distro package install. This is how we get binary libraries now anyway…

We have considered it, yes, and we really haven’t discarded the idea! However:

  • It requires an impressive amount of work (Linux distributions alone are numerous, external contributions are not always easy and it requires to have people that are both Topiary-enthusiasts and users of the distributions in question).

  • It is more of a long-term project: even if we packaged Topiary for APT, for instance, users of Debian stable and Ubuntu LTS would not be likely to get Topiary on their machine in a long time.

  • The conf-* packages work well for stable products where it does not matter so much that a developer working on eg. Debian stable and another developer working on macOS with Homebrew have very different version of Topiary. For a formatting tool that might tweak the way it formats OCaml between versions, this could be a big issue.

So I still think it is going to happen eventually, but it will take a long while. In the meantime, the two package managers that I use are Nix and OPAM. I already put Topiary in nixpkgs and I was exploring the idea of distributing with OPAM, hence my intervention here. But the goal is not to push for something that is not the intended use of OPAM either, obviously, I was just wondering if it was a possibility!

Also, to clarify: I am not part of the team that develops Topiary. I am however pretty close to them (working in the same company) and I have been helping out sometimes and following the project closely just because I am very interested in it. Packaging Topiary in OPAM is on their agenda but this is mostly me trying to do that out of enthusiasm.

There is at least one package in the opam repository which has this build strategy: tezos-rust-libs. It depends on conf-rust-2021 which ensure that --edition 2021 is a valid option. This level of granularity in version dependency might not be sufficient for your use-case, but it’s not necessarily to be discarded immediately.

1 Like

That is very interesting! And the problem we were mentioning of downloading dependencies in the sandbox is solved with vendoring; quite clever! It still does require people to have a Rust toolchain on their machine (with a recent enough version of Rust so that can be a problem maybe on some distributions) but that’s quite a nice solution. I will explore that.

1 Like

I played around with the idea and it does seem to work quite well! For now, things are in a not-so-clean repository of mine. I could clean that up and open a PR on opam-repository if that made sense. Also, I would probably move this repository under Tweag’s organisation.

Sadly, one of the dependencies of Topiary requires Rust 1.65.0 which was only published in early November 2022, so most distributions don’t have it yet. Still, it seems to compile fine on Archlinux, Fedora and Ubuntu (LTS and latest). cf this GitHub Actions run. (Actually, in some cases, eg. CentOS, the error seems somewhat unrelated, but I don’t know those distributions well enough to understand what’s going on.)

I will try to add some smoke tests and some documentation explaining how this works and how to update (and credit tezos-rust-libs because this is clearly just a copy of their idea). After that, if we think it’s relevant and interesting, I’d move towards a PR on opam-repository. There’d be some things to figure out, and in particular whether it’d be interesting to have a conf-rust-1_65-or-above or something.

What do you guys think?

What you can do is have a conf-rust-2022/conf-rust-at-least-1-65 package which only has entries for supported distributions. With such a package:

a. People using the supported distributions will have a prompt to install the expected packages.
b. People using another distribution can still install your package but they’ll have to install the expected version of rust on their own, prior to running the opam install.

I don’t know what’s the best thing to do re: depending on precise rust versions vs depending on rust editions. I don’t know that we want to have a conf-rust-x.x.x package for each released version of rust (actually we’d need to add a -y prefix to track the version of the package rather than the version of rust). On the other hand, this might be needed for some projects. I can see several options:

  • A plethora of conf-rust packages. (That seems complicated for opam-repository because it would add so many conf- packages with different distribution support.)
  • A change in opam allowing to track version of other compiler, not just OCaml’s. (That seems complicated for opam because the scope is unclear and the feature complex.)
  • The package installation starts by checking the rust version and fails if unavailable (with a message along the lines of “topiary needs rust >= 1.65”). (That seems complicated for the users because opam install doesn’t just work.)
  • The package depends on a rust edition rather than a version. (That seems complicated for topiary because it might require some rewrites.)

Anyone has other ideas?

Tiny note: the repository now moved under Tweag’s organisation, so you will find it here. Things seem to work just fine; on a machine with Cargo and Rustc (>= 1.65):

$ opam pin add topiary.dev https://github.com/tweag/topiary-opam.git
Package topiary does not exist, create as a NEW package? [Y/n] y
[topiary.dev] synchronised (git+https://github.com/tweag/topiary-opam.git)
topiary is now pinned to git+https://github.com/tweag/topiary-opam.git (version dev)

The following actions will be performed:
  ∗ install topiary dev*
Do you want to continue? [Y/n] y

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
⬇ retrieved topiary.dev  (no changes)
∗ installed topiary.dev
Done.

$ type topiary
topiary is ~/.opam/ocaml-system/bin/topiary

$ echo 'open     Foo' | topiary -l ocaml
open Foo
2 Likes

Now that the packaging in tweag/topiary-opam stabilised and seems pretty good, I have proceeded to move the discussion to a pull request on opam-repository. This should give us a bit more context thanks to the (very thorough) CI and the maybe wider audience. Thank you very much for your messages and ideas; let’s see where this leads us!

1 Like

Thank you all very much for your help and advice; the PR above has now been merged and Topiary made it into OCaml packages!

According to my CI, it builds with the system version of Rust on Alpine, Archlinux, Fedora and Ubuntu (last and LTS). Users of CentOS, Debian (all kinds), OracleLinux and openSUSE will sadly have to wait a little bit. (Unless they provide Rust themselves with eg. rustup.)

Now there seems to be a tiny delay between the PR being merged and the OPAM CLI showing Topiary. Also, I have it on good authority that the Topiary team is preparing a new release, so unless you’re extremely enthusiastic about Topiary, I suggest you wait a bit before trying!

4 Likes

Concur strongly. I will go to my grave insisting that OPAM is the source package manager that should be adopted by the C++ community, which badly needs a good one and only has multiple terrible ones.

Well, this isn’t helped by the fact that OPAM literally stands for “OCaml package manager”. Granted, this could be meant to imply that it’s simply written in OCaml, but the OPAM homepage states

opam is a source-based package manager for OCaml

Having said that, I too would like to see OPAM used outside of the “OCaml ecosystem”. I doubt that will happen, though, until someone actually does use it for their C/C++/etc. project(s). Are there any examples of this out there?

Okay then. Let’s change that to: opam stands for “the Other package manager” :–)

That being said I’m not especially attached to opam. Realistically no one will likely want to have an obscure language in the dependency cone of their infrastructure. But I’d really like a single tool that offers the same kind of functionality to interact with software written in other languages.

The proliferation of language specific package managers is ridiculous, a waste of our time and a hindrance for nice language interactions. A few people pointed me to guix as being a nix without the odd configuration language (can’t tell, did not try to use nix, only read the excellent phd thesis) but sadly is not even multiplatform.

2 Likes

Just FYI, I was able to opam install topiary this afternoon without a hitch!

I’m looking forward to tinkering around with customizing the default OCaml query(ies), maybe as an ocamlformat replacement. Are you familiar with any already-extant queries that satisfy particular styles?

(FYI, Topiary Playground is currently erroring for all of the languages except for JSON, which was not the case a few weeks ago.)

Glad to read that! In the meanwhile, Topiary v0.2.0 got released and made its way into OPAM as well. It comes with some improvements on the OCaml support so you might want to upgrade before playing too much with it!

At this point, the only query file I know for OCaml is the one shipped with Topiary itself and I am not aware of other ones, but I’m sure they will show up eventually! Looking forward to what you can do with it : -)

It’s since been tracked and fixed. A stupid mistake really (more bugs are anyways). Thank you for reporting!

1 Like