# Depending on non-OCaml languages from the opam repository
**URL:** https://discuss.ocaml.org/t/depending-on-non-ocaml-languages-from-the-opam-repository/12585
**Category:** Ecosystem
**Tags:** opam
**Created:** [July 10, 2023, 10:33am UTC](https://discuss.ocaml.org/t/depending-on-non-ocaml-languages-from-the-opam-repository/12585 "2023-07-10T10:33:40Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@avsm](https://discuss.ocaml.org/u/avsm)
#### Post date: [July 10, 2023, 10:33am UTC](https://discuss.ocaml.org/t/depending-on-non-ocaml-languages-from-the-opam-repository/12585/1 "2023-07-10T10:33:40Z")
I’m currently reviewing feature requests for the opam repository, and one of the most common ones is for us to **support non-OCaml toolchains as dependencies from OCaml packages submitted to our package repository**. In recent years, there have been a number of OCaml libraries that depend on Rust, Python or Node, and cannot be easily tested in our automated infrastructure (which currently uses a fixed base image per distribution).
I’ve put together a prototype way we might solve this easily, without taking on the burden of maintaining non-OCaml toolchains ourselves with limited maintainer resources. Opinions and ideas welcome on this thread, and the repository is at: [GitHub - avsm/opam-lang-repo: Install various language development tools via opam](https://github.com/avsm/opam-lang-repo)
## A multi-language devcontainer package repository
[Devcontainers](https://containers.dev) are an emerging mechanism to use container runtimes as a full-fledged development environment. They can support multiple programming languages in one filesystem by means of [features](https://containers.dev/implementors/features/), which allow for the activation of a given toolchain alongside others. For example, using features allows for the simultaneous use of Python, Rust and OCaml within one container image, whereas with traditional devcontainers there would be a separate container for each toolchain.
## Using the opam solver to manage feature selection
The [opam](https://opam.ocaml.org) package manager integrates a builtin constraint solver that allows for the selection of a compatible set of dependencies from a package repository that contains all released versions of all packages.
This repository translates published devcontainers into opam packages, such that devcontainer features can be selected by simply adding dependencies to an opam package. Additionally, version constraints on the desired tooling can be added to pick the required versions. For example:
```auto
depends: [ "dev-rust" {>="1.68"}
"dev-ocaml" {>="4.12" & < "5.0"}
"dev-python"
"dev-python-optimize" ]
```
This picks a version of Rust greater than 1.68, and any OCaml compiler between 4.12-4.14, and any Python compiler with the `optimize` flag activated for more efficient code generation.
## The Good News
This solution frees the opam-repo maintainers from having to support the myriad other toolchains, and lets us depend on them freely from opam. By adding explicit dependencies like this, we can continue to run automated end-to-end tests for new and existing packages in the OCaml ecosystem, even when they do not exclusively use OCaml.
## The Bad News
There are still some limitations to figure out before this is production worthy:
- The devcontainer installation busts the opam security sandbox, and so cannot be installed simultaneously with normal packages. It would be ok in a CI system where sandboxing is normally disabled. Another option is for these packages to not actually perform the installation, but generate a single `install.sh` with all the right environment variables. An image generator could then run that script to generate a base image.
- opam doesn’t currently support composing remote repositories, so some strategy is needed for how to keep this generated repo in sync with anything included in the central repository.
- Need to support devcontainer boolean defaults correctly (e.g. Python feature), and figure out what to do about arbitrary string options. Env variables could be used to pass in values, but opam can’t recompile if these variables change. Dune does support systematic env variable tracking and recompile if it changes, so this would work in a monorepo.
- Need to extract feature dependencies into the opam formula as well.
- Something, something, Nix, instead?
---
### Author:  [@raphael-proust](https://discuss.ocaml.org/u/raphael-proust)
#### Post date: [July 10, 2023, 12:53pm UTC](https://discuss.ocaml.org/t/depending-on-non-ocaml-languages-from-the-opam-repository/12585/2 "2023-07-10T12:53:09Z")
> [@avsm](#):
>
> This repository translates published devcontainers into opam packages, such that devcontainer features can be selected by simply adding dependencies to an opam package. Additionally, version constraints on the desired tooling can be added to pick the required versions.
A feature of Archlinux packages is that the version of the library/software/w/e being installed is tracked separately from the version of the package installing it. E.g., [this python3 package](https://gitlab.archlinux.org/archlinux/packaging/packages/python/-/blob/3.11.3-2/PKGBUILD?ref_type=tags) is for Python version 3.11.3 but it is the second such package (hence the `2` in the package name and also in the `pkgrel` field in the package file).
This is useful for decorelating software changes and packaging changes. And I think it could possibly be useful in this case where we have packages to describe some pieces of software that have their own versioning.
Would it be useful for devcontainers? Maybe even for some `conf-` packages?
IIRC opam notion of version (e.g., what’s a valid version number, how to compare version numbers) is based on Debian. Is that the case? Still the case? How would a `pkgrel` work if that’s the case?
---
### Author:  [@avsm](https://discuss.ocaml.org/u/avsm)
#### Post date: [July 10, 2023, 1:14pm UTC](https://discuss.ocaml.org/t/depending-on-non-ocaml-languages-from-the-opam-repository/12585/3 "2023-07-10T13:14:56Z")
> [@raphael-proust](#):
>
> A feature of Archlinux packages is that the version of the library/software/w/e being installed is tracked separately from the version of the package installing it. E.g., [this python3 package](https://gitlab.archlinux.org/archlinux/packaging/packages/python/-/blob/3.11.3-2/PKGBUILD?ref_type=tags) is for Python version 3.11.3 but it is the second such package (hence the `2` in the package name and also in the `pkgrel` field in the package file).
That’s a good question, and important to make the whole thing reproducible. In the case of devcontainer descriptions, there is a separate version number for the actual installation script (e.g. [see this for Python](https://github.com/devcontainers/features/blob/main/src/python/devcontainer-feature.json#L3)), and really ought to be tracked somewhere.
The versions we are exposing to users of the repository will _not_ be this version, but rather the version of the programming languages themselves, much as we do with `ocaml {>= "4.14"}` today in the opam-repository.
> [@raphael-proust](#):
>
> IIRC opam notion of version (e.g., what’s a valid version number, how to compare version numbers) is based on Debian. Is that the case? Still the case? How would a `pkgrel` work if that’s the case?
Solving this would also solve the “package revisionism” issue, but would require a much bigger change to the opam repository. It needs to happen, but probably separately from this thread. See [package revisionism can break things · Issue #10531 · ocaml/opam-repository · GitHub](https://github.com/ocaml/opam-repository/issues/10531) for a related issue.
I don’t have a good answer for how to track the meta-versions yet without manually tracking epochs (which seems error prone)… it’s something that is worth all of us collectively thinking about though.
---
### Author:  [@samoht](https://discuss.ocaml.org/u/samoht)
#### Post date: [July 11, 2023, 8:45am UTC](https://discuss.ocaml.org/t/depending-on-non-ocaml-languages-from-the-opam-repository/12585/4 "2023-07-11T08:45:52Z")
> This repository translates published devcontainers into opam packages, such that devcontainer features can be selected by simply adding dependencies to an opam package. Additionally, version constraints on the desired tooling can be added to pick the required versions.
This is a very nice trick! Do features already contain such constraints, or is it something that is expected only for end-users of these features?
I also see that you are using a lot of `depots` and `conf` packages. It’s a fine use of these features (similar to the way we do this in `opam-repository` for configuring the OCaml compiler) but I’m wondering if you could simplify some of this by using an obscure feature of `opam`: the ability to set build variables (either globally or per package) via the environment via setting up `OPAMVAR_` (or `OPAMVAR__`.
For instance, instead of [this](https://github.com/avsm/opam-lang-repo/blob/main/packages/dev-python/dev-python.3.11/opam#L11):
```auto
install: [
"env"
"VERSION=%{version}%"
"INSTALL_TOOLS=%{dev-python-install-tools:version}%" {dev-python-install-tools:installed}
"OPTIMIZE=%{dev-python-optimize:version}%" {dev-python-optimize:installed}
"INSTALL_PATH=%{dev-python-install-path:version}%" {dev-python-install-path:installed}
"INSTALL_JUPYTERLAB=%{dev-python-install-jupyterlab:version}%" {dev-python-install-jupyterlab:installed}
"CONFIGURE_JUPYTERLAB_ALLOW_ORIGIN=%{dev-python-configure-jupyterlab-allow-origin:version}%" {dev-python-configure-jupyterlab-allow-origin:installed}
"HTTP_PROXY=%{dev-python-http-proxy:version}%" {dev-python-http-proxy:installed}
"bash" "./src/python/install.sh"
]
```
You could set-up an environment by scrapping [devcontainer-feature.json](https://github.com/devcontainers/features/blob/c4ffeebdacc3c7c020ee7c286aae88959aa91671/src/python/devcontainer-feature.json#L34):
```shell
OPAMVAR_dev-python-install-path_install-tools=$(jq .options.installTools.default < devcontainer-feature.json)
OPAMVAR_dev-python-install-path_install-path=$(jq .options.installPath.default < devcontainer-feature.json)
opam [...]
```
I don’t know if it’s much simpler here, but i found this trick useful in the past when writing scripts. Also I’n not totally sure why you are encoding this variable into its `version` field. Couldn’t you just use a string and a package-level variable?
---
### Author:  [@Konstantin\_Olkhovski](https://discuss.ocaml.org/u/Konstantin_Olkhovski)
#### Post date: [July 12, 2023, 8:37am UTC](https://discuss.ocaml.org/t/depending-on-non-ocaml-languages-from-the-opam-repository/12585/5 "2023-07-12T08:37:26Z")
Devcontainers potentially solves the CI issue, but it does not seem to solve a problem of integrating opam with language-specific package managers. Currently using two Rust binding libraries in one binary will just explode at linking phase. See [Cargo/dune integration](https://discuss.ocaml.org/t/cargo-dune-integration/12484) and [Cargo/Opam packaging of a Rust/OCaml project - #9 by Konstantin\_Olkhovski](https://discuss.ocaml.org/t/cargo-opam-packaging-of-a-rust-ocaml-project/5743/9).
At least for Rust case it would probably be helpful to automatically create local repository of Rust crates, that are coming as part of opam packages, so that when I install an opam package with Rust bits, my project build rules can use that cargo repo, generated by opam, and use specific versions of Rust crates, so that bindings work as expected.
For Python probably some other approach is required. At least some metadata in opam files should be present to encode external Python or Rust dependencies, or presence of Python or Rust source code artifacts within the opam package, so that some tools can build on top of that metadata, get the pieces of Python/Rust code to appropriate places (local repo, etc) so that they are available during build of current project/running tests.
---
### Author:  [@ryang](https://discuss.ocaml.org/u/ryang)
#### Post date: [July 14, 2023, 2:37pm UTC](https://discuss.ocaml.org/t/depending-on-non-ocaml-languages-from-the-opam-repository/12585/6 "2023-07-14T14:37:45Z")
> [@avsm](#):
>
> Something, something, Nix, instead?
Nix is really good for multi-lingual projects. I’ve created a proof of concept for opam to use Nix to provide non-OCaml dependencies: [github.com/RyanGibb/opam-lang-repo-nix](https://github.com/RyanGibb/opam-lang-repo-nix).
You can add it to opam with `opam repo add opam-lang-repo-nix git+https://github.com/RyanGibb/opam-lang-repo-nix.git` and add a dependency to your opam file as, e.g. `nix-rustc`.
The opam files looks something like:
```auto
opam-version: "2.0"
synopsis: "Rust"
install: [
"nix-env"
"-iA"
"rustc"
"-f"
"https://github.com/NixOS/nixpkgs/archive/28e0126876d688cf5fd15da1c73fbaba256574f0.tar.gz"
]
```
Nix doesn’t, in general, package all the versions of a package ([github.com/NixOS/nixpkgs/issues/9682](https://github.com/NixOS/nixpkgs/issues/9682)). You can install a previous version by using a specific git revision of Nixpkgs. I’ve cherry-picked some git revisions using [lazamar.co.uk/nix-versions/](https://lazamar.co.uk/nix-versions/) to get an MVP working for the rust compiler and python interpreter. This is where the `28e0126876d688cf5fd15da1c73fbaba256574f0` you see above is from.
Some advantages:
1. Reproducible and deterministic dependencies. I don’t know a lot about devcontainers, so please correct me if I’m wrong, but it looks like they just run a large number of large ‘install’ bash scripts. I can imagine the user loosing track of the side-effects of these scripts, and the exact versions of the dependencies they pull in, and not being able to easily reproduce their development environment.
2. Benefit from the huge number of packages in Nixpkgs.
3. Allows multiple versions of a package to coexist on the same system without any interference. If the devcontainer install scripts are sandboxed/containerized (again I’m hazy on the details), then another advantage of Nix is it allows transparent sharing of identical dependencies.
4. Atomic upgrades and rollbacks, which again come for for free with Nix.
Some issues:
1. This requires nix to be installed. We could have opam install nix as well.
2. It similarly requires sandboxing to be disabled. We could pull in the Nixpkgs revision in advance but Nix will still have to pull in the sources for the derivation closure it’s building.
3. This uses `nix-env` to install dependencies to the local user’s profile. While this is less side-effectful than install scripts, as dependencies are still constrained to the Nix store, it would be more desirable to create a shell in which these dependencies are available. Nix flake `devShells` could be useful.
4. Similar to:
> [@avsm](#):
>
> opam doesn’t currently support composing remote repositories, so some strategy is needed for how to keep this generated repo in sync with anything included in the central repository.
- We would want a way to keep this repository up-to-date with upstream Nixpkgs. The project I linked to, [github.com/lazamar/nix-package-versions](https://github.com/lazamar/nix-package-versions), is already doing something very similar in keeping their website up-to-date.
A potential solution to issues 1, 2, and 3, would be for opam to generate a `flake.nix` file that contains all the dependencies. If we wanted to take this even further, opam could also generate nix derivations for packages from opam-repository. This could be through of as the inverse approach to something like github.com/tweag/opam-nix.
---
### Author:  [@pmonson711](https://discuss.ocaml.org/u/pmonson711)
#### Post date: [July 17, 2023, 1:59pm UTC](https://discuss.ocaml.org/t/depending-on-non-ocaml-languages-from-the-opam-repository/12585/7 "2023-07-17T13:59:00Z")
While I do think Nix would be a great solution for Linux and CI needs. I do worry that it would quickly become a hindrance for any other system. I know NetBSD and FreeBSD both have some Nix support, but every time I’ve attempted to use them the experience just hasn’t been worth it.
Sadly I suspect the devcontainers to have similar problems.
I don’t believe it’s a general solution to this problem, but for development [asdf-vm](https://asdf-vm.com/) has always worked well for me. I hope that whatever is tried I have an escape hatch to something similar on unsupported systems.