This question is maybe more related to NixOS than to OCaml, but I’m asking here, because it’s also about opam use in NixOS, and for sure, some of you will know.
I managed to install and use conf-libclang on NixOS. But for this I had to write a shell.nix and use nix-shell in order to have the needed libs and pkg-config available, even if those libraries are installed declarativalely from my configuration.nix file.
I also have to use opam from this nix-shell when I need to use conf-libclang (by instance, when I’m using clangml).
Is it the only way? Do I really have to remember that I should start the right nix-shell depending on the opam packages I’m using?
It sounds like you are wanting to use libraries from your system instead of building a derivation? It’s a little unclear to me.
Sometimes stuff like this will occur since some tools are coded to expect libraries in /usr/lib but this isn’t where Nix puts them. Similar issues occur with those using shebangs like #!/bin/bash instead of #!/usr/bin/env bash.
In general you want a clean slate, self-contained environment for building Nix derivations & never relying on host system. You can use something like inputsFrom = builtins.attrValues self.packages.${system}; in your Flake’s devShells area to get the inputs from all of your packages & nix develop (or with direnv, echo "use flake" > .envrc) will get you in an environment with needed libraries. Even then, when building in Nix you should prefer nix build … with a package using something like buildDunePackage for a stateless build instead of to doing anything in the mutable shell if you can help it otherwise you’re not really getting the benefits of Nix.
That is to say, you should build yourself a package, a flake.nix or default.nix in your project directory, & the derivation for that package should contain inputs for libclang or whatever else you need, & then run nix build … or nix-build. There is also opam-nix <GitHub - tweag/opam-nix: Turn opam-based OCaml projects into Nix derivations> but I’ve never used it, instead opting for what nixpkgs currated for me in say pkgs.ocaml-ng.ocamlPackages_5_1. There are some OCaml docs in nixpkgs @ doc/languages-frameworks/ocaml.section.md.
Building derivations isn’t difficult when you get the hang of them… my last OCaml+Nix hurdle tho is figuring out how to get the LSP to read from the Nix store instead of needing to run :!dune from Neovim which not only duplicates the efforts of nix build & inside the mutable shell, but I’ve not yet looked into it.
It sounds like you are wanting to use libraries from your system instead of building a derivation? It’s a little unclear to me.
Well. Since I had the libraries installed on my system, I thought I could use them from my development environment. I was wrong, apparently.
But I was able to use those libraries from a nix-shell where the corresponding packages (and pkg-config) were included. I certainly could remove those packages from the system configuration.nix.
Sometimes stuff like this will occur since some tools are coded to expect libraries in /usr/lib but this isn’t where Nix puts them. Similar issues occur with those using shebangs like #!/bin/bash instead of #!/usr/bin/env bash.
Interesting. I have also added this issue: https://github.com/thierry-martinez/clangml/issues/10 because one .sh file used during this package building is not working well in NixOS because it has #!/bin/bash instead of #!/usr/bin/env bash.
I do know nothing about flakes, and I’m not sure to understand why I would have to build a “derivation”, or the benefits?
Usually, I have one opam switch on my dev machine. And some of the installed opam packages are broken if I do not start from the right nix-shell.
Maybe, I should simply use local opam switches when this kind of packages are involved in a project + learn and use direnv to assure the nix-shell is started when in the project directory?
This is the Nix output that gets stored in the /nix/store. The advantage is that it’s built in a stateless environment of the Nix sandbox so it can be deemed “reproducible” (as opposed to statefully with make or dune stateful to the environment of your machine with system time, network requests, filesystem, etc. contributing to that state). This also now lets you tap into the other parts of Nix such as builders & substitutors. Any Nix build can specify one or many builders when you run nix build which means you can build packages in parallel (the derivation result is then downloaded to the host machine). Substitutors allow one or many users on the same architecture to share that derivation so that not ever user needs to be rebuilding something that’s already been built (& since it was done in the Nix sandbox & hashed, it can be substituted in just like memoization). In a traditional developer setting, $DEV_1 might be expected to pull the latest patchset & then rebuild in their machine despite $DEV_2 who is also running x86_64-linux having already built it successfully which wastes resources.
I have a mini PC at home running the same architecture as my laptop–when I run Nix build it will be split across both machines to speed up the process or if I’m trying to run my laptop to maximize for battery life but I have access to stable WiFi, I sometimes opt to build only on the server to not churn my laptop CPU+battery. Similarly since they are set to be mutual substitutors, when I build something with compile/feature flags not available from upstream Hydra for nixpkgs I can build it on one device & when I need to build it on the other, it will just be downloaded as Nix by default checks its substitutors to see if the package has already been built.
Using direnv + nix develop doesn’t get you out of your stateful environment. Nix is more than an NixOS since Nix the build tool can do things other tools don’t do by default. (The shameless plug: I wrote a post about how/why one might think about building a project in Nix–where devShells are mentioned near the end as a low-importance feature)
nixos by design doesn’t expose libraries specified in configuration.nix (headers, archives, and pkgconfig files, dev stuff) to the system environment, only shared objects and executables (runtime stuff) of those libraries if available. you can verify this by running find on /run/current-system.
derivations (including shells) are the only way to access library development files as far as I’m aware. a shell-based workflow is also what I use.
see: FAQ/I installed a library but my compiler is not finding it. Why? - NixOS Wiki
I’ve found out that using Opam during development is easier than using Nix for installing libraries.
Nixpkgs doesn’t define most libraries and is not uptodate as the OCaml libraries there are only intended to support the OCaml tools that update less regularly.
and is not uptodate as the OCaml libraries there are only intended to support the OCaml tools that update less regularly
It’s a lot of work to maintain packages, but they’ll accept any update from contributors like you to do a version bump. Aside things breaking, nixpkgs wants packages to be up-to-date as well as accepting packages even if they don’t have a tool/depedency inside the system.
I managed to install and use conf-libclang on NixOS. But for this I had to write a shell.nix and use nix-shell in order to have the needed libs and pkg-config available, even if those libraries are installed declaratively from my configuration.nix file.
nix-shell and it’s descendant nix develop set environment variables such as NIX_LDFLAGS which will support finding object files for clang. E.g.
I agree with @toastal that the Nix-native way of doing this would be to build it in a Nix derivation, but I also agree with @Juloo that sometimes just using Opam directly is the easiest to get up and running with a project.
I’ve created a PR for opam that should solve this issue with a Nix depext (system dependency) backend that has opam env managing these environment variables.
Aside things breaking, nixpkgs wants packages to be up-to-date as well as accepting packages even if they don’t have a tool/depedency inside the system.