Installing camlp5 on ocaml 5.0.0 / nixos

Anyone having problem installing camlp5 via opam on nixos?

[nix-shell:~]$ nix-shell -p pcre pcre.dev --run "ocaml --version; echo '=='; pkg-config libpcre; echo '=='; opam install camlp5 -y"
The OCaml toplevel, version 5.0.0
==
==
[NOTE] External dependency handling not supported for OS family 'nixos'.
       You can disable this check using 'opam option --global depext=false'
The following actions will be performed:
  ∗ install conf-libpcre        1       [required by pcre]
  ∗ install not-ocamlfind       0.09    [required by camlp5-buildscripts]
  ∗ install pcre                7.5.0   [required by camlp5]
  ∗ install camlp5-buildscripts 0.01    [required by camlp5]
  ∗ install camlp5              8.00.05
===== ∗ 5 =====

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
⬇ retrieved camlp5.8.00.05  (cached)
⬇ retrieved camlp5-buildscripts.0.01  (cached)
⬇ retrieved not-ocamlfind.0.09  (cached)
[ERROR] The compilation of conf-libpcre.1 failed at "pkg-config libpcre".
⬇ retrieved pcre.7.5.0  (cached)

I’m trying to figure out how to fix this; the problem appears to be pkg-config / pcre related.

1 Like

Hi, the problem is with that prereq. So try opam install conf-libpcre and that should also fail. Then you can back-track from there to figure out why it isn’t working.

1 Like

You’re right. conf-libpcre failed:

[nix-shell:~]$ opam install conf-libpcre
[NOTE] External dependency handling not supported for OS family 'nixos'.
       You can disable this check using 'opam option --global depext=false'
The following actions will be performed:
  ∗ install conf-libpcre 1

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
[ERROR] The compilation of conf-libpcre.1 failed at "pkg-config libpcre".

#=== ERROR while compiling conf-libpcre.1 =====================================#
# context     2.1.3 | linux/x86_64 | ocaml-base-compiler.5.0.0 | https://opam.ocaml.org#21f454ea
# path        ~/.opam/default/.opam-switch/build/conf-libpcre.1
# command     ~/.opam/opam-init/hooks/sandbox.sh build pkg-config libpcre
# exit-code   1
# env-file    ~/.opam/log/conf-libpcre-3414674-676fac.env
# output-file ~/.opam/log/conf-libpcre-3414674-676fac.out



<><> Error report <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
┌─ The following actions failed
│ λ build conf-libpcre 1
└─ 
╶─ No changes have been performed

[nix-shell:~]$ cat ~/.opam/log/conf-libpcre-3414674-676fac.out 

[nix-shell:~]$ 

Trying to run that build line manually:

[nix-shell:~]$ ~/.opam/opam-init/hooks/sandbox.sh build pkg-config libpcre
bwrap: execvp pkg-config: No such file or directory

[nix-shell:~]$ pkg-config
Must specify package names on the command line

[nix-shell:~]$ which pkg-config
/home/y/.nix-profile/bin/pkg-config

conf-libpcre now installed. Issue is now with not-ocamlfind

[nix-shell:~/t0]$ opam install conf-libpcre
[NOTE] External dependency handling not supported for OS family 'nixos'.
       You can disable this check using 'opam option --global depext=false'
[NOTE] Package conf-libpcre is already installed (current version is 1).

[nix-shell:~/t0]$ opam install not-ocamlfind
[NOTE] External dependency handling not supported for OS family 'nixos'.
       You can disable this check using 'opam option --global depext=false'
The following actions will be performed:
  ∗ install not-ocamlfind 0.09

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
⬇ retrieved not-ocamlfind.0.09  (cached)
[ERROR] The compilation of not-ocamlfind.0.09 failed at "./configure -bindir /home/y/.opam/default/bin -sitelib /home/y/.opam/default/lib -mandir /home/y/.opam/default/man -config
        /home/y/.opam/default/lib/findlib.conf -no-custom".

#=== ERROR while compiling not-ocamlfind.0.09 =================================#
# context     2.1.3 | linux/x86_64 | ocaml-base-compiler.5.0.0 | https://opam.ocaml.org#21f454ea
# path        ~/.opam/default/.opam-switch/build/not-ocamlfind.0.09
# command     ~/.opam/opam-init/hooks/sandbox.sh build ./configure -bindir /home/y/.opam/default/bin -sitelib /home/y/.opam/default/lib -mandir /home/y/.opam/default/man -config /home/y/.opam/default/lib/findlib.conf -no-custom
# exit-code   1
# env-file    ~/.opam/log/not-ocamlfind-3735263-662702.env
# output-file ~/.opam/log/not-ocamlfind-3735263-662702.out
### output ###
# bwrap: execvp ./configure: No such file or directory



<><> Error report <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
┌─ The following actions failed
│ λ build not-ocamlfind 0.09
└─ 
╶─ No changes have been performed

[nix-shell:~/t0]$ cat ~/.opam/log/not-ocamlfind-3735263-662702.out 
bwrap: execvp ./configure: No such file or directory

Two steps:

  1. is /bin/bash installed on your machine? b/c that’s what the “configure” script uses.

  2. You can unpack the source-tarball yourself (opam source not-ocamlfind) and then you can execute the build-steps yourself (you can get the opam file from opam-repository (on github), or there’s typically a nearly-up-to-date opam file in each source tarball).

What I’m saying is, if the arms-length of using opam to run your build-step fails, you can get closer and see what’s going wrong.

1 Like

BTW, this is could get pretty unpleasant unless you’re wanting to do only pure-ocaml work.

1 Like
which bash
/nix/store/k2b87dvmz4mbs3d29z9fy18ja874q4nf-bash-interactive-5.1-p16/bin/bash

Creating a symlink to that from /bin/bash appears to have solved. the problem. not-ocamlfind & camlp5 installed now.

Thanks everyone!

This could probably be reported to the maintainer of the configure script, as using the hardcoded path /bin/bash is discouraged nowadays in favor of /usr/bin/env bash.

I’m the maintainer, and … I have never seen such guidance. I mean … if /bin/bash can’t be counted-upon, why should we assume that /usr/bin/env can be counted-upon for that matter?

ETA: after all, /usr/bin/env is much, much, much younger than /bin/sh, /bin/csh, and probably /bin/bash. If the answer were “use /bin/sh”, I’d be OK with that, I guess. But /usr/bin/env ?

3 Likes

Ditto. I’ve never heard that /usr/bin/env bash was encouraged over /bin/bash. And I’ve found the /usr/bin/env formulation to be especially problematic when running in Windows bash shells because random user’s PATHs are nightmares. Specifically, you can be in one bash shell, and within that run a #!/usr/bin/env bash shebang script … you accidentally pull in another bash (Git Bash / Chocolately bash / sqoop bash / Cygwin bash / MSYS2 bash). You will get very difficult to understand problems, or you will get some DLL conflict. In contrast /bin/bash or /bin/sh always resolves correctly within a shell. Example bug: Scoop bash.exe (etc.) conflicts with v1.2.0-prerel7 Installer · Issue #34 · diskuv/dkml-installer-ocaml · GitHub

My go to references are:

  • POSIX specifications which say not to hardcode /bin/sh but to use either command -v sh or getconf PATH at install time. No mention of /usr/bin/env.
  • shellcheck is usually pretty compliant with real-world shells. It accepts both /bin/bash and /usr/bin/env bash.
1 Like

As a datapoint of n=1, I’m not sure this is worth changing. I think the intersection of (nixos, camlp5) users is sufficiently tiny that they will probably find this thread.

In fact, first result on Google for “camlp5 nixos” shows this thread for me.

I’m old-skool UNIX jock. I was raised in the sysadmin tradition of “don’t put . in your path” [and putting at the end (instead of the front) isn’t a valid workaround]. And … the idea that somehow critical system utilities don’t have fixed locations in the filesystem namespace is … anathema. Anathema.

I understand the value proposition of Nix, and while I don’t want it, I still understand it. But … at the end of the day, the boot-loader can’t be accessed via a pathname that’s a hexadecimal string. And that pretty much holds for a nontrivial set of critical system utilities and libraries. And /bin/bash is one of those.

Maybe in some future world, I’ll change my mind: after all, I was a “/bin/csh” guy for 20+ years, but today I’m a “/bin/bash” guy.

But today? Critical system utilities need to be in fixed locations in the filesystem namespace, and developers ought to be able to rely on that assumption.

Heh, the wag in me asks: “the script that invokes command -v sh and then edits the shellscripts it’s about to install – what’s it written in? bash? perl?” This seems like a recipe for vicious circles.

Not complaining about you: you’re telling us about what’s documented in that POSIX spec. Peeking into that spec (boy howdy, that’s amusing, thank you for this, @jbeckford !!) l see that they suggest you should write a script that installs your shellscripts, and that script … SHOULD USE sed to edit the scripts!

I mean … sed is supposed to be more standardized than “sh” ? [eye-roll emoji]

I remember back in the day (2003) I wrote a bunch of problem-determination-gathering scripts in Perl. But IBM … in their eternal wisdom … decided that Perl wasn’t installed on systems by default, so instead of fricken’ installing it they tasked my colleague D. (great guy, but younger, and willing to bend to idiocy) with rewriting all these scripts into sed/awk. Turns out sed/awk on all these various POSIX-compliant UNIX systems are NOT semantically identical. Poor D. Poor D. OF COURSE, they also said “you can’t assume GNU sed/awk either, silly programmer!”

He spent a long, long time working thru all the interesting idiocies of AIX/HPUX/Solaris/etc sed/awk.

There’s a minimum subset of UNIX that I simply am not willing to forego. I mean, in 1994 we could sort of stipulate that UNIX might lose, and we all oughta learn Winders. Today? Nah, brah. UNIX won.

ETA: and with it so did the baseline GNU toolset.

2 Likes

But it is only a datapoint of n=1 when you intersect your problem with camlp5. The problem affects nixos and Windows users, for all packages that use /bin/sh (problems for nixos) or /usr/bin/env sh (problems for Windows).

I’ve cut an issue at Consider following POSIX standards when installing .sh scripts · Issue #5454 · ocaml/opam · GitHub because I think the POSIX spec is right: the shebang should be changed at install time, and opam install sounds like the right place to do that. Not high priority, but still a problem that will waste a lot of time for end-users.

That doesn’t fix the entire problem: anybody building packages that use autoconf/automake/configure, is writing shellscripts that aren’t installed but only run at build-time. Those scripts also need to be changed before being run. And … even assuming that we can change them, we still need to write a program to do it … what will that program be written in?

It all comes down to whether one can trust that anything in the filesystem namespace is fixed. The POSIX spec assumes that you can trust that some basic set of utilities is at known locations – they propose using sed to change shebang lines, after all.

But also: what’s the urgency of this? Nix and NixOS aren’t supported by opam (not part of CI), and until they are, what’s the point of doing this work?

Urgency?

But it does affect Windows; the last time I checked that platform is supported (or soon to be) by opam.

Can you enlighten me? How are configure/autoconf/automake found on windows? How is bash found?

Oh, that is tricky and needs a bit of an essay. Sorry. I’ll restrict most of it to just Cygwin-derived shells, which covers the vast majority of the normal Windows shell use cases today. The minority use cases I talk about at the bottom.

Cygwin has a DLL called cygwin1.dll. MSYS2 has a DLL called msys-2.0.dll which is similar so I’ll ignore it. DLLs behave somewhat similarly to .so shared libraries … the TEXT (code) and DATA segments are mapped into the virtual address space of the process they are linked to (analog: ld.so) or loaded into (analog: dlopen). But DLLs can have well-defined entry and exit functions that automatically get called whenever they are loaded into a process. What cygwin1.dll does is load a shared memory object when it is attached to any process, so that all processes linked with cygwin1.dll see the same shared memory object.

All the normal GNU binaries like bash.exe, dash.exe (the POSIX shell), ls.exe and make.exe and all the GNU autoconf tools are linked with cygwin1.dll. And these GNU binaries are compiled using normal POSIX C code like fopen.

fopen is a good example. cygwin1.dll implements fopen("/etc/passwd", "r") by consulting cygwin1.dll/../etc/fstab to find the mapping between /etc/passwd and the Windows file system; for speed the mapping is stashed into the shared memory object. The really important point is that each cygwin1.dll has its own shared memory object which gives its own file system, its own POSIX locks, and so on.

On my machine:

  • Cygwin64 Terminal has /etc/passwd -> Z:\cygwin64\etc\passwd
  • MSYS2 Terminal has /etc/passwd -> Z:\msys64\etc\passwd
  • Git Bash’s embedded MSYS2 has /etc/passwd -> C:\Program Files\Git\etc\passwd
  • Diskuv OCaml’s embedded MSYS2 has /etc/passwd -> C:\Users\beckf\AppData\Local\Programs\DiskuvOCaml\tools\MSYS2\etc\passwd
  • vcpkg’s embedded MSYS2 has /etc/passwd -> Z:\source\cpkgs\vcpkg\downloads\tools\msys2\9a1ec3f33446b195\etc\passwd

cygwin1.dll also proxies the getenv("PATH") function. It takes all of the PATH entries from Windows and translates them into the / Unix-y file system.

These / reverse mappings are relative to the specific cygwin1.dll. So if I were in Cygwin64 Terminal I would see:

  • Z:\cygwin64\etc\passwd -> /etc/passwd
  • Z:\msys64\etc\passwd -> /cygdrive/z/msys64/etc/passwd
  • C:\Program Files\Git\etc\passwd -> /cygdrive/c/Program Files/Git/etc/passwd
  • and so on

And if I were in MSYS2 Terminal I would see the same pattern (except it doesn’t use /cygdrive):

  • Z:\cygwin64\etc\passwd -> /z/cygwin64/etc/passwd
  • Z:\msys64\etc\passwd -> /etc/passwd
  • C:\Program Files\Git\etc\passwd -> /c/Program Files/Git/etc/passwd
  • and so on

So there is no loss of fidelity moving between PATHs on Windows and “Unix”.

Finally, if you understood everything I wrote above, can you predict what happens when:

  1. Your Windows PATH is a seemingly reasonable:

    C:\Program Files\Git\bin;C:\WINDOWS\system32;C:\WINDOWS
    
  2. You are inside the MSYS2 Terminal shell Z:\msys64\usr\bin\bash.exe.

  3. You run a script with a shebang:

    #!/usr/bin/env bash
    echo hi
    

Which bash will be used? MSYS2 Terminal’s or Git’s?

How about if you run the following?

#!/bin/bash
echo hi

Minority use cases: WSL2 shells

Very very annoyingly if I type bash on the Command Prompt I immediately enter WSL2 Ubuntu Linux. I don’t know when that was introduced, and I don’t know the mechanism how that works. All I know is that I’ve had shell scripts magically invoke Ubuntu Linux from Cygwin-derived shells.

Yet another complexifier.


The problem of invoking the correct shell is solved as long as every package uses #!/bin/sh rather than #!/usr/bin/env sh. Good luck trying to get that to happen, especially retroactively! Or we can do what the POSIX spec suggests … let one place (the installer) determine the correct shebang. And as a bonus, nixos can magically start working as well.

As a stop-gap, I have wrappers around common OCaml tools (so far dune.exe and opam.exe) that constructs sanitized PATHs for Diskuv OCaml native Windows users. See crossplatform-functions.sh and with_dkml.ml. Those hacks are the only reason why this isn’t high priority for Windows.

In #3, did you mean it the other way around? B/c the way it’s written, it sounds like #!/bin/sh is the thing to do, and #!/usr/bin/env sh is broken ?