Installing camlp5 on ocaml 5.0.0 / nixos

[Chet_Murthy]

I was actually asking a much, much simpler question. Lots of GNU-based projects are built using build-systems that rely on GNU make, autoconf, automake, etc. Are those projects going to build correctly on Windows ?

[jbeckford]

@Chet_Murthy

it sounds like #!/bin/sh is the thing to do, and #!/usr/bin/env sh is broken ?

Yes. #!/usr/bin/env is broken. Please don’t use it! Even if it works on nixos. (For any future readers of this thread).

@Chet_Murthy

Lots of GNU-based projects are built using build-systems that rely on GNU make, autoconf, automake, etc. Are those projects going to build correctly on Windows ?

The simplest answer is “no”. The slightly more nuanced answer is “often yes, but for significant numbers of users it will be no”. The biggest problem, completely unrelated to this thread, is that GNU build tools … especially GNU Make and its non-GNU cousin pkg-config … rely on paths not containing spaces. Spaces are quite common on Windows as home directory names are often FirstName LastName.

But why care about GNU build tools for GNU based projects?

GNU build tools are dead. Their use was dying 10 years ago, and it has almost died today. I realize that the OCaml compiler uses GNU build tools (and that has a viral effect on our small OCaml community), but broadly speaking the C world has moved on. Real numbers:

• Share of C developers using GNU autotools in 2019: 8%. Source: https://www.jetbrains.com/lp/devecosystem-2019/cpp/
• Share of C developers using GNU autotools in 2022: 2%. Source: https://www.jetbrains.com/lp/devecosystem-2022/cpp/#Which-project-models-or-build-systems-do-you-regularly-use

There is some nuance in there; for example, many many people are still hand crafting Makefiles. But I’ve seen other sources (share of Google searches, etc.) show similar numbers for people using the full GNU autotools stack.

In contrast, the GNU shell (bash) is thriving.

I’m assuming you are making some correlation between GNU bash and GNU autotools; the data above suggests there is an inverse correlation.

[Chet_Murthy]

@jbeckford

But why care about GNU build tools for GNU based projects?

When I build complex multi-language projects (e.g. C++, OCaml, Python), I use GNU build-tools. A project like GitHub - chetmurthy/thrift-nicejson: Hack on nice json wire format for thrift for example. But also, they’re a stand-in for the general issue of “UNIX compatibility”.

It’s funny: you point at the POSIX specs, which encourage using sed to customize scripts. Others have suggested that #!/usr/bin/env yadda-yadda is the way to go. None of this is tenable. B/c when you build a significant project, you’ll have to update many different scripts that do different things. What scripting language will the script that invokes sed be written-in? And how will that script get customized? And how will the script that customizes that script get customized?

It’s farcical. Part of UNIX standards is the location and meaning of standard system utilities. We’re meant to be able to rely on the location and semantics of these suckers.

[jbeckford]

I think you are misreading the specs. The guidance is:

… update the “#!” script appropriately as it is being installed (for example, with sed)

sed is a (parenthetical!) example of implementing the guidance, not the guidance itself. In this thread the “as it is being installed” is being performed by opam install. No sed is necessary inside the OCaml source code of opam. And that chain that ends with “how will the script that customizes that script get customized?” … which I assume was written for humor … doesn’t make sense. Nor does that chain help solve the real problem that end-users face.

The narrow nixos issue in this thread is actually simple to solve by installing a hard link in the sandbox. The flow is that source files (including configure.sh) get installed into a sandbox first by opam install. I think it is mostly rsync to get to that point. Make a hard link from the canonical location of the shell (ie. getconf PATH search with a command -v sh fallback; ignore the rest of the POSIX guidance) to /bin/sh inside the sandbox. Ditto for /bin/bash. And I suspect some modification of that may work on Windows as well if it could support a bwrap-like sandbox.

[Chet_Murthy]

@jbeckford

In this thread the “as it is being installed” is being performed by opam install. No sed is necessary inside the OCaml source code of opam.

The thesis here is that in order to build a package, we need a non-script tool ? You don’t see a problem with that? UNIX has for nearly fifty years learned how to use scripting languages for many, many administrative tasks, and now, because we want to discard the normal guarantees for where critical system utilities will live (and their semantics), we’re going to discard that, and switch to complex compiled binaries ?

I mean, opam is incredibly opaque and difficult-to-debug when it goes wrong. The idea of opam being intimately involved in bashing on the contents of the files of my build-tree before it builds isn’t reassuring.

And for what? I mean, what’s the gain? This isn’t necessary on any UNIX system except when using a distribution that I’d characterize as fundamentalist. There’s nothing wrong with application software being versioned as Nix does. But the base operating system? Versioning that in this manner is foolishness.

[Chet_Murthy]

I mean, we’re not talking about even Python (2.x, 3) or Perl (5.maybe many versions) here. We’re talking about bash. The Bourne shell. It’s the oldest shell in UNIX. It’s incredibly stable. Versioning it with a hexadecimal string in the pathname … that’s … that’s …

[otini]

@Chet_Murthy

I’m the maintainer, and … I have never seen such guidance. I mean … if /bin/bash can’t be counted-upon, why should we assume that /usr/bin/env can be counted-upon for that matter?

ETA: after all, /usr/bin/env is much, much, much younger than /bin/sh, /bin/csh, and probably /bin/bash. If the answer were “use /bin/sh”, I’d be OK with that, I guess. But /usr/bin/env ?

What I’ve been told is that e.g. bash is at /usr/local/bin/bash on *BSD, so /usr/bin/env bash is the only thing that works across all Unix-derived systems (NixOS included). I haven’t studied the matter further.

[Frederic_Loyer]

And the bash is optional like all things among the ports. I guess that some other non Linux systems may share this characteristic.

Then it is better to depends on plain old sh.

[Chet_Murthy]

Ah, /bin/sh – sure that’s fine. I had assumed that @otini was suggesting that the guidance was somehow that hard-coding any shell was verboten. I’m 100% cool with switching to old-skool Bourne shell.

[xavierleroy]

@Chet_Murthy

But IBM … in their eternal wisdom … decided that Perl wasn’t installed on systems by default, so instead of fricken’ installing it they tasked my colleague D. (great guy, but younger, and willing to bend to idiocy) with rewriting all these scripts into sed/awk. Turns out sed/awk on all these various POSIX-compliant UNIX systems are NOT semantically identical. Poor D. Poor D. OF COURSE, they also said “you can’t assume GNU sed/awk either, silly programmer!” He spent a long, long time working thru all the interesting idiocies of AIX/HPUX/Solaris/etc sed/awk.

Nice (and sad, and probably very common) story.

You’ll be happy to learn that for the OCaml core system, we’re working through the idiocies of sed/awk on various platforms just like D. was, and that my proposals to use better tools for scripting have been blocked or deflected so far. My proposal was Python, because we’re in the 21st century, but even Perl would be a huge improvement on what we currently have. The current deflection of the proposal is Lua, which doesn’t inspire much confidence in me.

[Chet_Murthy]

Xavier,

Omigod. I am horrified. Yes, Python would be a perfectly cromulent stand-in for Perl. And I’d go further and say “Python 2.x”. Pick something old, so that it’s going to be really, really stable. Lua … my mind boggles, really it does. It’s got nowhere near the adoption of Python/Perl, and in the end that adoption is what matters. Everything else is secondary.

[c-cube]

Lua isn’t necessarily a bad choice, imho. It’s not the most popular but
it’s far from dead (in particular it’s growing in the (n)vim ecosystem!)
and it’s a very small, self contained, C program. I guess perl has the
same properties but is far more, erm, idiosyncratic. Python is larger
and more of a moving target.

[Chet_Murthy]

@c-cube

Python is larger
and more of a moving target.

Re: Perl, I will only note that Debian uses Perl extensively internally: as long as that remains the case, Perl will remain very much alive. Sure, it’s not en vogue for sysadmins anymore. But it’s still the backbone of Debian’s management code.

Re: Python, that’s why I suggested Python 2.x and not 3.x. B/c 2.x is going to be around forever, and there will be few new features added to it, so no churn.

[c-cube]

At this point, python 2.x is less popular, less maintained, and less widely deployed than Lua .

Anyway, I don’t have much stakes in this game, but having to install python 2 to compile OCaml would be very sad.

[anuragsoni]

@Chet_Murthy

Re: Python, that’s why I suggested Python 2.x and not 3.x. B/c 2.x is going to be around forever, and there will be few new features added to it, so no churn.

Python 2 is also increasingly not going to be packaged out of the box on most linux distros (and macOS) etc.

[Chet_Murthy]

@anuragsoni

Python 2 is also increasingly not going to be packaged out of the box on most linux distros (and macOS) etc.

It isn’t a question of whether it’s packaged out-of-the-box or not; what matters, is that it can be installed by the system package-manager in a straightforward way. Now, I’m not arguing that one must stick th Python 2.x; rather, I’m noting that as long as it’s widely-available, the fact that it is a slow-moving target is a good thing, for critical build tooling.

[jbeckford]

Or just do what 57% (which is +15% in 3 years) of the rest of the C-based world is doing for build tooling and scripting: using CMake. One binary, available from almost all package managers, works on Windows, all the scripting frills (downloading, checksumming, file/dir operations, spawning, string manipulation, etc.) and first-class support from IDEs and C/ASM compiler vendors.

Yep, its syntax sucks and it has to maintain a legacy (non-“modern”) semantics model that should never have been adopted. But for better or worse CMake won decisively, and in 2023 the lack of portable scripting across operating systems is now a solved problem. Edit: Solved for complex projects like ocaml/ocaml, not for one-off scripts.

[rgrinberg]

One more +1 for lua. It can be easily compiled with $(CC) *.c -o lua and hence easily included in any project. It would be awesome if OCaml could shed its build dependencies.

[Release-Candidate]

Just a word of caution: as Python 2.7 is end of life since 2020 that means no security updates which makes this a no-option in many environments.

Theoretically there is Active State Python (Linux, MacOS and Windows only) which still fixes CVEs:

Python 2 Security Vulnerability (CVE) Updates

[Frederic_Loyer]

A CVE on Python2 can be problematic if a Python2 application is exposed on the Web, registered as a setuid program (or listed in allowed sudo programs) or feed with untrusted content. Such exposition could allow the use of CVE the gain non authorized provileges.

If Python2 is only accessible from a shell, the users which can launch it have already the possibility to execute whatever they what. The CVE wont help them to gain more privileges. As an opam author which would want to do something bad in the destination hosts, there would be far more obvious ways to proceed than to depend on a Python CVE.