Windows compiler support in opam 2.2.0~beta2

Ecosystem
, ,
1

On behalf of the entire opam team, but also with a personal sense of relief, I’m very pleased to announce that the process of upstreaming support for Windows OCaml to opam-repository in ocaml/opam-repository#25861 finally started on Friday!

There’s a full blog post with details on how you can try this out now with opam 2.2.0~beta2. The TL;DR, assuming you have winget on your Windows system (open the Microsoft Store app and install the App Installer package from Microsoft if you don’t) then you can issue:

winget install Git.Git

if you don’t have Git for Windows and:

winget install opam

if you don’t yet have the 2.2.0~beta2 binary. You must then launch a fresh Command Prompt / PowerShell session. For there, you can then run:

opam init git+https://github.com/dra27/opam-repository.git#windows-initial

or

opam init -a --no-git-location --cygwin-internal-install git+https://github.com/dra27/opam-repository.git#windows-initial

if you’d like to be asked fewer questions. There is a known and big pause when updating the repository. However, after a little bit of time (coffee, or a sword battle, if that’s your thing), you should then be faced with a fully initialised opam with ocaml-base-compiler.5.2.0 installed for the mingw-w64 amd64 port of native Windows opam.

Things with depexts will likely not work: the blog post contains details on how to get started with PRs, but issues are also helpful.

The blog post covers what we regard as the “default use case” - that is a native Windows user expecting to use this new OCaml thing they heard about natively. i.e. not running in WSL or Cygwin or MSYS2 or any other “are you sure can’t just install Linux on that?” approach.

However, all the other use cases matter too! You’re meant to be able to run native Windows opam from your own Cygwin or MSYS2 mintty bash terminal; we are aiming for the opam 2.2.0 binary to be a drop-in replacement (apart from setting os-distribution to "cygwinports") for “OCaml for Windows” for legacy use with the “sunset” repository; you’re meant to be able to provide your own Cygwin or MSYS2 installation if you really need to (and you really might!). But we do need help testing all of it :slightly_smiling_face:

We anticipate one further beta of opam 2.2.0 by the end of the month. From the Windows perspective, this will fix a known bug in the environment variable handling (see ocaml/opam#5838) but will also hopefully straighten out the behaviour of opam init for some of these “non-default” use cases. We’re then hoping to rocket towards a release candidate in June :rocket:

Happy Windows hacking! Please open issues; please ask for further help; please have fun!

David, on behalf of the opam team.

24 Likes
2

I wanted to test that by upgrading my old vagrant setup (probably devised under your supervision :–) for local CI but sadly my laptop changed architecture meanwhile and it seems ms has no plans to distribute ARM dev vms so far.

But congratulations to have gotten that through[1] !

  1. I remember a long time ago going with you to some meeting with people at MSR whose names escape me right now (probably members of the f# team?) where you’d show us early prototypes of this. ↩︎

2 Likes
3

Student here, first time posting so I’m not sure if this is a suitable place to put this or a correct format.

Most things have gone smoothly apart only 1 thing actually not working and another being unclear

In the installation process one step that confused me was the step describing how to set up the opam environment via opam env and then asking if I would like to select a different shell.

<><> Required setup - please read <><><><><><><><><><><><><><><><><><><><><> :camel:

In normal operation, opam only alters files within ~\AppData\Local\opam.

However, to best integrate with your system, some environment variables
should be set. When you want to access your opam installation, you will
need to run:

(& opam env) -split ‘\r?\n’ | ForEach-Object { Invoke-Expression $_ }

You can always re-run this setup with ‘opam init’ later.
opam doesn’t have any configuration options for powershell; you will have to run (& opam env) -split ‘\r?\n’ | ForEach-Object { Invoke-Expression $_ } whenever you change you current ‘opam switch’ or start a new terminal session.
Alternatively, would you like to select a different shell? [y/n]

I ended up saying no to this and it all worked out but, I don’t really get what would happen if I said yes, would it just show me the alternative commands or would it do something else? It’s really not clear.

The other question about this segment is where do I put this command to run later? On linux I’d stick it in my bashrc/profile and such a thing does exist for powershell (the location of this file is found via the $profile variable), but running powershell scripts is disabled by default so you’ve got to set the execution policy to Unrestricted via Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope CurrentUser Is this the recommended way to set up the opam env on windows?

Here’s the execution policy docs

After install I wanted to see if the vscode plugin worked with a newly generated dune project.
Installing ocaml-lsp-server worked but ocamlformat didn’t install with the message “ERROR while fetching sources for ocp-indent.1.8.1” the file it was trying to access seemed to exist but couldn’t be opened even as an admin

my username has been replaced with Username where needed in the error message below

#=== ERROR while fetching sources for ocp-indent.1.8.1 ========================#
OpamSolution.Fetch_fail(“C:\Users\Username\AppData\Local\Programs\opam\bin\opam.exe: "lstat" failed on C:\Users\Username\AppData\Local\Temp\opam-192256-ab6380\ocp-indent-1.8.1\tests\inplace\link.ml: No such file or directory”)

Overall the experience of installing it was mostly painless with one confusing point and a package not building but an overall a good experience of something that’s not fully released. I did try compiling a more complex project and that seemed to work too

4 Likes
4

The ocp-indent error was reported to us 2 days ago and a fix was merged today in ocaml/opam#5953 so it shouldn’t happen in the upcoming and last beta (beta3).

Is this the recommended way to set up the opam env on windows?

The piece of powershell script is the equivalent of eval $(opam env) that you would do manually on POSIX-type shells. This would be run inside the terminal “whenever you change you current ‘opam switch’ or start a new terminal session.”, not as a script, only as a manual command. The script method (equivalent of putting things into ~/.zshrc or similar) is not supported for powershell or cmd at the moment unless I’m mistaken.

would it just show me the alternative commands or would it do something else?

yes, it would give you a menu and ask you which of the fish, csh, zsh, sh, bash, pwsh, powershell or cmd shell you would prefer using. I agree the question is not asked in the best way or maybe not in the right order, or it may not be very relevant. I’m not sure yet.

I’ve opened ocaml/opam#5955 to discuss these issues.
Thanks a lot for beta testing opam on Windows!

1 Like
5

Welcome! Thank you for trying this out, and this (along with GH issues, etc.) is totally a right place :slightly_smiling_face:

This is something which will improve in a subsequent release (along with auto completion) - for PowerShell, we can directly use hooks in the same way as we do on Linux, and in fact opam already has (unactivated) C code which can create the same effect in Cmd where it’s not necessary to run anything after an opam command.

For both PowerShell and Cmd, I highly recommend installing clink (winget install clink), which adds full readline support… so at least the rune available with CTRL+R (PowerShell has other ways to configure this without clink, but I still recommend clink!). I also occasionally use opam switch as a cheap way of being reminded it, as it’s displayed if the environment is out of sync.

2 Likes
6

This is a shame - I have at various points in the past chipped away at Windows arm64 support for OCaml, which might make this better (although I’ve never tried putting arm64 Windows in a VM - I’ve got a feeling Microsoft might make that harder than one might hope for too)

Thank you :slightly_smiling_face: I actually dug out the slides from that meeting while preparing the text for the PR - it was at least you, me, @avsm, @msprotz and @ishtiaq! That proposal had Windows-specific special instructions in it, though :scream_cat:

1 Like
7

I am running into the following error after configuring for an internal cygwin install:

❯ opam switch create 5.2.0

<><> Installing new switch packages <><><><><><><><><><><><><><><><><><><><>  🐫
Switch invariant: ["ocaml-base-compiler" {= "5.2.0"} | "ocaml-system" {= "5.2.0"}]

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><>  🐫
∗ installed base-bigarray.base
∗ installed base-threads.base
∗ installed base-unix.base
∗ installed ocaml-options-vanilla.1
⬇ retrieved ocaml-base-compiler.5.2.0  (cached)
[ERROR] The compilation of ocaml-base-compiler.5.2.0 failed at "make -j7".

#=== ERROR while compiling ocaml-base-compiler.5.2.0 ==========================#
# context     2.2.0~beta2 | win32/x86_64 |  | https://opam.ocaml.org#ec28f29e
# path        ~\AppData\Local\opam\5.2.0\.opam-switch\build\ocaml-base-compiler.5.2.0
# command     ~\AppData\Local\opam\.cygwin\root\bin\make.exe -j7
# exit-code   2
# env-file    ~\AppData\Local\opam\log\ocaml-base-compiler-4636-8a581d.env
# output-file ~\AppData\Local\opam\log\ocaml-base-compiler-4636-8a581d.out
### output ###
# C:/msys64/ucrt64/include/_mingw.h:271:2: error: #error Only Win32 target is supported!
# [...]
#       |  ^~~~~
# In file included from C:/msys64/ucrt64/include/stdarg.h:140,
#                  from C:/msys64/ucrt64/lib/gcc/x86_64-w64-mingw32/13.2.0/include/stdarg.h:1,
#                  from runtime/caml/misc.h:27:
# C:/msys64/ucrt64/include/_mingw_stdarg.h:11:2: error: #error Only Win32 target is supported!
#    11 | #error Only Win32 target is supported!
#       |  ^~~~~
# make[1]: *** [Makefile:1297: runtime/sak.o] Error 1
# make[1]: *** Waiting for unfinished jobs....
# make[1]: Leaving directory '/cygdrive/c/Users/***/AppData/Local/opam/5.2.0/.opam-switch/build/ocaml-base-compiler.5.2.0'
# make: *** [Makefile:830: world.opt] Error 2

Don’t know how to troubleshoot further…

Thanks!

8

Opam 2.2 installs Cygwin, not msys2. Did you have a preexisting MSYS2 installed globally (in your PATH)?

9

I do have an msys2 installation, but msys2.exe was not part of PATH…

I should also say that I don’t get presented with the choice of git installation and unix infrastructure reliably, meaning that after uninstalling and starting “from scratch”, running opam init git+... finishes quickly without prompting me, so I assume some configuration state persists? I would like to attempt to manually point opam to my cygwin installation.

10

I’ve successfully installed. I needed to manually install patch, make and mingw64-x86_64-gcc-core using my own cygwin installation. It seems that the opam-internal cygwin installation failed somehow.

11

Thanks for trying this out! As @jbeckford notes, something has caused that MSYS2 installation to be partially picked up.

Can I ask where you were running opam init from - a Command Prompt session; PowerShell; an MSYS2 terminal?

Yes, indeed - if you want a complete reset you need to delete the opam root completely, which by default is in %LOCALAPPDATA%\opam (it’s referred to as ~\AppData\Local\opam in the error log)

Did you configure that by telling opam init where the Cygwin installation was, or by adjusting your PATH?

12

I am running everything from a PowerShell session and I adjusted my path to make binaries installed via cygwin available.

I am now running into a similar issue with pkg-config as described in the blog post, albeit not when installing ssl. Am I correct in assuming that for that package to build, upstream would need to make changes so that pkgconf gets called during the build on windows like here?

13

That’s right, yes - once the main compiler PR is merged in opam-repository, it should be possible to make additional PRs to opam-repository to update the conf- packages as needed… help very much appreciated with those (as with the maintenance of other conf- packages in opam-repository).

If noone beats me to it, I intend to put together a generator to maintain the packages somewhat more automatically, but there are other things higher up my task list at the moment!

1 Like
14

There are other binaries in the PATH (not msys2.exe) that would cause the issue you experienced. Sorry if I didn’t ask precisely enough.

MSYS2 and Cygwin are deeply incompatible with each other. I suspect you will experience more hard-to-troubleshoot issues with opam 2.2 if you don’t find the culprit to your original problem.

Can you dump the following from a new PowerShell terminal (I use and recommend xxx to redact paths):

PS C:\Users\beckf> echo $env:PATH
C:\Program Files\Eclipse Adoptium\jdk-17.0.8.7-hotspot\bin;Z:\ProgramFiles\openssl-1.0.2u-win64-1;Z:\ProgramFiles\Miniconda3;Z:\ProgramFiles\Miniconda3\Library\mingw-w64\bin;Z:\ProgramFiles\Miniconda3\Library\usr\bin;Z:\ProgramFiles\Miniconda3\Library\bin;Z:\ProgramFiles\Miniconda3\Scripts;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;xxx;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\Calibre2\;C:\Program Files\PuTTY\;C:\Program Files\GitHub CLI\;xxx;C:\Program Files\WireGuard\;Z:\ProgramFiles\glab;Z:\ProgramFiles\nodejs\;Z:\ProgramFiles\CMake\bin;C:\Program Files\gs\gs10.02.0\bin;C:\Program Files\usbipd-win\;C:\Program Files (x86)\Gpg4win\..\GnuPG\bin;C:\Program Files\Git\cmd;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\ProgramData\chocolatey\bin;C:\Program Files\PowerShell\7\;C:\Program Files\Docker\Docker\resources\bin;C:\Program Files\1Password CLI;C:\Users\beckf\AppData\Local\Programs\DKMLNA~1\usr\bin;C:\Users\beckf\AppData\Local\Programs\DKMLNA~1\bin;C:\Users\beckf\AppData\Local\Programs\opam\bin;C:\Users\beckf\.cargo\bin;C:\Users\beckf\AppData\Local\Microsoft\WindowsApps;Z:\ProgramFiles\ngrok;C:\Users\beckf\AppData\Local\Programs\Microsoft VS Code\bin;Z:\ProgramFiles\apache-maven-3.8.4\bin;Z:\ProgramFiles\unison-v2.51.5-ocaml-4.12.0-x86_64.windows\bin;Z:\ProgramFiles\CMake\bin;C:\Users\beckf\.dotnet\tools;C:\Users\beckf\AppData\Roaming\npm;xxx;C:\Users\beckf\AppData\Local\Microsoft\WinGet\Links;C:\Users\beckf\AppData\Local\Microsoft\WindowsApps;xxx;
PS C:\Users\beckf> echo $env:PKG_CONFIG_PATH
PS C:\Users\beckf> echo $env:INCLUDE
PS C:\Users\beckf> echo $env:LIB
PS C:\Users\beckf> where.exe bash
C:\Windows\System32\bash.exe
C:\Users\beckf\AppData\Local\Microsoft\WindowsApps\bash.exe
PS C:\Users\beckf> where.exe sh
INFO: Could not find files for the given pattern(s).
PS C:\Users\beckf> where.exe gcc
INFO: Could not find files for the given pattern(s).
PS C:\Users\beckf> where.exe make
INFO: Could not find files for the given pattern(s).
PS C:\Users\beckf> where.exe opam
C:\Users\beckf\AppData\Local\Programs\DkMLNative\bin\opam.exe
C:\Users\beckf\AppData\Local\Programs\opam\bin\opam.exe
1 Like
15 
❯ echo $env:PATH
C:\Users\xxx\AppData\Local\opam\5.0.0\bin;C:\cygwin64\usr\x86_64-w64-mingw32\sys-root\mingw\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files\WezTerm;C:\Program Files\dotnet\;C:\Users\xxx\AppData\Roaming\nvm;C:\Program Files\nodejs;C:\Program Files\Git\cmd;C:\Program Files\Calibre2\;C:\Users\xxx\AppData\Local\Programs\opam\bin;C:\Users\xxx\AppData\Local\Programs\Python\Launcher\;C:\Program Files (x86)\Elm\0.19.1\bin;C:\Users\xxx\scoop\shims;C:\Users\xxx\AppData\Local\Microsoft\WindowsApps;C:\Program Files\Neovim\bin;C:\Users\xxx\.dotnet\tools;C:\msys64\ucrt64\bin;C:\Users\xxx\AppData\Local\Programs\qutebrowser;C:\Users\xxx\AppData\Local\Programs\Microsoft VS Code\bin;C:\Users\xxx\AppData\Roaming\nvm;C:\Program Files\nodejs;C:\bin;C:\Users\xxx\AppData\Local\GitHubDesktop\bin;C:\ProgramData\kento\GitHubDesktop\bin;C:\cygwin64\bin;C:\Users\xxx\AppData\Local\Programs\DkMLNative\bin

❯ echo $env:PKG_CONFIG_PATH
#nothing

❯ echo $env:INCLUDE
#nothing

❯ echo $env:LIB
#nothing

❯ where.exe bash
C:\Windows\System32\bash.exe
C:\Users\xxx\scoop\shims\bash.exe
C:\Users\xxx\AppData\Local\Microsoft\WindowsApps\bash.exe
C:\cygwin64\bin\bash.exe

❯ where.exe sh
C:\Users\xxx\scoop\shims\sh.exe
C:\cygwin64\bin\sh.exe

❯ where.exe gcc
INFO: Could not find files for the given pattern(s).

❯ where.exe make
C:\cygwin64\bin\make.exe

❯ where.exe opam
C:\Users\xxx\AppData\Local\Programs\opam\bin\opam.exe
C:\Users\xxx\AppData\Local\Programs\DkMLNative\bin\opam.exe
16

Thanks @kentookura . The problem is C:\msys64\ucrt64\bin; in your PATH.

Given how many bash shells are in your PATH (Scoop, Cygwin, WSL2) it sounds like you can safely remove the C:\msys64\ucrt64\bin entry in Control Panel / System Properties / Environment Variables to avoid related problems in the future.

(And it would be good if opam 2.2 emitted a warning in your situation)

1 Like
17

Do you know (before I try to reproduce it) what binaries in there can cause OCaml’s configure script to get upset?

18

Work already in the pipeline for this for beta3! However, this shouldn’t have required a warning - I’m anticipating that the internal Cygwin failed because it allowed its bin directory to be moved further down PATH past that directory

19

My guesses for what brought in the MinGW include file C:/msys64/ucrt64/include/_mingw.h are only guesses:

  • Maybe pkgconf.exe or pkg-config.exe
  • Maybe something related to GCC or autoconf (I hope that Cygwin/autoconf is not PATH searching for system/MinGW include files)
  • (most likely) The shebang at the top of ocaml/configure at 51e4d9af2c58d7d60f8cd2ce82424ffbacd56540 · ocaml/ocaml · GitHub is sufficient for a PATH search of another shell (if, for example, the shell that calls ./configure is not sh.exe but bash.exe or dash.exe)

Maybe it is better not to guess. @kentookura , can you dump dir C:\msys64\ucrt64\bin so the issue can be reproduced?

20

Sorry, but I have uninstalled msys2 to get everything to work… I think I added the path entry manually a while back when I was working on something else. However, the path was not empty when I created my initial post