OCaml on Windows: the long paths issue

Ecosystem
1

Not very long ago, I managed to solve the long paths issue for OCaml and I want to share how.

While building a large project, I ran into the path issues - Dune would fail with errors saying a certain file present in a long file was not found. Shortening libraries names helped. But what was really needed was manifesting OCaml tools so that they can perform I/O on long paths as described here on MSDN docs.

Guidelines there expect binary artifacts (dll or exe) have to be manifested with something like this

<application xmlns="urn:schemas-microsoft-com:asm.v3">
    <windowsSettings xmlns:ws2="http://schemas.microsoft.com/SMI/2016/WindowsSettings">
        <ws2:longPathAware>true</ws2:longPathAware>
    </windowsSettings>
</application>

Ideally, the application being built is also manifested so.

I use esy as my daily driver for Windows development. I have always found the workflow on Windows with esy reasonable and never stopped using it since I discovered. Over time, I have taken up maintenance of the project too.

OCaml on Windows is practical only with Cygwin. Not because the tools needs a Unix like environment - they have evolved significants over the years since I have been using OCaml on Windows. Cygwin is convenient because of the libraries in the ecosystem - for instance all the native libraries from GNU project. GNU projects are easily built inside Cygwin/MSYS/mingw but not so outside.

I was worried that I’d have to write some gory hack in the compiler to manifest every binary artifact - dune had to be manifested too. There are many ways to embed the manifest with tools windres etc. But fortunately, I found that Mingw already embeds all executables with a default one. It can be found here among the Cygwin projects

I decided to fork the repository and use the manifest MSDN docs recommended. You can find it here This helped me fix my project, but I wanted this for the rest of the community too.

Esy uses a utility called esy-bash which packages the Cygwin environment for the user. We find that users have mostly uniform setups this way. Helps with reproducibility. I decided to tweak esy-bash to package my fork of the windows-default-manifest along with other Cygwin packages. This way OCaml users dont have to worry about the long paths issue anymore.

This has been available since esy 0.7.0. If you’d like to try the latest esy, it is available on NPM and can be installed with npm i -g esy

I’d love to hear your thoughts on this approach. More than happy to contribute a solution like this for opam too.

9 Likes
2

The MSDN docs say (italics added):

To enable the new long path behavior per application, two conditions must be met. A registry value must be set, and the application manifest must include the longPathAware element.

I also tested on a single Windows PC that both conditions were needed before long paths worked using that method. I suspect on your PC Get-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" returns 1 when the default is 0.

The problematic condition is “A registry value must be set”. That requires Administrator privileges and making a global change to the PC that affects other applications. (Unix analogy: It would be like having to edit /etc/sysctl.conf to install OCaml.)

The “correct” way is to canonicalize the paths and prepend the extended-length prefix \\?\ for any absolute paths before calling any of the functions listed at the bottom of the MSDN docs (ex. C:/a/b/c is converted to \\?\C:\a\b\c).

For example:

would need the canonicalize+prepend any path that is absolute.

Specifically:

  1. Make a helper function void caml_win32_extended_path(const wchar_t* path, size_t buflen, wchar_t* new_path) that does the canonicalization + prepending if and only if the path is absolute.
  2. Search through the ocaml/ocaml code for call sites for the 32 functions (ex. CreateFile). For each call site, add a call to the helper function caml_win32_extended_path before the call site (ex. runtime/win32.c#L844).

I think that will fix the Windows long paths bug completely without requiring an administrative hack of a developer’s machine.

(I marvel at how long this bug has stayed open! Or maybe there were so few devs meaningfully using OCaml+Windows. Please open a bug at ocaml/ocaml if there isn’t one already. It could be a good starter issue for someone wanting to contribute to ocaml/ocaml.)

3

Wouldn’t //?/C:/a/b/c work? I had thought that Windows APIs accepted forward-slash everywhere now.

4

There is a purple note (3rd paragraph) of the MSDN article that addresses your question.

1 Like
5

This wasn’t practical in my case - I had to build the ligo compiler on Windows and there were very long paths created by Dune.