Opam install Curl failed

newbie question:

(I know that this is going to sound more like a firewall problem. But given DOD and corporate regulations, I cannot get I.T. to help here. So I want to see if there is a way to trick OPAM to allow me to manually use tar downloads, or get curl to work).

I have opam version 2.0.5 installed on windows 10 (using cygwin).

I tried to opam install utop

all but 3 out of the 22 packages installed fine, so i don’t think it is network issues.
What can I do about the 3 that don’t install, and why is this happening? (I have tried on two seperate days).


<><> Gathering sources ><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
[camomile.1.0.2] found in cache
[charInfo_width.1.1.0] found in cache
[cppo.1.6.6] found in cache
[dune.2.1.2] found in cache
[dune-configurator.2.1.2] found in cache
[dune-private-libs.2.1.2] found in cache
[lambda-term.2.0.3] found in cache
[lwt.5.1.1] found in cache
[lwt_log.1.1.1] found in cache
[lwt_react.1.1.3] found in cache
[mmap.1.1.0] found in cache
[ocamlbuild.0.14.0] found in cache
[ocplib-endian.1.0] found in cache
[result.1.4] found in cache
[WARNING] ocamlfind.1.8.1: download of
          http://download.camlcity.org/download/findlib-1.8.1.tar.gz failed
          (http://download.camlcity.org/download/findlib-1.8.1.tar.gz (Curl
          failed: "D:\\cygwin64\\bin\\curl.exe --write-out %{http_code}\\n
          --retry 3 --retry-delay 2 --user-agent opam/2.0.5 -L -o
          D:/cygwin64/home/ysg4206/.opam/ocaml-variants.4.07.1+mingw64c/.opam-sw
itch/sources/ocamlfind.1.8.1/findlib-1.8.1.tar.gz.part
          http://download.camlcity.org/download/findlib-1.8.1.tar.gz" exited
          with code 28)), trying mirror
[utop.2.4.3] found in cachep] [topkg.1.0.1: http]
Processing 21/22: [ocamlfind.1.8.1: http] [topkg.1.0.1: htt
[zed.2.0.4] found in cache
[ERROR] The sources of the following couldn't be obtained, aborting:
          - ocamlfind.1.8.1: Curl failed
          - react.1.2.1: Curl failed
          - topkg.1.0.1: Curl failed


ysg4206@YSG4206 ~
$ opam --version
2.0.5

*** UPDATES ***

you will find the same me in discord or discuss, no worries :slight_smile: All three packages are using http (which is quite uncommon nowadays, and unsafe), it might be a coincidence, but my guess it the crux of the problem. You can find the source url of a package here (opam.ocaml.org/packages/ocamlfind) (just substitute the name), see the source field. And it looks like that your curl is not using a proxy at all. – [ivg]

If CURL was trying to go straight to the internet, then none of the other OPAM installs would work (HTTPS or HTTP). Our firewall blocks ALL requests that don’t use the local proxy.

In my case I have an additional proxy on my local machine (NTLM) that fakes the authorization to the corporate proxy.

So HTTP_PROXY and HTTPS_PROXY are set to localhost:3128 and I can see the traffic going out.

As I have promised, here I am, again :slight_smile:

Besides, it could be also that your corporate firewall is scanning the contents and thinks that those packages are malicious. Since all other packages are using https it is impossible to scan them, therefore they are let in. This is just a guess. In any case, let’s go through the options, that you have.

  1. When I was in exactly the same situation as you’re, we were just having our own local mirror of opam, which we were downloading (and occasionally synchronizing) at home. And then (in a violation of rules) bringing it to work on a flash drive. I think that the simplest way to do this is to use the opam admin cache command. You can then serve opam locally using the opam admin index and then setting up a local HTTP or HTTPS server.
  2. If you want a normal up-to-date access to the opam repository and you believe that it is legal and you want have any problems with your security administrators, then you should keep trying different options for curl or wget. Once, you find the right set of options, you can plug them using either the OPAMFETCH environment variable or setting the download-command in your configuration (preferred). Like this is your starting point,

"D:\\cygwin64\\bin\\curl.exe --user-agent opam/2.0.5 -L -o findlib.tar.gz http://download.camlcity.org/download/findlib-1.8.1.tar.gz"

as soon as you successfully download it, just specify the required flags substituting any real urls and files with corresponding patterns, e.g., %{url}%. This is all well-documented in opam, see opam --help for more information.

  1. If nothing works, there is a failsafe solution, just download these three packages. Using git or your browser. I wouldn’t be surprized, that opam source <package-name> will work for you. Untar them if they are tared. And pin each of the offending packages using opam pin add <package-name> <location-of-the-untarged-package>. Besides, before doing this, you can try a little bit easier approach - opam pin add <package-name> --dev-repo. It will still use opam, but hopefully the git protocol, which may or may not work in your settings. Doesn’t hurt to try :slight_smile:

With all that said, it is really a security issue that those packages are served via http. In general, it is very easy to hijack the traffic and divert it to a malicious host that will serve a backdoored ocamlfind with the same md5 sum as the original one. In general, opam should prohibit the http protocol at all.

Let me tell you where I am at.

I tried:

$ opam pin add ocamlfind --dev-repo
Processing: [ocamlfind.1.8.1: git]Username for 'https://gitlab.com':

and rather than silence, I was able to watch the traffic go out from my local proxy (which will make the request the same as if it was coming from the browser)

2020/01/23 15:08:21 //gitlab.camlcity.org:443
2020/01/23 15:08:21 //gitlab.com:443

But it seems it needs more information (like username) to get access to gitlab.com (since the the opam source ocamlfind blocks, and never proceeds.)

I can tell you that it seems that their is something wrong with the curl setup for using the HTTP_PROXY (maybe it needs lower case), since

opam source ocamlfind

never touches my local proxy. And I see from getenv that only HTTP_PROXY is define d and not http_proxy

Windows environment variables are case-insensitive. Linux env variables are case sensitive. So I used cygwin to define http_proxy the same as HTTP_PROXY and

opam install utop is working:

<><> Gathering sources ><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
[camomile.1.0.2] found in cache
[charInfo_width.1.1.0] found in cache
[cppo.1.6.6] found in cache
[dune.2.1.2] found in cache
[dune-configurator.2.1.2] found in cache
[dune-private-libs.2.1.2] found in cache
[lambda-term.2.0.3] found in cache
[lwt.5.1.1] found in cache
[lwt_log.1.1.1] found in cache
[lwt_react.1.1.3] found in cache
[mmap.1.1.0] found in cache
[ocamlbuild.0.14.0] found in cache
[ocplib-endian.1.0] found in cache
[ocamlfind.1.8.1] downloaded from http://download.camlcity.org/download/findlib-
1.8.1.tar.gz
[result.1.4] found in cache
[react.1.2.1] downloaded from http://erratique.ch/software/react/releases/react-
1.2.1.tbz
[utop.2.4.3] found in cache
[topkg.1.0.1] downloaded from http://erratique.ch/software/topkg/rel
eases/topkg-1.0.1.tbz
[zed.2.0.4] found in cache

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
Processing  2/44: [conf-m4: sh echo | m4] [dune: ocaml boot
Processing  3/44: [conf-m4: sh echo | m4] [dune: ocaml boot
Processing  4/44: [conf-m4: sh echo | m4] [dune: ocaml boot
∗ installed seq.base
Processing  5/44: [conf-m4: sh echo | m4] [dune: ocaml boot
Processing  5/44: [conf-m4: sh echo | m4] [dune: ocaml boot
Processing  5/44: [dune: ocaml bootstrap.ml] [ocamlbuild: m
∗ installed conf-m4.1
Processing  6/44: [dune: ocaml bootstrap.ml] [ocamlbuild: m
Processing  7/44: [dune: ocaml bootstrap.ml]
Processing  7/44: [dune: ocaml bootstrap.ml] [ocamlbuild: m
Processing  7/44: [dune: ocaml bootstrap.ml] [ocamlbuild: m
Processing  7/44: [dune: ocaml bootstrap.ml] [ocamlbuild: m
Processing  7/44: [dune: ocaml bootstrap.ml] [ocamlbuild: m
Processing  7/44: [dune: ocaml bootstrap.ml] [ocamlbuild: m
Processing  8/44: [dune: ocaml bootstrap.ml] [ocamlbuild: m
Processing  8/44: [dune: ocaml bootstrap.ml] [ocamlfind: ma
∗ installed ocamlfind.1.8.1
∗ installed base-bytes.basetrap.ml]
Processing 11/44: [dune: ocaml bootstrap.ml] [ocamlbuild: m
Processing 11/44: [dune: ocaml bootstrap.ml] [ocamlbuild: m
Processing 11/44: [dune: ocaml bootstrap.ml] [ocamlbuild: i
∗ installed ocamlbuild.0.14.0
Processing 12/44: [dune: ocaml bootstrap.ml] [topkg: ocaml
∗ installed topkg.1.0.1ootstrap.ml]
Processing 14/44: [dune: ocaml bootstrap.ml] [react: ocaml
∗ installed react.1.2.1ootstrap.ml]
∗ installed dune.2.1.2.exe build]l]
Processing 18/44: [camomile: ocaml configure.ml] [cppo: dun
Processing 19/44: [camomile: ocaml configure.ml] [cppo: dun
Processing 20/44: [camomile: ocaml configure.ml] [cppo: dun
Processing 21/44: [camomile: ocaml configure.ml] [cppo: dun
Processing 21/44: [camomile: ocaml configure.ml]
∗ installed cppo.1.6.6
Processing 22/44: [camomile: ocaml configure.ml]
Processing 23/44: [camomile: ocaml configure.ml]
Processing 23/44: [camomile: dune build] [dune-private-libs
Processing 23/44: [camomile: dune build] [dune-private-libs
Processing 23/44: [camomile: dune build] [dune-private-libs
∗ installed mmap.1.1.0
Processing 24/44: [camomile: dune build] [dune-private-libs
Processing 24/44: [camomile: dune build] [dune-private-libs
∗ installed result.1.4
Processing 25/44: [camomile: dune build] [dune-private-libs
Processing 25/44: [camomile: dune build] [dune-private-libs
Processing 25/44: [camomile: dune build] [dune-private-libs
Processing 26/44: [camomile: dune build] [dune-private-libs
∗ installed ocplib-endian.1.0
Processing 26/44: [camomile: dune build] [dune-private-libs
∗ installed dune-private-libs.2.1.2
Processing 28/44: [camomile: dune build] [dune-configurator
∗ installed camomile.1.0.2m: dune build]
Processing 30/44: [charInfo_width: dune build] [dune-configurat
∗ installed dune-configurator.2.1.2d]
Processing 32/44: [charInfo_width: dune build] [lwt: dune b
∗ installed charInfo_width.1.1.0
∗ installed zed.2.0.4uild] [zed: dune build]
∗ installed lwt.5.1.1uild]
Processing 38/44: [lwt_log: dune build] [lwt_react: dune bu
∗ installed lwt_log.1.1.1 build]
∗ installed lwt_react.1.1.3uild]
∗ installed lambda-term.2.0.3uild]
∗ installed utop.2.4.3uild]
Done.

<><> lwt.5.1.1 installed successfully <><><><><><><><><><><><><><><><><><><><><>
=> Lwt 5.0.0 has made some minor breaking changes. See
     https://github.com/ocsigen/lwt/issues/584

ysg4206@YSG4206 ~
$ utop
1 Like