(I know that this is going to sound more like a firewall problem. But given DOD and corporate regulations, I cannot get I.T. to help here. So I want to see if there is a way to trick OPAM to allow me to manually use tar downloads, or get curl to work).
I have opam version 2.0.5 installed on windows 10 (using cygwin).
I tried to opam install utop
all but 3 out of the 22 packages installed fine, so i don’t think it is network issues.
What can I do about the 3 that don’t install, and why is this happening? (I have tried on two seperate days).
<><> Gathering sources ><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
[camomile.1.0.2] found in cache
[charInfo_width.1.1.0] found in cache
[cppo.1.6.6] found in cache
[dune.2.1.2] found in cache
[dune-configurator.2.1.2] found in cache
[dune-private-libs.2.1.2] found in cache
[lambda-term.2.0.3] found in cache
[lwt.5.1.1] found in cache
[lwt_log.1.1.1] found in cache
[lwt_react.1.1.3] found in cache
[mmap.1.1.0] found in cache
[ocamlbuild.0.14.0] found in cache
[ocplib-endian.1.0] found in cache
[result.1.4] found in cache
[WARNING] ocamlfind.1.8.1: download of
http://download.camlcity.org/download/findlib-1.8.1.tar.gz failed
(http://download.camlcity.org/download/findlib-1.8.1.tar.gz (Curl
failed: "D:\\cygwin64\\bin\\curl.exe --write-out %{http_code}\\n
--retry 3 --retry-delay 2 --user-agent opam/2.0.5 -L -o
D:/cygwin64/home/ysg4206/.opam/ocaml-variants.4.07.1+mingw64c/.opam-sw
itch/sources/ocamlfind.1.8.1/findlib-1.8.1.tar.gz.part
http://download.camlcity.org/download/findlib-1.8.1.tar.gz" exited
with code 28)), trying mirror
[utop.2.4.3] found in cachep] [topkg.1.0.1: http]
Processing 21/22: [ocamlfind.1.8.1: http] [topkg.1.0.1: htt
[zed.2.0.4] found in cache
[ERROR] The sources of the following couldn't be obtained, aborting:
- ocamlfind.1.8.1: Curl failed
- react.1.2.1: Curl failed
- topkg.1.0.1: Curl failed
ysg4206@YSG4206 ~
$ opam --version
2.0.5
*** UPDATES ***
you will find the same me in discord or discuss, no worries All three packages are using http (which is quite uncommon nowadays, and unsafe), it might be a coincidence, but my guess it the crux of the problem. You can find the source url of a package here (opam.ocaml.org/packages/ocamlfind) (just substitute the name), see the source field. And it looks like that your curl is not using a proxy at all. – [ivg]
If CURL was trying to go straight to the internet, then none of the other OPAM installs would work (HTTPS or HTTP). Our firewall blocks ALL requests that don’t use the local proxy.
In my case I have an additional proxy on my local machine (NTLM) that fakes the authorization to the corporate proxy.
So HTTP_PROXY and HTTPS_PROXY are set to localhost:3128 and I can see the traffic going out.
Besides, it could be also that your corporate firewall is scanning the contents and thinks that those packages are malicious. Since all other packages are using https it is impossible to scan them, therefore they are let in. This is just a guess. In any case, let’s go through the options, that you have.
When I was in exactly the same situation as you’re, we were just having our own local mirror of opam, which we were downloading (and occasionally synchronizing) at home. And then (in a violation of rules) bringing it to work on a flash drive. I think that the simplest way to do this is to use the opam admin cache command. You can then serve opam locally using the opam admin index and then setting up a local HTTP or HTTPS server.
If you want a normal up-to-date access to the opam repository and you believe that it is legal and you want have any problems with your security administrators, then you should keep trying different options for curl or wget. Once, you find the right set of options, you can plug them using either the OPAMFETCH environment variable or setting the download-command in your configuration (preferred). Like this is your starting point,
as soon as you successfully download it, just specify the required flags substituting any real urls and files with corresponding patterns, e.g., %{url}%. This is all well-documented in opam, see opam --help for more information.
If nothing works, there is a failsafe solution, just download these three packages. Using git or your browser. I wouldn’t be surprized, that opam source <package-name> will work for you. Untar them if they are tared. And pin each of the offending packages using opam pin add <package-name> <location-of-the-untarged-package>. Besides, before doing this, you can try a little bit easier approach - opam pin add <package-name> --dev-repo. It will still use opam, but hopefully the git protocol, which may or may not work in your settings. Doesn’t hurt to try
With all that said, it is really a security issue that those packages are served via http. In general, it is very easy to hijack the traffic and divert it to a malicious host that will serve a backdoored ocamlfind with the same md5 sum as the original one. In general, opam should prohibit the http protocol at all.
and rather than silence, I was able to watch the traffic go out from my local proxy (which will make the request the same as if it was coming from the browser)
Windows environment variables are case-insensitive. Linux env variables are case sensitive. So I used cygwin to define http_proxy the same as HTTP_PROXY and
opam install utop is working:
<><> Gathering sources ><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
[camomile.1.0.2] found in cache
[charInfo_width.1.1.0] found in cache
[cppo.1.6.6] found in cache
[dune.2.1.2] found in cache
[dune-configurator.2.1.2] found in cache
[dune-private-libs.2.1.2] found in cache
[lambda-term.2.0.3] found in cache
[lwt.5.1.1] found in cache
[lwt_log.1.1.1] found in cache
[lwt_react.1.1.3] found in cache
[mmap.1.1.0] found in cache
[ocamlbuild.0.14.0] found in cache
[ocplib-endian.1.0] found in cache
[ocamlfind.1.8.1] downloaded from http://download.camlcity.org/download/findlib-
1.8.1.tar.gz
[result.1.4] found in cache
[react.1.2.1] downloaded from http://erratique.ch/software/react/releases/react-
1.2.1.tbz
[utop.2.4.3] found in cache
[topkg.1.0.1] downloaded from http://erratique.ch/software/topkg/rel
eases/topkg-1.0.1.tbz
[zed.2.0.4] found in cache
<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
Processing 2/44: [conf-m4: sh echo | m4] [dune: ocaml boot
Processing 3/44: [conf-m4: sh echo | m4] [dune: ocaml boot
Processing 4/44: [conf-m4: sh echo | m4] [dune: ocaml boot
∗ installed seq.base
Processing 5/44: [conf-m4: sh echo | m4] [dune: ocaml boot
Processing 5/44: [conf-m4: sh echo | m4] [dune: ocaml boot
Processing 5/44: [dune: ocaml bootstrap.ml] [ocamlbuild: m
∗ installed conf-m4.1
Processing 6/44: [dune: ocaml bootstrap.ml] [ocamlbuild: m
Processing 7/44: [dune: ocaml bootstrap.ml]
Processing 7/44: [dune: ocaml bootstrap.ml] [ocamlbuild: m
Processing 7/44: [dune: ocaml bootstrap.ml] [ocamlbuild: m
Processing 7/44: [dune: ocaml bootstrap.ml] [ocamlbuild: m
Processing 7/44: [dune: ocaml bootstrap.ml] [ocamlbuild: m
Processing 7/44: [dune: ocaml bootstrap.ml] [ocamlbuild: m
Processing 8/44: [dune: ocaml bootstrap.ml] [ocamlbuild: m
Processing 8/44: [dune: ocaml bootstrap.ml] [ocamlfind: ma
∗ installed ocamlfind.1.8.1
∗ installed base-bytes.basetrap.ml]
Processing 11/44: [dune: ocaml bootstrap.ml] [ocamlbuild: m
Processing 11/44: [dune: ocaml bootstrap.ml] [ocamlbuild: m
Processing 11/44: [dune: ocaml bootstrap.ml] [ocamlbuild: i
∗ installed ocamlbuild.0.14.0
Processing 12/44: [dune: ocaml bootstrap.ml] [topkg: ocaml
∗ installed topkg.1.0.1ootstrap.ml]
Processing 14/44: [dune: ocaml bootstrap.ml] [react: ocaml
∗ installed react.1.2.1ootstrap.ml]
∗ installed dune.2.1.2.exe build]l]
Processing 18/44: [camomile: ocaml configure.ml] [cppo: dun
Processing 19/44: [camomile: ocaml configure.ml] [cppo: dun
Processing 20/44: [camomile: ocaml configure.ml] [cppo: dun
Processing 21/44: [camomile: ocaml configure.ml] [cppo: dun
Processing 21/44: [camomile: ocaml configure.ml]
∗ installed cppo.1.6.6
Processing 22/44: [camomile: ocaml configure.ml]
Processing 23/44: [camomile: ocaml configure.ml]
Processing 23/44: [camomile: dune build] [dune-private-libs
Processing 23/44: [camomile: dune build] [dune-private-libs
Processing 23/44: [camomile: dune build] [dune-private-libs
∗ installed mmap.1.1.0
Processing 24/44: [camomile: dune build] [dune-private-libs
Processing 24/44: [camomile: dune build] [dune-private-libs
∗ installed result.1.4
Processing 25/44: [camomile: dune build] [dune-private-libs
Processing 25/44: [camomile: dune build] [dune-private-libs
Processing 25/44: [camomile: dune build] [dune-private-libs
Processing 26/44: [camomile: dune build] [dune-private-libs
∗ installed ocplib-endian.1.0
Processing 26/44: [camomile: dune build] [dune-private-libs
∗ installed dune-private-libs.2.1.2
Processing 28/44: [camomile: dune build] [dune-configurator
∗ installed camomile.1.0.2m: dune build]
Processing 30/44: [charInfo_width: dune build] [dune-configurat
∗ installed dune-configurator.2.1.2d]
Processing 32/44: [charInfo_width: dune build] [lwt: dune b
∗ installed charInfo_width.1.1.0
∗ installed zed.2.0.4uild] [zed: dune build]
∗ installed lwt.5.1.1uild]
Processing 38/44: [lwt_log: dune build] [lwt_react: dune bu
∗ installed lwt_log.1.1.1 build]
∗ installed lwt_react.1.1.3uild]
∗ installed lambda-term.2.0.3uild]
∗ installed utop.2.4.3uild]
Done.
<><> lwt.5.1.1 installed successfully <><><><><><><><><><><><><><><><><><><><><>
=> Lwt 5.0.0 has made some minor breaking changes. See
https://github.com/ocsigen/lwt/issues/584
ysg4206@YSG4206 ~
$ utop
All three packages are using