Question on native executable dependencies

Ecosystem
,
1

Hello everyone,

I am new to OCaml and I am currently testing some small programs just to see how I could use it.

In that process, I have written a basic program that fetches a web page from a given url, and I have come up with this (using the ezcurl package):

open Ezcurl

let url = "https://info.cern.ch/"

let () =
  let res = Ezcurl.get ~url () in
  let output = (match res with
  | Ok response -> response.body
  | Error (_,s) -> "Error: " ^ s) in
  print_endline ("Result:\n" ^ output)

The program works as expected (it compiles and runs fine as a native executable). The thing is, I had a look at the executable dependencies and hereā€™s what I got:

$ ldd _build/default/test_curl.exe
        linux-vdso.so.1 (0x00007ffd749f7000)
        libcurl-gnutls.so.4 => /lib/x86_64-linux-gnu/libcurl-gnutls.so.4 (0x00007fd8e38d4000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd8e37f5000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd8e3614000)
        libnghttp2.so.14 => /lib/x86_64-linux-gnu/libnghttp2.so.14 (0x00007fd8e35e5000)
        libidn2.so.0 => /lib/x86_64-linux-gnu/libidn2.so.0 (0x00007fd8e35b4000)
        librtmp.so.1 => /lib/x86_64-linux-gnu/librtmp.so.1 (0x00007fd8e3593000)
        libssh2.so.1 => /lib/x86_64-linux-gnu/libssh2.so.1 (0x00007fd8e3552000)
        libpsl.so.5 => /lib/x86_64-linux-gnu/libpsl.so.5 (0x00007fd8e353e000)
        libnettle.so.8 => /lib/x86_64-linux-gnu/libnettle.so.8 (0x00007fd8e34f0000)
        libgnutls.so.30 => /lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007fd8e32d4000)
        libgssapi_krb5.so.2 => /lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007fd8e3282000)
        libldap-2.5.so.0 => /lib/x86_64-linux-gnu/libldap-2.5.so.0 (0x00007fd8e3221000)
        liblber-2.5.so.0 => /lib/x86_64-linux-gnu/liblber-2.5.so.0 (0x00007fd8e3211000)
        libzstd.so.1 => /lib/x86_64-linux-gnu/libzstd.so.1 (0x00007fd8e3155000)
        libbrotlidec.so.1 => /lib/x86_64-linux-gnu/libbrotlidec.so.1 (0x00007fd8e3148000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fd8e3129000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fd8e3b22000)
        libunistring.so.2 => /lib/x86_64-linux-gnu/libunistring.so.2 (0x00007fd8e2f71000)
        libhogweed.so.6 => /lib/x86_64-linux-gnu/libhogweed.so.6 (0x00007fd8e2f28000)
        libgmp.so.10 => /lib/x86_64-linux-gnu/libgmp.so.10 (0x00007fd8e2ea7000)
        libcrypto.so.3 => /lib/x86_64-linux-gnu/libcrypto.so.3 (0x00007fd8e2a25000)
        libp11-kit.so.0 => /lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007fd8e28f1000)
        libtasn1.so.6 => /lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007fd8e28da000)
        libkrb5.so.3 => /lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007fd8e2800000)
        libk5crypto.so.3 => /lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007fd8e27d3000)
        libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007fd8e27cd000)
        libkrb5support.so.0 => /lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007fd8e27bf000)
        libsasl2.so.2 => /lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007fd8e27a0000)
        libbrotlicommon.so.1 => /lib/x86_64-linux-gnu/libbrotlicommon.so.1 (0x00007fd8e277d000)
        libffi.so.8 => /lib/x86_64-linux-gnu/libffi.so.8 (0x00007fd8e2771000)
        libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007fd8e276a000)
        libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007fd8e2759000)

I guess itā€™s normal but I was somewhat surprised by the number of dependencies.
My question is: is there a known method to reduce these somehow (like static linking or other mechanism) ?

I am not too worried by this at this stage, itā€™s just that Iā€™m wondering about portability in case of deployment of my executable to another Linux box.

Thanks in advance,

My environment:

$ uname -a
Linux 55e345739265 6.1.0-13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.55-1 (2023-09-29) x86_64 GNU/Linux
$ ocamlopt --version
4.13.1
$ opam --version
2.1.2
$ dune --version
3.14.0
$ opam list
# Packages matching: installed
# Name        # Installed # Synopsis
base-bigarray base
base-threads  base
base-unix     base
conf-libcurl  2           Virtual package relying on a libcurl system installation
dune          3.14.0      Fast, portable, and opinionated build system
ezcurl        0.2.4       Friendly wrapper around OCurl
ocaml         4.13.1      The OCaml compiler (virtual package)
ocaml-config  2           OCaml Switch Configuration
ocaml-system  4.13.1      The OCaml compiler (system version, from outside of opam)
ocamlfind     1.9.6       A library manager for OCaml
ocurl         0.9.2       Bindings to libcurl

My dune file:

(executable
  (name test_curl)
  (libraries ezcurl)
  (modes byte native))
2

The blog post Generating static and portable executables with OCaml may help you.

2 Likes
3

In addition to the linked blog post, you might find it informative to compare the output you get to ldd /usr/bin/curl - on my machine most of the dependencies you list are dependencies of curl (or more likely libcurl) itself.

1 Like
4

Thanks for the pointer.

I went through the rabbit hole and managed to get somewhere in the end (but not without sweat as you can imagineā€¦:).

I used the tips given on that page, as well as another hint seen somewhere on the net about libcurl being compiled with tons of features (and therefore dependencies) by default.

I ended up re-compiling libcurl from sources with many features disabled, and using that custom-built libcurl to link with my ocaml program.

For the record (in case anyone hits the same issue in the future), libcurl was configured+built using:

$  ./configure --disable-debug --disable-ftp --disable-ldap --disable-ldaps --disable-rtsp --disable-proxy --disable-dict --disable-telnet --disable-tftp --disable-pop3 --disable-imap --disable-smb --disable-smtp --disable-gopher --disable-manual --disable-ipv6 --disable-sspi --disable-ntlm-wb --disable-tls-srp --without-nghttp2 --without-libidn2 --without-libssh2 --without-brotli --with-openssl

I also had to install a couple extra libraries on my debian box to have it compiling:

$ apt install libpsl-dev libidn2-dev libunistring-dev libssl-dev

I then fiddled with my dune file, as inspired by the link you referred to, until the program compiled.

My dune file currently looks like this:

(executable
  (name test_curl)
  (libraries ezcurl)
  (flags (:standard 
          -noautolink
          -cclib -L/usr/lib/x86_64-linux-gnu
          -cclib -L/usr/lib/ocaml
          -cclib -L/root/.opam/default/lib/curl
          -cclib -lcurl-helper
          -cclib -lm
          -cclib -lthreadsnat
          -cclib -lunix
          -cclib /root/curl-8.6.0/lib/.libs/libcurl.a
          -cclib /usr/lib/x86_64-linux-gnu/libpsl.a
          -cclib /usr/lib/x86_64-linux-gnu/libidn2.a
          -cclib /usr/lib/x86_64-linux-gnu/libssl.a
          -cclib /usr/lib/x86_64-linux-gnu/libcrypto.a
          -cclib /usr/lib/x86_64-linux-gnu/libunistring.a
          -cclib /root/.opam/default/lib/curl/curl.a
          -cclib /root/.opam/default/lib/ezcurl/core/ezcurl_core.a
          -cclib /root/.opam/default/lib/ezcurl/ezcurl.a
          -cclib /usr/lib/ocaml/stdlib.a
          -cclib /usr/lib/ocaml/threads/threads.a
          -cclib /usr/lib/ocaml/unix.a))
  (modes byte native))

It can probably be trimmed down some more (I have yet to look into this), but at least it gives a working native executable.

Looking at the dependencies, I now get:

$ ls -l _build/default/test_curl.exe 
-r-xr-xr-x 1 root root 8863632 Feb 21 22:26 _build/default/test_curl.exe

$ ldd _build/default/test_curl.exe 
        linux-vdso.so.1 (0x00007ffdaffec000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fbdca58b000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fbdca3aa000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fbdcad78000)

ā€¦which seems much more lightweight (or at least self-contained) to me.

I am wondering if there is a simpler way to achieve that, because I just went ā€˜trial and error styleā€™ on this.

Thanks a lot for your help so far anyway. Any further comments appreciated.

5

Another option could be to use Cohttp and the tls package (not ssl which pulls in OpenSSL), given they donā€™t have any C dependencies they would get statically compiled into your executable.

1 Like
6

Yes indeed. It confirms that curl/libcurl is by default built with all batteries included. Which is probably the most convenient for most end users.

I just wasnā€™t aware that curl was supporting that many protocols / use cases.

Admittedly, as application developers, our needs can differ somewhat, especially if we aim for a trimmed-down solution.

7

Thanks for the suggestion. I am going to look into this.
Iā€™ll post back any (positive or negative) result here.

8

Posting results of my quick survey of the Cohttp package.

My minimal test program, translated for that package, looks like the following:

open Cohttp_lwt_unix

let url = "https://info.cern.ch/" 

let get_url = Lwt.bind 
  (Uri.of_string url |> Client.get) 
  (fun (_, body) -> Cohttp_lwt.Body.to_string body)

let () =
  let body = Lwt_main.run get_url in
  print_endline ("Response:\n" ^ body)

The Cohttp package was installed using:

$ opam install lwt cohttp cohttp-lwt-unix tls-lwt

ā€¦which added a whopping 70 extra packages to my opam repository.

My dune file simply contains:

(executable
  (name test_cohttp)
  (libraries lwt cohttp cohttp-lwt-unix)
  (modes byte native))

I could build it without issues, and the compiled program behaves exactly as expected (same as previous one).

The good surprise is that the generated output is, as advertised by @Leonidas, rather light on dependencies:

$ ldd _build/default/test_cohttp.exe
        linux-vdso.so.1 (0x00007ffc05f4d000)
        libgmp.so.10 => /lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f37ccd41000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f37ccc62000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f37cca81000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f37cd560000)
$ ls -l _build/default/test_cohttp.exe 
-r-xr-xr-x 1 root root 15277768 Feb 22 15:38 _build/default/test_cohttp.exe

I must admit that it required much less effort to produce an (almost) self-contained native executable (compared to Ezcurl), due to fewer system dependencies. So I do concur that Cohttp is a good pick for that scenario :wink:

The tradeoff being, at least for a beginner like me, that this package makes use of some more involved OCaml concepts (like monadic constructs). I guess in time I wonā€™t see that as an impedimentā€¦

9

Thatā€™s true, it does introduce monadic IO to your code. You can rewrite it using the let*-syntax, then it becomes more similar to non-monadic code (untested):

Besides Cohttp there is also httpaf but I am not sure what its current status is.

10

If all you are doing comprises getting a web page from a URL, you donā€™t need SSL/TLS, and you want to avoid dependencies, it is fairly trivial to write a function which makes a GET request using Lwtā€™s built-in socket functions, so avoiding any additional library requirements.

If you do need SSL/TLS then the ssl (and lwt_ssl) packages in opam are fairly light on dependencies. Also, for simple scripts written in OCaml as an alternative to using bash, in the past I have launched the curl executable by means of Unix.create_process and piped the webpages on stdout. The latter avoids all hard dependencies, at the cost of launching a new process for reading a page.

11

This is what curly does, FYI (and it has very few dependencies).

12

If you want to look-up on HTTP libraries available from the OCaml ecosystem, you should take a look on this great post: Simple, modern HTTP client library? - #14 by edwin

It summarizes available libraries with dependencies and the discussion on this subject is interesting. For the example, Iā€™m more into the idea of having only OCaml dependencies instead of using a C library that may require as much code and dependencies as if it were made in OCaml.

Another point, this time related to the production of a static executable, is that you can currently produce portable and static executables using Esperanto - however, the work required is certainly greater than a simple executable compiled with OCaml and the -static option (this article describes how to do this).

5 Likes
13

Cohttp has also been ported to Eio and can use direct-style concurrent I/O now instead of monadic I/O: ocaml-cohttp/cohttp-eio/examples/client1.ml at master Ā· mirage/ocaml-cohttp Ā· GitHub

1 Like
14

Thanks for the heads-up - that is indeed a handy utility for scripting.

15

Thanks for the pointer. Itā€™s interesting that you bring this up because I did come across it as I was looking for a ā€œcheap and cheerfulā€ OCaml HTTP package. This topic is indeed great for anyone looking to make an informed choice among the multiple options that are out there. And itā€™s precisely that topic that made me give Ezcurl a try (over any other listed package) considering how few system dependencies it was stated to require.

Even though, I have to say, that I find the assertion of Ezcurl having only 1 system dependency to be somewhat misleading given what you obtain by default (as I experienced and described in my initial post).

I think there should be an asterisk saying ā€œyou can get down to 1 dependency for Ezcurl if a) you work hard to strip libcurl from all un-needed dependencies and b) donā€™t need SSL (which then voids the ā€˜OpenSSL-enabledā€™ checkmark in the ranking table)ā€. :wink:

Anyway itā€™s just me rambling, I really appreciate and respect the time/effort that is put into making such topics that are especially useful for beginners like me (and I am glad that such topics exist!).

That sounds like ā€¦ an impressive feat.
I didnā€™t know this was possible in OCaml. Sounds like thereā€™s some wicked technical wizardy inside ! Thank you for the tip (and congratulations since you seem to be the author of that marvel), I will definitely check that out.

1 Like
16

This may very well be exactly what I wished for.

I have been reading a bit more about Cohttp and the lwt library it is based on and, while I understand that itā€™s probably the ā€œstate of the artā€ in the domain, at this stage I am still a bit wary of the extra cognitive load that the use of monadic constructs is going to introduce into my programs.

The thing is, I have been dabbling with Haskell a bit (which is a wonderful language to learn by the way), I get what monads bring to the table, yet I canā€™t get over the fact that I find monadic-code harder to read and reason about, all things considered. Debugging experience is also another point of concern.

Itā€™s probably my lack of experience talking, but I tend to prefer OCamlā€™s more balanced way of not forcing monads on you on every effectful operation.

So, thanks for mentioning that ā€œclassicalā€ alternative :+1:

1 Like