Your production web stack in 2020

[anuragsoni]

@keleshev I haven’t landed on a connection pooling solution for an async Postgres client at the moment (for lwt I think ocsigen project has a resource pool library that can be plugged in very easily).

My goal is to work towards contributing a new driver for caqti so I can benefit from connection pooling and other features caqti provides while still being able to use a pure OCaml Postgres client.

In the meantime I’d lean towards a solution like pgbouncer (https://www.pgbouncer.org/usage.html) that can handle session level connection pooling for Postgres.

[kanishka]

Your observability is pretty thorough. Are you looking at open telemetry or honeycomb next?

[anuragsoni]

No plans at looking at these at the moment.
Honeycomb looks interesting. I wasn’t aware of them so thanks for pointing that out!

[Khady]

Backend side:

• lwt
• httpev (from devkit), routes, atdgen + some code generation to generate a typed web server and http clients (bucklescript and native) from a specification
• slowly getting tyxml in for static and correct html generation
• ocaml-mysql + sqlgg, elasticsearch + esgg, redis as databases
• extprot as a serialization format for a lot of internal data exchange or storage
• log and logstash (from devkit) for logging and monitoring
• web (from devkit) as an http client on top of ocurl
• mostly ounit and dune expect tests for testing

Frontend side (subject to change in the future)

• bucklescript + reason react
• bindings to a lot of different libs for specific usage (less “original” code that is not product specific)

The main benefit of all this code is that we can have types that are propagated from the DB to the frontend through the web server, thanks to everything being written in ocaml.

[Haudegen]

Backend:

• Webmachine (Cohttp)
• Lwt
• Postgresql-ocaml, Ezpostgresql
• Yojson
• Xmlm
• Jingoo
• Csv, Csv-lwt
• Netclient (email)
• Safepass

Frontend: Vue.js

REST-clients: Cohttp.

[smondet]

Not my “full-2020-stack” but I just deployed the port of https://tqtezos.github.io/TZComet/ from Tyxml_js/React/ReactiveData to Tyxml-lwd from @let-def

• cf. https://github.com/let-def/lwd — OCaml 2020 had a talk about Nottui/Lwd
• it’s a client-side-only JSOO app, but it can talk to public Tezos nodes, so technically the “backend” is Cohttp/Lwt-based likely behind nginx-like proxies.

Pretty happy with Tyxml_lwd:

• I initially hit a couple of bugs, that @let-def fixed very fast (I think I was the first bigger user).
• It is overall not that different from the Tyxml_js combo, but the API is simpler and feels way cleaner:
  • there is no “reactiveData” awkward middle-person (or let’s say its equivalent is better integrated in the Lwd case)
  • there are less (no?) traps with React.S.value throwing exceptions or == equality being used in subtle ways
  • There is no Tyxml_js.Html Vs Tyxml_js.Html.R distinction, which makes it easier to build nicer UI abstractions on top of HTML.
  • Bonus points for let empty () = Lwd.pure Lwd_seq.empty and
    let ( % ) a b = Lwd.map2 ~f:Lwd_seq.concat a b
    allowing me to have an efficient and very practical “HTML monoid” interface.

[lepoetemaudit]

Not exactly a full web stack as it’s essentially a REST API I’m working on, but:

Backend:

• Httpaf
• Lwt
• pgocaml (with the ppx)
• Yojson
• Cohttp (HTTP client)
• amqp
• redis-lwt

Routing etc., is handled by internal libs wrapping Httpaf, I developed some functor and ppx stuff for defining an OpenAPI HTTP server in a type safe and declarative way so it’s pretty specialised.

Frontend (for internal utilities and prototyping):

• js_of_caml
• ocaml-vdom

[rizo]

Backend

• opium - Currently using master (includes httpaf) with a couple of custom unpublished modifications. Overall I’m happy with how simple it is and like the direction of the project. Many auxiliary utilities for session, migrations, auth management are still missing.
• sqlgg - Takes care of generating DB management modules based on SQL schema. A very nice tool when it works. The error messages are pretty confusing sometimes and there are some limitations with NULL handling in generated queries. Highly recommend this approach for projects with simple models.
• yojson/cstruct/csv - we work with both binary and textual formats in our APIs. These are some of the serialisation packages we currently use. Very reliable overall. Recently started using BARE which is a bit more experimental.
• coap (unpublished) - An HTTP-like protocol for constrained/embedded devices. We wrote our own implementation and have been testing it for a few month now.
• mqtt-client (unpublished) - used to sync real-time data between our servers. The implementation is a bit incomplete but has been working reliably.
• irmin - used for long-term distributed storage of our time-series events.
• swagger - used to generate a specification for our API. The integration is still work in progress, but it’s nice to be compatible with standard tooling.

Frontend

• ReasonReact - provides a very solid development experience, but we have been recently considering switching to something based on js_of_ocaml. Not only I’m somewhat confused by the latest ReScript changes, but also need to share some small libraries between the backend and the frontend.
• Sx (unreleased) - a custom CSS-based styling library inspired by https://tailwindcss.com.

[Konstantin_Olkhovski]

Backend:

• Cassandra as a database (unpublished bindings to DataStax cpp driver’s C API along with unpublished ppx for type safe code generation)
• In-house Httpaf based (we use fork by @anmonteiro) HTTP framework specialized for the way we work with microservices (framework also supports node.js platform, but bypassess Httpaf and uses node.js HTTP API directly for performance)
• Routes for HTTP routing
• Lwt for concurrency library
• stdlib: Core for native/Core_kernel for jsoo
• atdgen for API types, sometimes yojson directly
• prometheus (https://github.com/mirage/prometheus) for metrics reporting
• logs (https://github.com/dbuenzli/logs) for logging
• alcotest, ppx_expect, ppx_inline_tests for tests
• crowbar for fuzzing

CI:

• opam-bin greatly helps speed up our CI pipelines, and beats caching _opam folder in our environment

Frontend:

• BuckleScript with Reason (trying to figure out what the hack we should do with this ReScript story…)
• React as a framework
• JS promises for concurrency

The most challenging part of building all this was Httpaf-based framework. We started with vanilla Httpaf, forked Lwt adapter to bring in TLS and some other features. Our adapter did not work really well (it was quickly hacked together), and we stumbled upon some bugs in Httpaf that manifested in our peculiar use case (storage systems). Then we tried Antonio’s fork, which had those bugs fixed, and found some new bugs there Many thanks to @anmonteiro, all bugs were sorted out really fast! We wouldn’t be able to push our HTTP framework to a usable state without his work and help. So far everything works pretty stable and we have a service on this stack running in production.

[ruyblast]

All with eliom too (with integrated pgocaml, js_of_ocaml and lwt). I use also nginx as a reverse proxy (mainly for TLS), and I use Foundation as a css framework (I find it really above others I tried before).

[keleshev]

I’m overwhelmed by all responses in this thread, I think all of these setups are super interesting and each of them deserves a blog post or something!

[roddy]

This is for a privateish personal project with a couple of hundred users. I’ve not been working on it in the past couple of months but it’s still in use.

Backend:
Database - Postgres/caqti/ppx_rapper
Framework - master opium with httpaf, my own framework on top of that (not sure it is stable enough to recommend for others currently)
Stdlib - core
Concurrency - Lwt
Other - In this project I also use aws-s3, and my own HTTP client library (based on cohttp) for things like connecting to slack and email sending APIs (although for the latter I would use letters if starting from scratch again, I just couldn’t find a sufficiently easy to use lwt SMTP client when I began). In other projects I’ve used ocaml-websocket, which seems good. For the first project, I send web push notifications by shelling out to a Python script; this is almost possible to do in OCaml but just needs an implementation of ECDSA in a crypto library.

Frontend: ReScript (with Reason (not ReScript) syntax) and ReasonReact. I think it is very useful to share types (or slightly more, I have experimented with sharing specifications of REST endpoints) between frontend and backend, and rarely useful to share significant amounts of logic. For apps where this is true I definitely like ReScript over Js_of_ocaml, especially if you are doing a reasonable amount of JS interop. I use yojson on the frontend as well as the backend, but in hindsight this is probably a bit silly and isn’t required to share type definitions (would be better to use decco instead). I wrote some bindings to SWR which I think is nice. Just use the builtin stdlib, I don’t find anything more to be necessary for the kind of frontend stuff I’m doing.

Deployment:
Backend - a few server processes in individual docker containers, behind (also dockerized) nginx. Postgres also in docker (possibly not necessary).
Frontend - nginx in docker.
Everything runs on one VPS, deployed/managed with ansible and bash. I do various admin tasks from a docker container with dune utop for the project.

[hcarty]

Like a few others here, my projects have been more backend/API focused, so very little frontend code to speak of. It’s mostly internal systems talking to one another to form data pipelines.

• opam, dune, latest OCaml, stdlib, Lwt, yojson, cmdliner and a handful of other libraries as the base
• cohttp with a custom routing, logging, query parsing layer for servers (to be released when I can make time to package it properly)
• cohttp via ezrest for clients
• ocaml-mysql via ezmysql talking to mariadb when a relational database is needed. Mwt helps bridge the concurrency gap between ocaml-mysql and Lwt
• Elasticsearch for lots of things
• logs for logging, rendered to JSON and shipped to Elasticsearch
• alcotest, mdx and ppx_expect for tests
• Services are built + deployed in containers on Linux, bare VMs when running on Windows
• Internal command line tools our team and those around us are built as native binaries for Linux, WIndows, macOS

[mro]

writing personal mini-services scaling towards n=1, I do

• CGI “on the metal” (env+stdio)
• deliver raw data (e.g. gpx)
• make nice in the browser via xslt and require no Javascript.

e.g. https://demo.mro.name/geohash.cgi

P.S.: Doing https requests on arm/linux would be nice. Currently painful.

[lambdam]

Hello,

I am mainly a Clojure developer in my professional life but am very interested
in OCaml for quite some time.

I am interested in the question of web development also for a long time and here
are my impressions:

1. Ocsigen seems, from the “outside world”, to be the goto solution for web
   development in OCaml but I’m surprised to see nobody using it. If I get it
   well, using Eliom is not using the whole package of Ocsigen.
2. The whole OCaml ecosystem seems to switch to dune. When I read the docs of
   Ocsigen, it’s full of Makefile commands. Is it to understand that Ocsigen
   is behind the trend in terms of “modernness”?
3. By reading the answers, I understand that many people use their own set of
   small plugable libraries (this is actually what people do in the Clojure
   ecosystem where there is no fullstack framework at all). Is it the common way
   of doing web development in the OCaml community nowadays?

With Clojure, we are very used to REPL based programming. At dev time, we often
use libraries that implement the “Reloaded workflow”. It is a system/dependency
injections approach. At dev time it enables one to start the system with proper
stateful dependencies (like database) and the cleanly stop the system, reload
the code and restart the system. Two modern versions of this approach are
integrant and
clip. Is there something similar in the OCaml
world? Or is it useless due to OCaml compiler speed and some other mechanism
that handles stateful part cleanly?

Thanks a lot

[ruyblast]

Hi @Lambdam,

I did not mention it, but I use the whole stack of Ocsigen (from ocsigen-start). Maybe some people are using the term Eliom to talk about ocsigen stack, I am guilty of doing so. I was very little experienced in web development (a bit of js and typescript) when I decided to use it, and I did not try ocaml alternatives so I can not talk about them. I would not have been comfortable with picking and using different libraries, and this standalone framework was perfect to me.
I had to go through a lot of documentation, maybe for a full-time month (going through tutorials too), but I found it quite clear and written with care, so it was ok. I think when one wants to do something quite specific, it is a bit a pain to go through all the docs to catch the logic, because ocsigen does “everything” (including many things I could not imagine possible ^^). Too me, this is a pro when you want to have a lot of options in one tool (client-server app, cross-platform, reactive programming, elm architecture with vdom…) but a con when you have limited needs that you know will cover your project. I think one who learned to use different libraries over times have a lot of options too of course, but the learning path seems less linear.

Ultimately, when you have to decide where to start, I think it depends of your use case and how much you are familiar with some practice of web programming. Hopefully, the community is very helpful in my experience, I am sure we will help you find a tooling set that suites your need or your expectations.

I do not know what most people are doing, but there are definitely serious industrial players using ocsigen (starting from besport), and it is still under development as far as I know, so it is not “old stuff”. Dune is really a great tool, and I personally would prefer ocsigen-start would build with it. However the Makefile is definitely doing the job right, so I understand they currently focus on other challenges they face (I certainly would do the same choice). I hope some day I will have resources to contribute on this

For REPL based programming, I know nothing about it, but as you state ocaml compilation is fast (and even faster with dune :o).

I think ocaml is a really good language for the web. I notice some like there type preserved from database storage to front-end, I agree so much on this! To me, web use cases are vast and different, that is why web stacks are heterogeneous (it is the case in most languages). I think all people here present good stacks that met their need. Hope you will find yours and that you enjoy, in ocaml or another language

Best

[Vincent_Balat]

At Be Sport our stack is 100% Ocsigen (Lwt, js_of_ocaml, Eliom, Tyxml…)

Be Sport is a social network about sports (Web and mobile apps).

A few words about the stack we use:

• The mobile (iOS and Android) and Web apps are written with one single multi-tier Eliom code (we use Ocsigen-Start as a basis). Lwt + js_of_ocaml
• We use mainly postgresql with PGOcaml (even if we are not 100% happy with this solution). We use a pool a read-only db servers (currently with pgpool). We use Elasticsearch for our search engine.
• Everything is hosted on AWS, with docker containers, and Elastic load balancer to dispatch to several instances of Ocsigen server (with SNS and SQS to dispatch notifications to all servers, and Firebase to send notification to mobile apps).

[mudrz]

this is really helpful, it would be really appreciated to include some info on user base / requests per second/day, whether it is a commercial project and etc. to get a better understanding on whether the stack is battle tested;

also consider adding missing frameworks to techempower, so that we have a better visibility on the expected performance of different frameworks and stacks

TechEmpower Web Framework Performance Comparison

(code here: https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/frameworks/OCaml)

[avsm]

@roddy

and my own HTTP client library (based on cohttp)

@roddy, this looks really interesting for upstream cohttp. We’d definitely be up for merging this into the upstream library if you would consider submitting a PR. Cohttp’s Client is (deliberately) low-level HTTP, and the plan was always to have something higher-level that would handle issues of handling redirects and pipelining. Quests looks pretty much like it fits this bill at first glance.

[rgrinberg]

Nothing against Quests as it looks like a fine library, but perhaps we should high levels on their own and focus on providing a solid foundation? There are other client libraries that look pretty good (resto comes to mind). It seems advantageous to keep cohttp as a shared & unified layer for various http types and allow various client implementations to be compatible. I can see plenty of other client implementations that may reuse these foundations; nsurl and curl based clients come to mind.