Your production web stack in 2020

Ecosystem
1

Hey folks, especially those running web back-ends and front-ends in OCaml/Reason: what is your stack?

There has been a similar question previously, however, two years is a long time, so, maybe, things have changed since then.

In particular:

Back end:

  • Database, database driver, migrations, any other database-access layer, for example, a ppx
  • Web server and framework
  • Concurency library (Lwt/Async)
  • Any other thing worth mentioning, like ocaml-graphql-server

Front end:

  • Transpiler: Js_of_ocaml or ReScript (BuckleScript)
  • Framework
  • Concurrency library (Lwt/Async/JS Promises)

Iā€™m asking because Iā€™m curious to see which combinations of these technologies are battle-tested. Certain kinds of issues only arise (and get solved) when used at certain scale. Itā€™s also interesting what combinations of technologies work well together. For example, httpaf+async appears like a well tested path, while httpaf+lwt seems like more experimental (my guess). Or, for example, caqti+lwt seems like a tried path, while caqti+async might not be.

To share my experience, Iā€™ve made some prototypes using Windows+Cohttp+Lwt on the back end and Linux+BuckleScript+ReasonReact+JS promises on the front end, but I canā€™t judge about the robustness of the solution. Certainly there was a lot of friction when trying to share the code between the two ā€œendsā€. Like writing it in two different languages. Which is now sort of acknowledge with ReScript becoming a fork of OCaml.

Looking forward to hear about your experiences!

17 Likes
2

At day job:

For personal projects:

Iā€™ve been really happy using Async and most of my work still uses it since core + async is an ecosystem Iā€™m most familiar with. I really like asyncā€™s monitors (probably my favorite part of the library), and the consistent experience that one gets when using the various libraries from the Janestreet ecosystem (Iā€™m really grateful for all the open source work coming out of Janestreet, as I benefit a lot from it).

Do note that while most of my list is libraries from the async side of ecosystem, Iā€™ve been experimenting with Lwt recently as iā€™m really interested in the mirage ecosystem so my set of preferred libraries might change over time as I start writing more mirage unikernels :smile:

9 Likes
3

@anuragsoni looks interesting! Do you use anything for DB connection pooling?

4

Eliom
Thatā€™s the whole stack
į•™( ā€¢Ģ€ ā—” ā€¢Ģ )į•—

5 Likes
5

@keleshev I havenā€™t landed on a connection pooling solution for an async Postgres client at the moment (for lwt I think ocsigen project has a resource pool library that can be plugged in very easily).

My goal is to work towards contributing a new driver for caqti so I can benefit from connection pooling and other features caqti provides while still being able to use a pure OCaml Postgres client.

In the meantime Iā€™d lean towards a solution like pgbouncer (https://www.pgbouncer.org/usage.html) that can handle session level connection pooling for Postgres.

5 Likes
6

Your observability is pretty thorough. Are you looking at open telemetry or honeycomb next?

1 Like
7

No plans at looking at these at the moment.
Honeycomb looks interesting. I wasnā€™t aware of them so thanks for pointing that out!

8

Backend side:

  • lwt
  • httpev (from devkit), routes, atdgen + some code generation to generate a typed web server and http clients (bucklescript and native) from a specification
  • slowly getting tyxml in for static and correct html generation
  • ocaml-mysql + sqlgg, elasticsearch + esgg, redis as databases
  • extprot as a serialization format for a lot of internal data exchange or storage
  • log and logstash (from devkit) for logging and monitoring
  • web (from devkit) as an http client on top of ocurl
  • mostly ounit and dune expect tests for testing

Frontend side (subject to change in the future)

  • bucklescript + reason react
  • bindings to a lot of different libs for specific usage (less ā€œoriginalā€ code that is not product specific)

The main benefit of all this code is that we can have types that are propagated from the DB to the frontend through the web server, thanks to everything being written in ocaml.

8 Likes
9

Backend:

  • Webmachine (Cohttp)
  • Lwt
  • Postgresql-ocaml, Ezpostgresql
  • Yojson
  • Xmlm
  • Jingoo
  • Csv, Csv-lwt
  • Netclient (email)
  • Safepass

Frontend: Vue.js

REST-clients: Cohttp.

3 Likes
10

Not my ā€œfull-2020-stackā€ but I just deployed the port of https://tqtezos.github.io/TZComet/ from Tyxml_js/React/ReactiveData to Tyxml-lwd from @let-def

  • cf. https://github.com/let-def/lwd ā€” OCaml 2020 had a talk about Nottui/Lwd
  • itā€™s a client-side-only JSOO app, but it can talk to public Tezos nodes, so technically the ā€œbackendā€ is Cohttp/Lwt-based likely behind nginx-like proxies.

Pretty happy with Tyxml_lwd:

  • I initially hit a couple of bugs, that @let-def fixed very fast (I think I was the first bigger user).
  • It is overall not that different from the Tyxml_js combo, but the API is simpler and feels way cleaner:
    • there is no ā€œreactiveDataā€ awkward middle-person (or letā€™s say its equivalent is better integrated in the Lwd case)
    • there are less (no?) traps with React.S.value throwing exceptions or == equality being used in subtle ways
    • There is no Tyxml_js.Html Vs Tyxml_js.Html.R distinction, which makes it easier to build nicer UI abstractions on top of HTML.
    • Bonus points for let empty () = Lwd.pure Lwd_seq.empty and
      let ( % ) a b = Lwd.map2 ~f:Lwd_seq.concat a b
      allowing me to have an efficient and very practical ā€œHTML monoidā€ interface.
9 Likes
11

Not exactly a full web stack as itā€™s essentially a REST API Iā€™m working on, but:

Backend:

  • Httpaf
  • Lwt
  • pgocaml (with the ppx)
  • Yojson
  • Cohttp (HTTP client)
  • amqp
  • redis-lwt

Routing etc., is handled by internal libs wrapping Httpaf, I developed some functor and ppx stuff for defining an OpenAPI HTTP server in a type safe and declarative way so itā€™s pretty specialised.

Frontend (for internal utilities and prototyping):

  • js_of_caml
  • ocaml-vdom
5 Likes
12

Backend

  • opium - Currently using master (includes httpaf) with a couple of custom unpublished modifications. Overall Iā€™m happy with how simple it is and like the direction of the project. Many auxiliary utilities for session, migrations, auth management are still missing.
  • sqlgg - Takes care of generating DB management modules based on SQL schema. A very nice tool when it works. The error messages are pretty confusing sometimes and there are some limitations with NULL handling in generated queries. Highly recommend this approach for projects with simple models.
  • yojson/cstruct/csv - we work with both binary and textual formats in our APIs. These are some of the serialisation packages we currently use. Very reliable overall. Recently started using BARE which is a bit more experimental.
  • coap (unpublished) - An HTTP-like protocol for constrained/embedded devices. We wrote our own implementation and have been testing it for a few month now.
  • mqtt-client (unpublished) - used to sync real-time data between our servers. The implementation is a bit incomplete but has been working reliably.
  • irmin - used for long-term distributed storage of our time-series events.
  • swagger - used to generate a specification for our API. The integration is still work in progress, but itā€™s nice to be compatible with standard tooling.

Frontend

  • ReasonReact - provides a very solid development experience, but we have been recently considering switching to something based on js_of_ocaml. Not only Iā€™m somewhat confused by the latest ReScript changes, but also need to share some small libraries between the backend and the frontend.
  • Sx (unreleased) - a custom CSS-based styling library inspired by https://tailwindcss.com.
12 Likes
13

Backend:

  • Cassandra as a database (unpublished bindings to DataStax cpp driverā€™s C API along with unpublished ppx for type safe code generation)
  • In-house Httpaf based (we use fork by @anmonteiro) HTTP framework specialized for the way we work with microservices (framework also supports node.js platform, but bypassess Httpaf and uses node.js HTTP API directly for performance)
  • Routes for HTTP routing
  • Lwt for concurrency library
  • stdlib: Core for native/Core_kernel for jsoo
  • atdgen for API types, sometimes yojson directly
  • prometheus (https://github.com/mirage/prometheus) for metrics reporting
  • logs (https://github.com/dbuenzli/logs) for logging
  • alcotest, ppx_expect, ppx_inline_tests for tests
  • crowbar for fuzzing

CI:

  • opam-bin greatly helps speed up our CI pipelines, and beats caching _opam folder in our environment

Frontend:

  • BuckleScript with Reason (trying to figure out what the hack we should do with this ReScript storyā€¦)
  • React as a framework
  • JS promises for concurrency

The most challenging part of building all this was Httpaf-based framework. We started with vanilla Httpaf, forked Lwt adapter to bring in TLS and some other features. Our adapter did not work really well (it was quickly hacked together), and we stumbled upon some bugs in Httpaf that manifested in our peculiar use case (storage systems). Then we tried Antonioā€™s fork, which had those bugs fixed, and found some new bugs there :smiley: Many thanks to @anmonteiro, all bugs were sorted out really fast! We wouldnā€™t be able to push our HTTP framework to a usable state without his work and help. So far everything works pretty stable and we have a service on this stack running in production.

7 Likes
14

All with eliom too :slight_smile: (with integrated pgocaml, js_of_ocaml and lwt). I use also nginx as a reverse proxy (mainly for TLS), and I use Foundation as a css framework (I find it really above others I tried before).

4 Likes
15

Iā€™m overwhelmed by all responses in this thread, I think all of these setups are super interesting and each of them deserves a blog post or something!

2 Likes
16

This is for a privateish personal project with a couple of hundred users. Iā€™ve not been working on it in the past couple of months but itā€™s still in use.

Backend:
Database - Postgres/caqti/ppx_rapper
Framework - master opium with httpaf, my own framework on top of that (not sure it is stable enough to recommend for others currently)
Stdlib - core
Concurrency - Lwt
Other - In this project I also use aws-s3, and my own HTTP client library (based on cohttp) for things like connecting to slack and email sending APIs (although for the latter I would use letters if starting from scratch again, I just couldnā€™t find a sufficiently easy to use lwt SMTP client when I began). In other projects Iā€™ve used ocaml-websocket, which seems good. For the first project, I send web push notifications by shelling out to a Python script; this is almost possible to do in OCaml but just needs an implementation of ECDSA in a crypto library.

Frontend: ReScript (with Reason (not ReScript) syntax) and ReasonReact. I think it is very useful to share types (or slightly more, I have experimented with sharing specifications of REST endpoints) between frontend and backend, and rarely useful to share significant amounts of logic. For apps where this is true I definitely like ReScript over Js_of_ocaml, especially if you are doing a reasonable amount of JS interop. I use yojson on the frontend as well as the backend, but in hindsight this is probably a bit silly and isnā€™t required to share type definitions (would be better to use decco instead). I wrote some bindings to SWR which I think is nice. Just use the builtin stdlib, I donā€™t find anything more to be necessary for the kind of frontend stuff Iā€™m doing.

Deployment:
Backend - a few server processes in individual docker containers, behind (also dockerized) nginx. Postgres also in docker (possibly not necessary).
Frontend - nginx in docker.
Everything runs on one VPS, deployed/managed with ansible and bash. I do various admin tasks from a docker container with dune utop for the project.

6 Likes
17

Like a few others here, my projects have been more backend/API focused, so very little frontend code to speak of. Itā€™s mostly internal systems talking to one another to form data pipelines.

  • opam, dune, latest OCaml, stdlib, Lwt, yojson, cmdliner and a handful of other libraries as the base
  • cohttp with a custom routing, logging, query parsing layer for servers (to be released when I can make time to package it properly)
  • cohttp via ezrest for clients
  • ocaml-mysql via ezmysql talking to mariadb when a relational database is needed. Mwt helps bridge the concurrency gap between ocaml-mysql and Lwt
  • Elasticsearch for lots of things
  • logs for logging, rendered to JSON and shipped to Elasticsearch
  • alcotest, mdx and ppx_expect for tests
  • Services are built + deployed in containers on Linux, bare VMs when running on Windows
  • Internal command line tools our team and those around us are built as native binaries for Linux, WIndows, macOS
5 Likes
18

writing personal mini-services scaling towards n=1, I do

  • CGI ā€œon the metalā€ (env+stdio)
  • deliver raw data (e.g. gpx)
  • make nice in the browser via xslt and require no Javascript.

e.g. https://demo.mro.name/geohash.cgi

P.S.: Doing https requests on arm/linux would be nice. Currently painful.

4 Likes
19

Hello,

I am mainly a Clojure developer in my professional life but am very interested
in OCaml for quite some time.

I am interested in the question of web development also for a long time and here
are my impressions:

  1. Ocsigen seems, from the ā€œoutside worldā€, to be the goto solution for web
    development in OCaml but Iā€™m surprised to see nobody using it. If I get it
    well, using Eliom is not using the whole package of Ocsigen.
  2. The whole OCaml ecosystem seems to switch to dune. When I read the docs of
    Ocsigen, itā€™s full of Makefile commands. Is it to understand that Ocsigen
    is behind the trend in terms of ā€œmodernnessā€?
  3. By reading the answers, I understand that many people use their own set of
    small plugable libraries (this is actually what people do in the Clojure
    ecosystem where there is no fullstack framework at all). Is it the common way
    of doing web development in the OCaml community nowadays?

With Clojure, we are very used to REPL based programming. At dev time, we often
use libraries that implement the ā€œReloaded workflowā€. It is a system/dependency
injections approach. At dev time it enables one to start the system with proper
stateful dependencies (like database) and the cleanly stop the system, reload
the code and restart the system. Two modern versions of this approach are
integrant and
clip. Is there something similar in the OCaml
world? Or is it useless due to OCaml compiler speed and some other mechanism
that handles stateful part cleanly?

Thanks a lot

3 Likes
20

Hi @Lambdam,

I did not mention it, but I use the whole stack of Ocsigen (from ocsigen-start). Maybe some people are using the term Eliom to talk about ocsigen stack, I am guilty of doing so. I was very little experienced in web development (a bit of js and typescript) when I decided to use it, and I did not try ocaml alternatives so I can not talk about them. I would not have been comfortable with picking and using different libraries, and this standalone framework was perfect to me.
I had to go through a lot of documentation, maybe for a full-time month (going through tutorials too), but I found it quite clear and written with care, so it was ok. I think when one wants to do something quite specific, it is a bit a pain to go through all the docs to catch the logic, because ocsigen does ā€œeverythingā€ (including many things I could not imagine possible ^^). Too me, this is a pro when you want to have a lot of options in one tool (client-server app, cross-platform, reactive programming, elm architecture with vdomā€¦) but a con when you have limited needs that you know will cover your project. I think one who learned to use different libraries over times have a lot of options too of course, but the learning path seems less linear.

Ultimately, when you have to decide where to start, I think it depends of your use case and how much you are familiar with some practice of web programming. Hopefully, the community is very helpful in my experience, I am sure we will help you find a tooling set that suites your need or your expectations.

I do not know what most people are doing, but there are definitely serious industrial players using ocsigen (starting from besport), and it is still under development as far as I know, so it is not ā€œold stuffā€. Dune is really a great tool, and I personally would prefer ocsigen-start would build with it. However the Makefile is definitely doing the job right, so I understand they currently focus on other challenges they face (I certainly would do the same choice). I hope some day I will have resources to contribute on this :slight_smile:

For REPL based programming, I know nothing about it, but as you state ocaml compilation is fast (and even faster with dune :o).

I think ocaml is a really good language for the web. I notice some like there type preserved from database storage to front-end, I agree so much on this! To me, web use cases are vast and different, that is why web stacks are heterogeneous (it is the case in most languages). I think all people here present good stacks that met their need. Hope you will find yours and that you enjoy, in ocaml or another language :wink:

Best

2 Likes