Reflections on my first completed application in OCaml

[cdaringe]

Greetings! I’m a new OCaml user. The intent of this post is simply to share my experience thus far in OCaml and hopefully generate some casual conversation about how other community users think about & use OCaml. What do I hope to get from this post? Critiques, affirmation, tips, whatever. I learned OCaml alone, in a bubble, during pandemic lockdown. Was my experience the classic OCaml learning experience, or was it unique in some way? I’m hoping to garner feedback from other community members. Talk at me!

What’s my project?

• https://github.com/cdaringe/freshawair

tl;dr, freshawair is a system that collects and presents air quality datas from my in-home air quality monitor.

Awair ships a mobile app to observe your air quality stats, but the app doesn’t let you observe data more than a week old. Further, the app does not let you bin the timeseries data. I wanted to be able to see macro, seasonal trends on various air quality metrics in my home. Lucky for me, the sensor unit exposes a HTTP API, allowing me to capture data easily and get busy. Sounds like a fun COVID “time to learn OCaml project!”

Why OCaml?

My key interests in selecting OCaml for this project were:

• Binaries. code that runs artifacts on the metal. no VM required! having a thin runtime was not inhibiting.

  • I was originally targeting ARM on a low powered device, so this was essential.

• functional language. I’m tired of the same old problems from other-langs. I’ve been on a kick, learning languages that seem to support or tend towards correctness.

• Effects. I recently worked on https://effects.js.org, and I wanted to experience effects, first-class, in a language.

How does the project work?

There are four parts:

• agent - ocaml - collects data from the awair http api, forwards data to the freshawair server
• server - ocaml - hosts api, hosts ui static assets
• database - timescaledb - omitted from discussion
• ui - react/typescript - omitted from discussion

The agent and the server could have been combined into one binary. However, the original designed called for the server and DB to live in the cloud, while the agent would exist in my local network only. The design still supports this, even though I now just deploy all of the things side-by-side on my NAS.

How’d it go?

Overall, pretty good. I’ve grown quite affectionate towards ML. Having done a bit of Elm before, and just a nibble of recreational Haskell, OCaml fundamentals were not hard to pick up. The module system, figuring out where common functionality lives (or if common functionality even existed in the core lib), & perhaps configuring builds all cumulatively took a little more time than I would have liked, but hey–that’s learning. Regardless of the the hype around multicore, which I am certainly excited about, pragmatically effects don’t exist in ocaml yet. No ARM support, ppx_tools incompat, etc in the multicore compiler. Effects was one of my primary decision drivers in selecting OCaml, so I felt a bit bamboozled as I slowly uncovered that this lack of compatibility with mainline compilers was the status-quo. Even so, I have no remorse.

What was great?

• discord community. thanks everyone for helping me with so many questions

• docker images/infrastructure. just top tier work right there.

• ocaml platform editor support for vscode

  • it has its quirks, but it’s also new. having a clear “this is the tooling to use” directive, and having it actually work (most of the time) was quite nice.

• responsive community

  • during the course of the project, i added comments in maybe 6-12 community GitHub projects. i got feedback in every single one. OCaml may be a small community, but it’s real people committed to the cause ;). what a delight this was. seriously! you are nerds helping helping nerds. be proud.

• Core as a std lib was easy to explore, and was a great start to hack away at an empty file.

• Opium. I originally wrote my server in plain-jane cohttp, and recently refactored it to opium. Love it!

What was just OK

It’s easier to remember recent suffering for me, so forgive the fact that the below lists are a bit longer than the “great!” list.

• Real World OCaml is a good book–truly. Even so, I personally do not think that it is the right book to bootstrap newbies, nor get people excited about OCaml. It’s more of a handbook, versus a guided tutorial on how to start doing productive things™ common to software development. It certainly beats the manual, and certainly beats nothing–100%.

  • Richard Feldman has words to say on the matter (https://corecursive.com/teaching-fp-with-richard-feldman/), where you can replace any mention of haskell with ocaml. To poorly distill his thoughts, FP pedagogy seems to focus first on underlying concepts then secondarily on application, versus focusing first on incremental, practical applications, then secondarily exploring required concepts during the ride. Having “learned” haskell and eventually ejecting from it, this interview resonated. OCaml is much more approachable than Haskell (better docs, tools, etc), but I’m still looking for “OCaml In Action”. RWO didn’t scratch my itch. It feels foolish to critique something that the community offered to me for free, so maybe I should just hush up :).

• ReasonML. ReasonML is what exposed me to OCaml, but once I started using OCaml, my interest in ReasonML dwindled rather quickly. That is a bummer because I love building UIs and I’m a react power user. I’m sure there’s an interesting history in this space (I’m totally oblivious), but I just can’t help but wonder what would have happened if all of that OCaml-ish UI work was reinvested in the OCaml community vs this UI-specific fork-ish project. I prefer the ML syntax, even though reason is supposedly catering to me, a TypeScript power user. When I finally ejected reason out of my OCaml project, I lost npm as a package source, and that caused some headache as I had to move fully into opam deps, vs getting deps from both places. It was pretty cool that I could co-locate .ml and .re files together, though! Maybe I’ll take some time to look at the history here (links welcomed). Anyway, I ended up doing my UI via create-react-app + typescript, to avoid the growing pains of figuring out ReScript or jsoo, while simulatenously learning OCaml :).

• json operations. It took me much too long to just figure out how to do the basics w/ JSON. I eventually landed on yojson & ppx_deriving_yojson.runtime, but was distracted by atdgen. I understand there are cost-benefit analyses required when choosing tools, but I was hoping to have landed on a solution within minutes of research. I won’t tell you how long it actually took me just to get my JSON serialization and deserialization code in place. Hint: too darn long!

• regex. https://pl-rants.net/posts/regexes-and-combinators-2/ was quite helpful

• dune & opam. both are seemingly polished tools. I wish they were married, into a single OCaml project management tool. I get that they have different roles, but #opinons.

  • Additionally, a .nvmrc pattern would be nice. For instance, I alias cd on my system to execute nvm/fnm use on every cd iff a .nvmrc is found, s.t. when getting to work for the day, I’m always have the right switch loaded.

What wasn’t great

• resolving compiler errors. no surprise.

  • commonly, in order to try and help narrow the problem space, i’d try and add explicit types, but even figuring out what types (let alone how to access to them) was often a bit tedious

  • FWIW, the reasonml error message formatter was awesome in improving OCaml compiler error messages. we should seriously consider baking that thing straight in.

• no integrated debugger. . MS is doing really great work with https://microsoft.github.io/debug-adapter-protocol/. maybe some motivated soul will be my hero someday.

• lwt promises vs core deferreds. whatever the async primitives will be in the future, we should probably dedupe this as a community. further, we should probably also bake them right into the stdlib if we are going to continue to have colored functions. i spent time studying deferreds only to later discover that most of the things I wanted to do and use were lwt-centric. RWO puts you down the deferreds path, but actual, for real ocaml seems more aligned on Lwt. Disagree? Then now we know we have a problem . I ended up refactoring a moderate amount of early code for this reason. It was a tad bit obnoxious, especially that it felt out-of-band with the learning trajectory set forth in RWO. Perhaps that’s on me :).

• Producing statically linked binaries proved unproductive. I can’t recall all of the errors I ran into, I was just hoping to flip a switch and get statically linked everything. Nope! Terrible assumption. Surely portable software can exist again one day.

• esy. It sounded like esy it’s was going to be great for bootstrapping a project, but ultimately understanding dune and opam was more productive, even as someone deeply familiar with package.json-isms. I used it for a while, and am glad to have removed it.

• string interpolation. i want automatic toplevel sprintf-like functionality in string literals. let str = "wah wah boo hoo I want ${featureName} yesterday!" ... would be pretttty cool

What’s next?

• I’d eventually like to try jsoo. I saw that some react-bindings are in work. Sweet. Even if there was an Elm-like UI lib, I’d be keen on that as well.

• Performance.

  • I’m pretty sure my postgresql cursor usage is blocking my single thread on the server. Async-ifying my postgres cursor for piping datas into my Lwt_stream seems worthwhile, if possible.
  • Request less data in the UI. Specifically, request only data that is expected to be painted. Currently I request a boat-load of data that the user may not be interested in.

• Tests. Don’t look!

• De-cruft-ify some malarkey. There’s some values hardcoded in there that really don’t belong. There’s perhaps some not-very OCamly patterns that need tidying up that suffer from beginner-isms.

Anyway. OCaml. Great language. Will I keep using it? Definitely! I’m glad to be here. Thx all!

[yawaramin]

Hi, thanks for this write-up, very interesting and super valuable to hear what new users are going through. FWIW, I think you made the right call going with TypeScript; when learning something new, it’s better to focus on one thing at a time. Btw, opam does support the .nvmrc style, see http://opam.ocaml.org/blog/opam-local-switches/

[cdaringe]

Oh nice, Ill try that out. Thanks

[EmileTrotignon]

Nice thoughts. I think I will do a library for string interpolation - it should just be a simple ppx.
You can already check out this project : https://github.com/EmileTrotignon/embedded_ocaml_templates,
that attempts to be EJS for OCaml (not every feature is here though), and provide the following syntax :

let name = "John"
let john = [%eml "Name : <%-name%>"] 

However I the ${...} syntax in a package that would not allow control structures, just interpolation, would definitely be useful. Maybe it even already exists, I will check before I try and implement it ^^

Edit : it does exists : https://github.com/bloomberg/ppx_string_interpolation

[patricoferris]

Hi @cdaringe :))

Thanks for the incredibly well written and useful write up, especially describing the things you like, think are okay and weren’t so great.

I can’t speak about how to resolve all of your pain-points (e.g. no integrated debugger) but what I can say is that:

guided tutorial on how to start doing productive things™

is actively being worked on, which includes more information (amongst other things) about

dune & opam

The (very WIP) set of “workflows” for getting productive things done in OCaml can be found here and are live here (easier to digest full list here). The goal is to fill these out more but also maintain the continuous integration which helps ensure they are working examples of how to be productive with the OCaml Platform (and friends). As I said, it’s early days but I think it is already in a position that at least one user might find something useful in there.

Good luck with the rest of your very cool project!

[cdaringe]

Beautiful. I will be trying both of these out.

[bcc32]

In case you needed more options, https://github.com/janestreet/ppx_string is another string interpolation ppx. It’s part of ppx_jane, so if you’re already using that, [%string] is available without any further work.