The current (“v2”) ocaml.org website was written a decade ago (presentation) and – while it has served us well – its age is showing as our community has grown. To put things in context, the opam package manager didn’t actually exist when we built the current iteration of the website.
TL;DR: preview the new package search at https://v3.ocaml.org/packages
In November 2020, the current maintainers put our heads together to rebuild the website to modern standards (“v3”). When it came to determining the goals of the website refresh, we were driven by the need to integrate the technological developments of the last decade – for example, the adoption of opam as the package management system (evidenced by close to 20,000 packages in the opam repository), and the emergence of common tools and workflows that benefit both individual developers as well as larger-scale industrial codebases.
In October 2020, the core OCaml team also issued a user survey to solicit information from our user community about priorities for the OCaml ecosystem. Here are the original results, original discuss thread, concrete suggestions and some analysis. Where appropriate, we reference the user survey results in the information below to try to ensure our decisions are as inclusive as possible.
v3.ocaml.org features
The v3 version of the website is a rewrite from scratch. This is mostly because the web technologies used in the 2011 version are outdated, and it is difficult to forward port them to a standard that meets modern accessibility requirements. Our own OCaml tooling ecosystem has also moved on tremendously in the last decade, so it was time for a fresh start. The new site features:
Integrated documentation and package management
When v2 was developed, the opam package manager didn’t exist yet and so we ended up with a fragmented ecosystem (opam.ocaml.org is separate and distinct from ocaml.org, and other package managers like GODI also had separate websites). Since then, opam metadata has become the standard in our ecosystem with over 20000 packages published.
The v3 site combines the existing package management subdomain with a new central documentation source (formerly codenamed docs.ocaml.org
) for all versions of all opam packages. This content is held directly within the new ocaml.org site and not a set of subdomains, reflecting a clear community direction. Our goal is for the live website to contain published package API docs within 30 minutes of the package being merged.
Separation of data editing from HTML/CSS generation
The v2 site combined (e.g.) fetching external newsfeeds with the HTML generation, which made it expensive to scale with new content as every build would take longer and longer.
The v3 site keeps a structured store of data as OCaml values, which is then used to generate the frontend website. We use ReScript/OCaml to generate the site content, and Ocurrent to automate the data pipelines. The data source repository is known as ood (OCaml.Org Data). Finally, the static HTML is combined with the dynamic API documentation generation and served via a Dream-based webserver.
- ocaml/ood is the OCaml.org data repository.
- ocaml/v3.ocaml.org is the ReScript-based frontend site generator
- ocurrent/ocaml-docs-ci is the OCurrent pipeline that incrementally builds the 19500+ packages and prepares them for publishing online with odoc and voodoo.
- ocaml/v3.ocaml.org-server is the Dream-based server that serves the overall site.
For users wishing to submit content fixes to the website, they need to do nothing more than clone ocaml/ood and submit the fix there, without a full website build.
Publishing original content
The v2 site syndicated from external sources, which worked well when the web (circa 2011) was centred around RSS/Atom feeds and blogging engines. The modern web (circa 2021) has moved towards social networking chatter, which greatly reduces the value of syndicating content. We also now have multiple discussion forums such as discuss.ocaml.org where original content is frequently posted.
There will there be original content on the v3 site, so that we can publish editorial content from around the community directly on ocaml.org. Original content can include interviews, featured news stories, informative forum posts, online talks and other media from past conferences, along with many other types of content.
To facilitate this without tying ourselves to a third-party website, we have deployed an instance of the open-source Peertube video hosting application, over at watch.ocaml.org. This provides an open API (free of adverts and third-party cookies) to view video material about OCaml through its rich history. The OCaml Workshop 2021 videos are being published on watch.ocaml.org as their primary source: OCaml Workshop 2021 - Watch OCaml
Accessible, sustainable and respectful of privacy
The v2 site grew organically over a decade, and so a multiplicity of ways to navigate it emerged. The v3 site design takes into account modern web design principles, restructuring the old content in accordance with methods that present it more compellingly. It is a total redesign that modernises the look and feel of the site to work well on mobile and other responsive targets, as well as being easier to navigate and more accessible.
In terms of technology, the frontend switches to Tailwind CSS and uses React/Rescript/NextJS to generate most of the site HTML. The backend data sources all remain in pure OCaml code, meaning that we retain the ability to migrate to new frontend technologies without a full rewrite in the future.
The v3 website has a simple privacy policy: there are no third-party cookies or tracking systems in place beyond server-side log analysis. This allows us to be compliant with GDPR rules, but also be accessible to the widest global audience.
In reflection of our strong committment to sustainability, we are also working on a “carbon policy” that will track and publish our energy usage across the full ocaml.org infrastructure (including the build cluster). This adds up to around ~1000 CPU cores of different architectures hosted around the world in various data centres, and we are aiming to be carbon neutral via the use of renewable energy and similar initiatives.
A quick tour of some nice bits of v3.ocaml.org
(This is accurate as of 4th Aug 2021)
- The Industrial Users page is driven from the ood data here.
- The Academic Excellence page has a world map of users as well and is driven from ood data here.
- The opam package search for ‘vg’ show a docs build (there are some broken images in the docs, which is a known issue). Try clicking around the different versions of the package as well, or searching for your own!
- The v3 tutorials are all checked with mdx to make sure that they are consistent with the latest version of OCaml.
- The events page shows past OCaml Workshops. Clicking on the 2020 one shows videos embedded from watch.ocaml.org and served directly from our new site instead of a third-party provider.
There are still lots of unfinished pages (most notably the front page is totally placeholder until we finish more of the leaf pages), but this should hopefully give you a good view on where the site is going!
2020 Survey Actions
The v3 website was partly driven by the feedback from our community as well as the core team’s discretion. We will now go through the feedback items.
Individual quotes from different answers
Pain point when Learning the OCaml Language
- “Lack of search engine friendly content. Other mainstream languages have plenty of short tutorials available on the web”: The few OCaml does have are not discoverable easily": On the new site we’re making the core tutorials easier to find. Community members are also working on new tutorial systems for beginners, and the website can link to these as they are published.
- “Up to date docs and learning materials and books”: The Language page will help users find this information in the future.
- “Poor documentation” and “Most libraries have very sparse documentation”: There are many, many comments like this one, and while we cannot solve this directly, having integrated and up-to-date API docs directly in OCaml.org should incentivise people to do a better job of adding ocamldoc comments to their code.
I am satisfied with OCaml’s package repositories, what would you change
- “Create a opam search repo”. This now exists, via v3.ocaml.org/packages and the search bar.
From the discuss thread
-
Yawaramin wrote: “Or Outreachy internships or other funded dev work at less than senior level? There are libraries out there in other languages that would be tremendously beneficial to port over to OCaml…”. Action: the OCaml Software Foundation and OCaml Labs have now run multiple Outreachy programs, with the most recent working on v3.ocaml.org itself. We need community help to add more mentors to scale this program – please volunteer!
-
Anil’s comment touches on all aspects (ocaml.org, mdx (the tutorials now use), ‘Our new CI will also feature macOS and Windows’ … getting there). Action: the v3 infrastructure uses all the best-of-breed tools the community has been developing in recent years. Tutorials are checked in CI via mdx, for example, so that they won’t bitrot with a new release of OCaml that changes outputs.
Items nearly achieved
- 42 people (6.2%) when answering “If one piece of the ecosystem could magically be made state-of-the-art, I would ask for” answered with a language website, 121 (18%) answered documentation for user libraries and 73 (10.8%) communication to outsiders.
- In question no 20, the preferred interaction methods differed for each group (beginner, intermediate, advanced, expert). In order to address this, we have integrated links to all the main interaction methods on “Around The Web”. “Around the Web” also serves as a common interaction method for all groups.
Works in progress
- When asked “What do you think are the main pain points that prevent OCaml adoption for new projects?” 310 (50.4%) said Too hard to hire and find developers and 181 (29.4%) said Too hard to learn – v3 incorporates an opportunities section and sets up a much more comprehensive environment for writing and expanding tutorials and documentation.
- When asked if OCaml best practises are understood roughly 40% said they were neutral, disagreed or strongly disagreed! Action: There’s been a phenomenal amount of work here including @cdaringe’s discuss post and @tmattio’s WIP workflows PR to ood.
- When asking whether or not people could easily find OCaml jobs, just shy of 50% responded with Neutral, Disagree, Strongly Disagree and I don’t know. Action: We have a
jobs
category on the discussion forum, which will be syndicated to the Opportunities page on the new website.
Long-term goals
- OFronds + OCaml Playground/Per package toplevel – making it easier and easier for beginners to get started in OCaml. We have a proof-of-concept of a toplevel built using JSOO, which we are aiming to integrate for every package that supports such a build.
Next Steps in 2021
Fall 2021
The v3 website is going into “public alpha” in August. This means that several pages aren’t yet complete at all, integration is continuing, and the design will be refined over the coming months.
The overall structure and sitemap is ready, and the technology choices have all been ratified and deployed.
The best way to give feedback would be to start a thread on discuss.ocaml.org. Bear in mind that the site is in an early stage, so be constructive and kind in your criticism (but also do not hold back!). If you feel that private feedback is more appropriate, then feel free to contact me directly on anil@recoil.org and I will forward your feedback to the team.
Winter 2021
As community feedback settles and the features are all implemented, the site will move into public beta. At this point, we will be working on refining the tail of bugs and preparing to swap websites.
Going live
When going live, the switch will happen as follows:
- the existing ocaml.org site will be snapshotted as static HTML and deployed on v2.ocaml.org with a header marking the site as “deprecated”
- we generate a sitemap of static HTML on v2.ocaml.org and, where the link doesn’t resolve on v3.ocaml.org, create a redirect to v2.ocaml.org. This will prevent dead links from existing content.
- v3.ocaml.org is aliased to ocaml.org via a DNS swap.
- opam.ocaml.org will continue to be operated indefinitely, since existing clients use that as the package mirror. The web content will gradually migrate to ocaml.org links in the course of 2022.
How can you contribute to the v3 effort?
There is an awful lot left to be done to get this site live, as always!
-
Frontend developer: The ReScript/NextJS-based frontend issue tracker has a maintained set of issues about what pages are remaining. There is also always general polish to be done as many of the pages are in their first design iteration. See the repository CONTRIBUTING.md file for more information on how to build the site.
-
OCaml developer: The ocaml/ood repository is where all the data that drives the site is stored. There are schemas there for videos, talks, events, workshops, papers, releases and so on. There’s a lot of data from the past 25 years to catch up on there, so please do feel free to create an issue on the tracker that you are working on something and start adding it. The OCaml proficiency level required here is fairly simple.
-
OCaml tooling hacker: If you are familiar with the OCaml Platform tools like opam and dune, then you may want to look at the dynamic v3.ocaml.org-server that drives the site and does the documentation rendering. The backend infrastructure is fairly complex, but all of the built artefacts are available over HTTP and the server retrieves them on startup, so you should be able to develop from the comfort of your own laptop.
-
Language translator: We have got the infrastructure in place to handle translated content. If you would be willing to help maintain a translation to a different language, we’d love to hear from you as the site content settles.
Team
The immediate team (as of Aug 2021) working on the v3 site consists of:
- Ashish Agarwal (Solvuu)
- Kanishka Azimi (Solvuu)
- Richard Davison (Solvuu)
- Patrick Ferris (OCaml Labs)
- Gemma Gordon (OCaml Labs)
- Isabella Leandersson (OCaml Labs)
- Anil Madhavapeddy (University of Cambridge)
- Thibault Mattio (Tarides)
- Tim McGilchrist (Tarides)
The documentation generation is courtesy of the odoc development team, with the site generation done by:
- Jon Ludlam (OCaml Labs)
- Jules Aguillon (Tarides)
- Lucas Pluvinage (Tarides)
- Thibault Mattio (Tarides)
We had three wonderful Outreachy interns help us this year:
- Diksha Gupta is working on automating and expanding v3.ocaml.org’s new and experimental peertube instance for hosting OCaml-related video content. This required adding functionality into ood to track the videos that have been published.
- Odinaka Joy is prototyping and building an online package service accessible through a graphQL endpoint in v3.ocaml.org-server. During the internship Joy has also prototyped js_of_ocaml web applications for consuming the data from the endpoint.
- Shreya Kumari Gupta has been a driving force in porting more content, pages and designs into v3 including adding all of the academic institutions that use OCaml into ood.
The site of course depends on a very large number of community published libraries in opam, as well as the Rescript compiler. There’s a lot of work left to do to improve the site and we warmly welcome more volunteers to assist us. If you’d like to join the ocaml.org website team, get in touch with Ashish Agarwal or Anil Madhavapeddy and we’d be pleased to help you onboard. You can also look at the various issue trackers for labelled issues on tasks that need to be completed.
Thank you to Xavier Leroy, Gabriel Scherer, David Allsopp, Thomas Gazagnaire, Frederic Bour, Florian Angeletti, Ashish Agarwal and many others for feedback on this post, and for Patrick Ferris and Bella Leandersson for helping to write it.