v3.OCaml.org: A roadmap for OCaml's online presence

The current (“v2”) ocaml.org website was written a decade ago (presentation) and – while it has served us well – its age is showing as our community has grown. To put things in context, the opam package manager didn’t actually exist when we built the current iteration of the website.

TL;DR: preview the new package search at https://v3.ocaml.org/packages

In November 2020, the current maintainers put our heads together to rebuild the website to modern standards (“v3”). When it came to determining the goals of the website refresh, we were driven by the need to integrate the technological developments of the last decade – for example, the adoption of opam as the package management system (evidenced by close to 20,000 packages in the opam repository), and the emergence of common tools and workflows that benefit both individual developers as well as larger-scale industrial codebases.

In October 2020, the core OCaml team also issued a user survey to solicit information from our user community about priorities for the OCaml ecosystem. Here are the original results, original discuss thread, concrete suggestions and some analysis. Where appropriate, we reference the user survey results in the information below to try to ensure our decisions are as inclusive as possible.

v3.ocaml.org features

The v3 version of the website is a rewrite from scratch. This is mostly because the web technologies used in the 2011 version are outdated, and it is difficult to forward port them to a standard that meets modern accessibility requirements. Our own OCaml tooling ecosystem has also moved on tremendously in the last decade, so it was time for a fresh start. The new site features:

Integrated documentation and package management

When v2 was developed, the opam package manager didn’t exist yet and so we ended up with a fragmented ecosystem (opam.ocaml.org is separate and distinct from ocaml.org, and other package managers like GODI also had separate websites). Since then, opam metadata has become the standard in our ecosystem with over 20000 packages published.

The v3 site combines the existing package management subdomain with a new central documentation source (formerly codenamed docs.ocaml.org) for all versions of all opam packages. This content is held directly within the new ocaml.org site and not a set of subdomains, reflecting a clear community direction. Our goal is for the live website to contain published package API docs within 30 minutes of the package being merged.

Separation of data editing from HTML/CSS generation

The v2 site combined (e.g.) fetching external newsfeeds with the HTML generation, which made it expensive to scale with new content as every build would take longer and longer.

The v3 site keeps a structured store of data as OCaml values, which is then used to generate the frontend website. We use ReScript/OCaml to generate the site content, and Ocurrent to automate the data pipelines. The data source repository is known as ood (OCaml.Org Data). Finally, the static HTML is combined with the dynamic API documentation generation and served via a Dream-based webserver.

For users wishing to submit content fixes to the website, they need to do nothing more than clone ocaml/ood and submit the fix there, without a full website build.

Publishing original content

The v2 site syndicated from external sources, which worked well when the web (circa 2011) was centred around RSS/Atom feeds and blogging engines. The modern web (circa 2021) has moved towards social networking chatter, which greatly reduces the value of syndicating content. We also now have multiple discussion forums such as discuss.ocaml.org where original content is frequently posted.

There will there be original content on the v3 site, so that we can publish editorial content from around the community directly on ocaml.org. Original content can include interviews, featured news stories, informative forum posts, online talks and other media from past conferences, along with many other types of content.

To facilitate this without tying ourselves to a third-party website, we have deployed an instance of the open-source Peertube video hosting application, over at watch.ocaml.org. This provides an open API (free of adverts and third-party cookies) to view video material about OCaml through its rich history. The OCaml Workshop 2021 videos are being published on watch.ocaml.org as their primary source: OCaml Workshop 2021 - Watch OCaml

Accessible, sustainable and respectful of privacy

The v2 site grew organically over a decade, and so a multiplicity of ways to navigate it emerged. The v3 site design takes into account modern web design principles, restructuring the old content in accordance with methods that present it more compellingly. It is a total redesign that modernises the look and feel of the site to work well on mobile and other responsive targets, as well as being easier to navigate and more accessible.

In terms of technology, the frontend switches to Tailwind CSS and uses React/Rescript/NextJS to generate most of the site HTML. The backend data sources all remain in pure OCaml code, meaning that we retain the ability to migrate to new frontend technologies without a full rewrite in the future.

The v3 website has a simple privacy policy: there are no third-party cookies or tracking systems in place beyond server-side log analysis. This allows us to be compliant with GDPR rules, but also be accessible to the widest global audience.

In reflection of our strong committment to sustainability, we are also working on a “carbon policy” that will track and publish our energy usage across the full ocaml.org infrastructure (including the build cluster). This adds up to around ~1000 CPU cores of different architectures hosted around the world in various data centres, and we are aiming to be carbon neutral via the use of renewable energy and similar initiatives.

A quick tour of some nice bits of v3.ocaml.org

(This is accurate as of 4th Aug 2021)

There are still lots of unfinished pages (most notably the front page is totally placeholder until we finish more of the leaf pages), but this should hopefully give you a good view on where the site is going!

2020 Survey Actions

The v3 website was partly driven by the feedback from our community as well as the core team’s discretion. We will now go through the feedback items.

Individual quotes from different answers

Pain point when Learning the OCaml Language

  • “Lack of search engine friendly content. Other mainstream languages have plenty of short tutorials available on the web”: The few OCaml does have are not discoverable easily": On the new site we’re making the core tutorials easier to find. Community members are also working on new tutorial systems for beginners, and the website can link to these as they are published.
  • “Up to date docs and learning materials and books”: The Language page will help users find this information in the future.
  • “Poor documentation” and “Most libraries have very sparse documentation”: There are many, many comments like this one, and while we cannot solve this directly, having integrated and up-to-date API docs directly in OCaml.org should incentivise people to do a better job of adding ocamldoc comments to their code.

I am satisfied with OCaml’s package repositories, what would you change

From the discuss thread

  • Yawaramin wrote: “Or Outreachy internships or other funded dev work at less than senior level? There are libraries out there in other languages that would be tremendously beneficial to port over to OCaml…”. Action: the OCaml Software Foundation and OCaml Labs have now run multiple Outreachy programs, with the most recent working on v3.ocaml.org itself. We need community help to add more mentors to scale this program – please volunteer!

  • Anil’s comment touches on all aspects (ocaml.org, mdx (the tutorials now use), ‘Our new CI will also feature macOS and Windows’ … getting there). Action: the v3 infrastructure uses all the best-of-breed tools the community has been developing in recent years. Tutorials are checked in CI via mdx, for example, so that they won’t bitrot with a new release of OCaml that changes outputs.

Items nearly achieved

  • 42 people (6.2%) when answering “If one piece of the ecosystem could magically be made state-of-the-art, I would ask for” answered with a language website, 121 (18%) answered documentation for user libraries and 73 (10.8%) communication to outsiders.
  • In question no 20, the preferred interaction methods differed for each group (beginner, intermediate, advanced, expert). In order to address this, we have integrated links to all the main interaction methods on “Around The Web”. “Around the Web” also serves as a common interaction method for all groups.

Works in progress

  • When asked “What do you think are the main pain points that prevent OCaml adoption for new projects?” 310 (50.4%) said Too hard to hire and find developers and 181 (29.4%) said Too hard to learn – v3 incorporates an opportunities section and sets up a much more comprehensive environment for writing and expanding tutorials and documentation.
  • When asked if OCaml best practises are understood roughly 40% said they were neutral, disagreed or strongly disagreed! Action: There’s been a phenomenal amount of work here including @cdaringe’s discuss post and @tmattio’s WIP workflows PR to ood.
  • When asking whether or not people could easily find OCaml jobs, just shy of 50% responded with Neutral, Disagree, Strongly Disagree and I don’t know. Action: We have a jobs category on the discussion forum, which will be syndicated to the Opportunities page on the new website.

Long-term goals

  • OFronds + OCaml Playground/Per package toplevel – making it easier and easier for beginners to get started in OCaml. We have a proof-of-concept of a toplevel built using JSOO, which we are aiming to integrate for every package that supports such a build.

Next Steps in 2021

Fall 2021

The v3 website is going into “public alpha” in August. This means that several pages aren’t yet complete at all, integration is continuing, and the design will be refined over the coming months.
The overall structure and sitemap is ready, and the technology choices have all been ratified and deployed.

The best way to give feedback would be to start a thread on discuss.ocaml.org. Bear in mind that the site is in an early stage, so be constructive and kind in your criticism (but also do not hold back!). If you feel that private feedback is more appropriate, then feel free to contact me directly on anil@recoil.org and I will forward your feedback to the team.

Winter 2021

As community feedback settles and the features are all implemented, the site will move into public beta. At this point, we will be working on refining the tail of bugs and preparing to swap websites.

Going live

When going live, the switch will happen as follows:

  • the existing ocaml.org site will be snapshotted as static HTML and deployed on v2.ocaml.org with a header marking the site as “deprecated”
  • we generate a sitemap of static HTML on v2.ocaml.org and, where the link doesn’t resolve on v3.ocaml.org, create a redirect to v2.ocaml.org. This will prevent dead links from existing content.
  • v3.ocaml.org is aliased to ocaml.org via a DNS swap.
  • opam.ocaml.org will continue to be operated indefinitely, since existing clients use that as the package mirror. The web content will gradually migrate to ocaml.org links in the course of 2022.

How can you contribute to the v3 effort?

There is an awful lot left to be done to get this site live, as always!

  • Frontend developer: The ReScript/NextJS-based frontend issue tracker has a maintained set of issues about what pages are remaining. There is also always general polish to be done as many of the pages are in their first design iteration. See the repository CONTRIBUTING.md file for more information on how to build the site.

  • OCaml developer: The ocaml/ood repository is where all the data that drives the site is stored. There are schemas there for videos, talks, events, workshops, papers, releases and so on. There’s a lot of data from the past 25 years to catch up on there, so please do feel free to create an issue on the tracker that you are working on something and start adding it. The OCaml proficiency level required here is fairly simple.

  • OCaml tooling hacker: If you are familiar with the OCaml Platform tools like opam and dune, then you may want to look at the dynamic v3.ocaml.org-server that drives the site and does the documentation rendering. The backend infrastructure is fairly complex, but all of the built artefacts are available over HTTP and the server retrieves them on startup, so you should be able to develop from the comfort of your own laptop.

  • Language translator: We have got the infrastructure in place to handle translated content. If you would be willing to help maintain a translation to a different language, we’d love to hear from you as the site content settles.

Team

The immediate team (as of Aug 2021) working on the v3 site consists of:

  • Ashish Agarwal (Solvuu)
  • Kanishka Azimi (Solvuu)
  • Richard Davison (Solvuu)
  • Patrick Ferris (OCaml Labs)
  • Gemma Gordon (OCaml Labs)
  • Isabella Leandersson (OCaml Labs)
  • Anil Madhavapeddy (University of Cambridge)
  • Thibault Mattio (Tarides)
  • Tim McGilchrist (Tarides)

The documentation generation is courtesy of the odoc development team, with the site generation done by:

  • Jon Ludlam (OCaml Labs)
  • Jules Aguillon (Tarides)
  • Lucas Pluvinage (Tarides)
  • Thibault Mattio (Tarides)

We had three wonderful Outreachy interns help us this year:

  • Diksha Gupta is working on automating and expanding v3.ocaml.org’s new and experimental peertube instance for hosting OCaml-related video content. This required adding functionality into ood to track the videos that have been published.
  • Odinaka Joy is prototyping and building an online package service accessible through a graphQL endpoint in v3.ocaml.org-server. During the internship Joy has also prototyped js_of_ocaml web applications for consuming the data from the endpoint.
  • Shreya Kumari Gupta has been a driving force in porting more content, pages and designs into v3 including adding all of the academic institutions that use OCaml into ood.

The site of course depends on a very large number of community published libraries in opam, as well as the Rescript compiler. There’s a lot of work left to do to improve the site and we warmly welcome more volunteers to assist us. If you’d like to join the ocaml.org website team, get in touch with Ashish Agarwal or Anil Madhavapeddy and we’d be pleased to help you onboard. You can also look at the various issue trackers for labelled issues on tasks that need to be completed.

Thank you to Xavier Leroy, Gabriel Scherer, David Allsopp, Thomas Gazagnaire, Frederic Bour, Florian Angeletti, Ashish Agarwal and many others for feedback on this post, and for Patrick Ferris and Bella Leandersson for helping to write it.

69 Likes

Thanks for the update Anil! The new website (and package search) look awesome!

Cheers,
Nicolas

5 Likes

Looking good! Definitely feels more modern than the current version.
I particularly like the package search as a central place to also access the documentation. That’s exactly the kind of stuff I imagine people want to see when they want OCaml to be more “Rust-like” in that regard.

Having said that, the colors on the packages landing page feel very aggressive to me. Might be my setup here, but I would like to have a slightly less harsh contrast.

Also, there is a bit of an overlap in content with https://ocamlverse.github.io/ for some things (eg best practices, community) but the (to me) most valuable feature is missing: The ecosystems overview, where I can find packages sorted thematically. Could such a section also have a place in the packages subpage somewhere?
Alternatively, maybe opam can allow to “tag” packages in the future so one could see all packages for graphics, databases etc.

1 Like

I’m very hopeful that everyone finds the packages/documentation part of the site useful! As part of that I’d love to get feedback in the form of issues (PRs would also be good!)

There are quite a few moving parts involved in the documentation generation, so it’s not always obvious where to file issues amongst at least 5 different repositories. In the interests of not putting people off filing issues, I’d like to suggest that all issues related to any page under the /packages or /p path get filed against v3.ocaml.org-server. We’ll track them there and then we can figure out where the problem actually needs fixing. There are already quite a few known issues there, so please take a look before filing a new one.

6 Likes

The new opam package search integrated with the doc is very nice.

2 Likes

Wow, I didn’t realize how much I wanted the integrated docs until I tried it. This is great! The rest of the site looks really good too. Thanks everyone working on this!

3 Likes

Looks great! With some nitpicks since this is a preview :slight_smile: Where should I open possible bugreports for the overview page of the package and for the documentation page? (In the latter case it is not clear if it is related to the website’s choice of options for odoc or something, or some changed behaviour of odoc, I might have to investigate.)

1 Like

Issues · ocaml/v3.ocaml.org-server · GitHub - as suggested above. We’re using bleeding-edge odoc which has been almost entirely rewritten since 1.5, so I wouldn’t be surprised if there are a few rough edges :slight_smile:

1 Like

Do we have access to all of the previous years’ workshops to add to watch.ocaml.org?
I can see pieces of 2015, 2017, 2020 and this year. @avsm

Is it possible to add the ML Workshop as well?

1 Like

This is all so lovely! The principles around privacy and sustainable energy sources are genuinely wonderful to see. I think we should be sure to advertise that when we spread word after launch. Everything looks amazing. :heart_eyes::camel:

5 Likes

the landing page looks much nicer and modern! really look forward to this project!

as someone who hasn’t used, but has been considering jsoo vs rescript for the FE of a project, I’m curious why ReScript was chosen over JSOO for the website FE?

1 Like

It’s a bit unclear what you meant in this paragraph. Does that mean that you plan to kill the ocaml planet ? I would find it a little bit sad.

One of the reason why you may feel it doesn’t work well may be that it has been constantly broken in the current version of the site…

Personally I have no interest in losing my time on “social” networks to get a feel of what’s happening around – and I don’t think a forum is any kind of replacement for it. It seems the big tech players really managed to convince everyone that syndication was broken after they decided to kill it for the sake of their own business interests.

FWIW I follow the haskell planet which does not break every two months and while I’m far from reading everything I’m quite happy with the articles I’m getting there; the amount of articles is reasonable, the signal to noise ratio is good and some of the content quite interesting.

5 Likes

As a part of the team that started thinking about v3.ocaml.org back in Autumn 2020, it is incredibly rewarding and encouraging to see such a positive response and constructive feedback to the new, work-in-progress version of the site. A lot of deliberation and effort has gone into what we currently have and now that is publicly available, hopefully will gain community support and momentum to improve it further. I thought I’d answer some of the questions people have here.

Indeed. Much like the OCaml compiler itself, the “main” ocaml.org website has a commitment to backwards compatibility as well as new content and features. What I mean by this is we are trying to incorporate everything in GitHub - ocaml/ocaml.org: The official OCaml website. as well as creating new content/features. This inevitably makes things a little slower, but best practices are in the works.

Afaik opam already has tags they are just not used that widely or in a standardised way for indicating things like graphics, databases etc. (in fact the server reads them). If some consensus was reached on this it should be a fairly straightforward and useful feature to add. In the meantime we could certainly add curated lists of packages to ood :)).

When making the technology choices, the main reasoning was “use the right tool for the job”. In my opinion (as an avid jsoo user) for what we wanted jsoo makes little sense. The Rescript frontend is essentially bindings to Nextjs (plus any other libraries from the JS universe that are needed). Rescript has good JSX support, integrates well with the npm world, has (arguably…) better FFI support with JS etc. When we need jsoo (e.g. the experimental plan to have integrated toplevels for packages) then we’ll reach for it.

That’s not to say you can’t use jsoo for front-end work, for example there are bindings to reactjs and FRP libraries like brr + note. Another very flexible way for building static sites with a sprinkling of JS when needed is using OCaml for HTML generation and alpine.js for bits of javascript (e.g. the preview app in ood).

Importantly, now that the data lives very much separately we’re also not tied to any particular front-end. In fact this allows us to share all of the data in ocaml.org (I think there’s more than people realise, that was certainly the case for me!) for example I wrote an in-complete “terminal rendering” of the data. Ood could equally be reused perhaps directly in vscode!

8 Likes

Enjoying the new design direction.

Big win here too is the use of modern and community-built tools like Dream and Rescript. It sets a great precedence of what and how a modern system can look. Makes OCaml feel very possible for those interested in adopting this stack.

7 Likes

This looks awesome and is definitely going to push the community and ecosystem forward! I’m so happy to see centralized documentation as a thing that’s doable now.

2 Likes

I also hope that OCaml organization will participate into the Google Summer of Code 2022, like other languages, e.g. Haskell or Julia, do. It will help to find more potential young contributors, moreover, will attract more attention to the OCaml ecosystem and the current roadmap. Apart from that, it will require dedicated GSoC pages about the program, potential ideas for the tasks. Furthermore, some organizations have a good policy of having some “good first issues” that should be solved by the students to apply for the GSoC. This might need a separate page as well.

Why I am writing this? It will need to be on the site as well, assuming OCaml will apply for the GSoC 2022.

See also Google Summer of Code and GSoC project on Coccinelle

2 Likes

The hardest question to answer is: should the site be a classic html site with very minimal interactivity, or should it be a richly interactive site? At first glance, one doesn’t really have uses for the interactivity, but then interesting possibilities start to creep in - an interactive map, interactive highlighting of table of contents position, a carousel to browse books. We’ll see if more desire for interactive features emerge, possibly when the manual starts to be integrated into the site.

If interactivity isn’t important, that leads to a wide path of technical options. If interactivity is important, the path gets narrower, but there are still a few interesting options.

1 Like

Not having spent so much time as part of the OCaml community, while reading people’s response regarding the topic in question, I can fairly judge that this is a better progress for OCaml at large. From own perspective, I’m confident to say that the new site and the incorporated features is so awesome. Thanks to the community’s support and efforts.

4 Likes

Many thanks for all the constructive comments and suggestions so far, and also for those who have gotten in touch to contribute. Please do keep them coming (either on this thread or on the various issue trackers that @jonludlam and @patricoferris have pointed to). I’ll answer some earlier questions here:

The styling of the /packages sub-URL does indeed differ from the main design, but this is simply due to a temporary technical detail. The majority of the site uses React/NextJS to generate the frontend, and this uses the now-trendy medium-contrast colours and also features like fast-page-switching that NextJS offers. However, the documentation portion generated around 2.7 million individual pages when run across the full opam repository, and so we restored to dynamic generation of the content for that. What’s going to happen next is a rationalisation of the code across the ReScript and OCaml frontends so that there will be no observable difference in the colour schemes across the full site.

Regarding creating a categorised list of recommendations, that is absolutely in scope for the v3 iteration of the site. However, this metadata should ideally live in the opam-repository (for example, using tags as you suggest, which opam already supports). If anyone would like to have a go at this, I’d encourage PRs to the opam-repository to add the relevant tag metadata for a codex. Meanwhile, @lambda_foo @tmattio and @patricoferris are working on the core OCaml Platform workflow information for the guides section of the website which will cover opam, merlin, lsp-server, dune and so on.

Absolutely. The watch.ocaml.org has held up nicely after the OCaml Workshop, so I think it’s in good shape to populate with more videos. This needs a volunteer to help us upload the past nine years of videos from YouTube to watch.ocaml.org. If anyone wants to have a go, please message me and I’ll create you an account.

I’m not sure why you think the current ocaml.org new feed has been broken – it’s been working fairly reliably for the past decade. The only real problem came up a few times when a feed’s domain expired and got taken over by domain squatters, at which point we got spam into the main page of ocaml.org.

What I meant with that part of the announcement is that the syndication feed should not be mistaken with original news on the website. Right now it’s difficult to distinguish official announcements (such as compiler or opam releases) as they are a little scattered (e.g. on opam.ocaml.org). The plan is to combine the platform-blog with the new website directly. I’ve also been considering just having a special tag on this forum so that nice announcement posts could also be syndicated to the website easily (for example, @gasche’s compiler newsletters).

My general desire is to grow the planet feed and syndication system, but to clearly demarcate them as not being published by ocaml.org and to manage them via more modern decentralised techniques that feature spam, moderation and archival. PeerTube is a good example of this for videos that is working well, and I’d welcome suggestions for Atom/RSS (there must be something in this space, ideally ActivityPub-based).

Depending on how the experiments go, it’s very likely that we’ll have a Matrix homeserver for ocaml.org where CI bots can report status information (see this prototype PR) for ocaml-ci that will also apply to opam-repository. The goal here is to for ocaml.org to publish its data using an open protocol, which can then be syndicated into whatever technologies are in vogue (e.g. Discord, Slack, Teams, …).

So if you spot some decentralised syndication system that you think might be interesting for OCaml, please do let me know. Even better, if you’d like to develop one to tailor it to our needs, let me know even sooner :wink:

8 Likes

Note that there’s already an OCaml channel (hosted on matrix.org but anyone can join). :slight_smile:

1 Like