DkCoder 0.2 - Scripting in OCaml

Ecosystem
,
1

I’m happy to announce the second release of DkCoder, an OCaml scripting tool.

The first release was about install ease: a couple clicks and four (4) minutes later you and your Windows and macOS users can start scripting. All users, including glibc-based Linux desktop users, can also use their Unix shells or Windows PowerShell. OCaml does not need to be pre-installed. Just copy and paste two lines (you’ll see some in this post) and your script is running and your project is editable with OCaml LSP.

This second release is about technical ease. The three “big” ideas in this release are:

  • You don’t write build files. If that sounds like /bin/sh that is intentional.
  • Almost every OCaml file is a script you can run. If that sounds like how Python scripts are almost indistinguishable from Python modules, that is intentional.
  • Almost every OCaml file can be referenced with a fully-qualified name. If that sounds like Java packages that is intentional.

Here are some examples:

  1. (one of my own scripts) The incomplete but growing DkCoder documentation is written in a script: https://diskuv.com/dksdk/coder/2024-intro-scripting/. The documentation is a side-effect of running tests.

    In a Unix shell or in PowerShell, the following will a) run tests using tezt, b) collect outputs, c) generate HTML documentation, and then d) serve the doc page on a tiny_httpd webserver for a quick preview:

    git clone --branch V0_2 https://gitlab.com/diskuv/samples/dkcoder/DkHelloScript.git

./DkHelloScript/dk DkRun_V0_2.Run -- DkHelloScript_Std.Y33Article --serve

    The following will print mixed Markdown/HTML that I can render and publish with a static site generator to a website:

    ./DkHelloScript/dk DkRun_V0_2.Run -- DkHelloScript_Std.Y33Article --doc --doc-format markdown

  2. (someone else’s) The Bogue demo game Snoke written by @sanette was “ported” to DkCoder. The port did not change a single line of the original code. I did re-arrange the directory structure (recall that there is a Java-like package mechanism underneath DkCoder) and I did add an extra .ml file. Run:

    git clone --branch V0_2 https://gitlab.com/diskuv/samples/dkcoder/SanetteBogue.git

./SanetteBogue/dk DkRun_V0_2.Run -- SanetteBogue_Snoke.Snoke

The remaining items for DkCoder before a 1.x release: auto-downloading remote libraries (mostly done), meta/codegen tools (in progress), conditional compilation (in design), and a security policy (in design).

But right now DkCoder is at a reasonable enough point that I can now recommend using it for your own scripts. With the usual caveats that this is a 0.x release.

I’d like some feedback, especially on pain points and missing must-have features.

Tech Details (if interested)

Very simplistically, DkCoder is a high-level build system that transparently manages lower-level build systems (today that is Dune). I think (?) DkCoder is the first build system to use the codept OCaml dependency analyzer. Huge huge thanks to @octachron for that tool.

The rather boring driver pipeline is:

  1. Seed a “universe” of modules with the single .ml file the user wants to run from the ./dk CLI, or seed with all the .ml files if run through OCaml LSP.
  2. Let codept analyse any module references inside the current universe. Any missing modules are located and added to the universe. Rinse and repeat until there is a closed universe with no more missing module references.
  3. Generate and/or incrementally update the build files. Each .ml file is mapped to a single OCaml .cma library.
  4. Run the chosen build tool (ie. Dune) and execute the code.

What does that pipeline give us? Even in this early 0.2 release you get some unusual benefits:

  • Step 2: The missing modules can be created implicitly. The Snoke game has font, image and sound assets. By using Tr1Assets.LocalDir in the code DkCoder automatically creates a module that has all the assets (think ocaml-crunch). If a script does not need the assets, the codept analysis knows it doesn’t use Tr1Assets, and the assets won’t waste time getting built.
  • Step 3: The one-to-one .ml/.cma correspondence means DkCoder can apply a unique set of compiler flags to each .ml file. You get the Java-like package structure by opening a unique set of modules per .ml with -open flags (nit: I also used implicitly created directory modules to let you navigate the packages in your source code).
  • Step 4: You can take the generated dune-project and dune files, tweak them and run them outside of DkCoder. That means you are not locked into DkCoder! You can alternatively do what I did with Snoke: make your project compatible with both regular dune (/ocamlbuild/etc.) and DkCoder. Either way, you only need to deal with two issues that arise from DkCoder’s bytecode compilation and prebuilt C libraries: a) build C dependencies yourself, and b) tell Dune to switch from bytecode mode to native code mode. If you are a mildly experienced Linux/OCaml user who understands the terms “opam”, “pkg-config”, “depexts”, and “dune-configurator”, this is a low bar.

Script references:

Enjoy! --j

17 Likes
2

The 0.3 version is now available. It has a publicly accessible GitHub - diskuv/dkcoder so issues can be filed, and cohttp 6.0.0~beta2 is now bundled.

Most important, 0.3 was sufficient to build the real production service Diskuv / Samples / DevOps / DkSubscribeWebhook · GitLab. It has a Dockerfile and Docker Compose for easy deployment to production, and the Docker container is based on Google’s GitHub - GoogleContainerTools/distroless: 🥑 Language focused docker images, minus the operating system. for a “small” size (well, 100MB is not small but it is not big either). The only executable in the container is ocamlrunx (no /bin/sh, etc.). In an ideal world where I had more time the service would be embedded inside MirageOS instead.

3 Likes