# DataKit dev reports

**URL:** <https://discuss.ocaml.org/t/datakit-dev-reports/224>\
**Category:** Community\
**Tags:** irmin, docker, mirageos\
**Created:** [May 18, 2017, 11:13pm UTC](https://discuss.ocaml.org/t/datakit-dev-reports/224 "2017-05-18T23:13:20Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![samoht](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/samoht/32/81_2.png) [@samoht](https://discuss.ocaml.org/u/samoht)\
**Post date:** [May 18, 2017, 11:13pm UTC](https://discuss.ocaml.org/t/datakit-dev-reports/224/1 "2017-05-18T23:13:20Z")

</div>

## DataKit: Orchestrate applications using a Git-like dataflow

_DataKit_ is a tool to orchestrate applications using a Git-like dataflow. It revisits the UNIX pipeline concept, with a modern twist: streams of tree-structured data instead of raw text. DataKit allows you to define complex build pipelines over version-controlled data.

DataKit is currently used as the coordination layer for [HyperKit](http://github.com/docker/hyperkit), the hypervisor component of [Docker for Mac and Windows](https://blog.docker.com/2016/03/docker-for-mac-windows-beta/), and for the [DataKitCI](https://github.com/moby/datakit/tree/master/ci) continuous integration system.

Development repository is [https://raw.githubusercontent.com/moby/datakit](https://raw.githubusercontent.com/moby/datakit)  
Community Slack: #datakit on [https://dockercommunity.slackarchive.io](https://dockercommunity.slackarchive.io)

# Weekly DataKit dev report for 2017-04-17 to 2017-04-23 (week 16)

This report covers weekly developments in the [moby/datakit](https://github.com/moby/datakit), [mirage/irmin](https://github.com/mirage/irmin),  
[mirage/ocaml-git](https://github.com/mirage/ocaml-git), and [mirage/ocaml-9p](https://github.com/mirage/ocaml-9p) repositories.

**TL;DR:** The effort this week has gone into preparing for a release of the development  
trees next week, and for the renaming of the project repositories to the  
[Moby Project](http://mobyproject.org) (see week17 for more on this topic).  
The use of multi-stage builds vastly decreased the size of the published containers to  
just 21MB, which makes deploying DataKit much more efficient!

## PRs merged

**Build and Packaging:**

- The default Dockerfile now uses [multi-stage builds](https://blog.mikesir87.io/2017/03/introducing-docker-multi-stage-builds/) to vastly shrink the size of the DataKit containers from 1.4-1.7GB to 21-22MB! ([moby/datakit#522](https://github.com/moby/datakit/pull/522) [@talex5](https://github.com/talex5)).
- Since the Docker Hub doesn’t support multi-stage container builds yet, the Dockerfile  
used for autobuilds was inlined to ensure that the published images stay in sync. ([moby/datakit#521](https://github.com/moby/datakit/pull/521) [@talex5](https://github.com/talex5)).
- The root `opam` file is now called `datakit.opam` to make it fit with the other sub-packages ([moby/datakit#525](https://github.com/moby/datakit/pull/525) [@talex5](https://github.com/talex5)).
- The repositories and self-ci example were update for new repository locations ([moby/datakit#528](https://github.com/moby/datakit/pull/528) and [moby/datakit#520](https://github.com/moby/datakit/pull/520) [@talex5](https://github.com/talex5)).
- Windows CI of DataKit was fixed up to account for recent changes ([moby/datakit#527](https://github.com/moby/datakit/pull/527) [moby/datakit#122](https://github.com/moby/datakit/pull/122) [@talex5](https://github.com/talex5) [@samoht](https://github.com/samoht)).

**Functionality improvements:**

- The GitHub bridge can now read its private key using Docker Swarm secret management, which avoids the need to spread the secret authentication token any further than necessary ([moby/datakit#519](https://github.com/moby/datakit/pull/519) [@talex5](https://github.com/talex5) [@avsm](https://github.com/avsm)).
- The DataKit server no longer exposes an HTTP server. It was only used for debugging before, and Irmin 1.0 no longer supports it ([moby/datakit#524](https://github.com/moby/datakit/pull/524) [@samoht](https://github.com/samoht) [@talex5](https://github.com/talex5)).
- As DataKit is used more in production, [@talex5](https://github.com/talex5) has been steadily improving error handling to ensure that callers can handle failures more gracefully, either via retrying or logging exceptions ([moby/datakit#526](https://github.com/moby/datakit/pull/526) [@talex5](https://github.com/talex5)).

### PRs with activity

- [@djs55](https://github.com/djs55) uses DataKit in Docker for Mac for configuration management, and so he proposed a scheme to make branch handling more robust for real-world use of this feature. His PR covers the case of software upgrades and user-supplied overrides for particular configuration keys ([moby/datakit#523](https://github.com/moby/datakit/pull/523)).
- [@avsm](https://github.com/avsm) requested that DataKit CI be able to monitor a complete GitHub organisation and add monitoring hooks by watching the rigth events. [@samoht](https://github.com/samoht) proposed a fix to this in [moby/datakit#419](https://github.com/moby/datakit/pull/419), but it has become outdated due to upstream changes, so he is rebasing it.
- [@samoht](https://github.com/samoht) is also working towards making DataKit log less verbose commit messages, to reduce the size of the state repository ([moby/datakit#476](https://github.com/moby/datakit/pull/476) [@samoht](https://github.com/samoht)).

## External Links or Blogs

- “[Optional Dependencies considered harmful](http://rgrinberg.com/posts/optional-dependencies-considered-harmful/)” by Rudi Grinberg explains very well why DataKit and Irmin now have explicit OPAM packages rather than `depopts` for optional functionality.
- “[Introducing Docker multistage builds](https://blog.mikesir87.io/2017/03/introducing-docker-multi-stage-builds)” explains how to achieve the extreme container shrinkage in your own projects.

Other reports in this series can be browsed directly in the repository at [moby/datakit:/reports](https://github.com/moby/datakit/tree/master/reports/).

---

<div class="post-metadata">

**Author:** ![samoht](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/samoht/32/81_2.png) [@samoht](https://discuss.ocaml.org/u/samoht)\
**Post date:** [May 18, 2017, 11:13pm UTC](https://discuss.ocaml.org/t/datakit-dev-reports/224/2 "2017-05-18T23:13:45Z")

</div>

# Weekly DataKit dev report for 2017-04-24 to 2017-04-30 (week 17)

This report covers weekly developments in the [moby/datakit](https://github.com/moby/datakit), [mirage/irmin](https://github.com/mirage/irmin), [mirage/ocaml-git](https://github.com/mirage/ocaml-git) and [mirage/ocaml-9p](https://github.com/mirage/ocaml-9p) repositories.

**TL;DR:** The project now has a new home in the [Moby Project](http://mobyproject.org) and the main DataKit repository was renamed. There were three releases this week, of [moby/datakit:0.10.0](https://github.com/moby/datakit/releases/tag/0.10.0), [mirage/irmin:1.1.0](https://github.com/mirage/irmin/releases/tag/1.1.0) and [mirage/ocaml-9p:v0.10.0](https://github.com/mirage/ocaml-9p/releases/tag/v0.10.0), with mega build speedups and improvements to the DataKit Continuous Integration engine.

## PRs merged

**Build and Packaging:**

- DataKit was ported to use the new [Jbuilder](https://github.com/janestreet/jbuilder) build system for OCaml, resulting in a 4x speedup of compilation time and a more consistent linking strategy for the various DataKit subcomponents. See [moby/datakit#532](https://github.com/moby/datakit/pull/532) [moby/datakit#539](https://github.com/moby/datakit/pull/539) and [mirage/ocaml-9p#119](https://github.com/mirage/ocaml-9p/pull/119) ([@diml](https://github.com/diml) [@talex5](https://github.com/talex5) [@dbuenzli](https://github.com/dbuenzli) [@samoht](https://github.com/samoht) [@djs55](https://github.com/djs55)).
- The DataKit trees were also refreshed to support the latest versions of Irmin, [Mirage3](https://mirage.io/blog/announcing-mirage-30-release) and Cmdliner ([moby/datakit#529](https://github.com/moby/datakit/pull/529) [moby/datakit#531](https://github.com/moby/datakit/pull/531) [@talex5](https://github.com/talex5) [@samoht](https://github.com/samoht)).
- The 9P implementation now also supports the latest Lwt 3.0.0 interfaces ([mirage/ocaml-9p#117](https://github.com/mirage/ocaml-9p/pull/117) [@djs55](https://github.com/djs55)).
- OPAM constraints for the `irmin-unix` and `irmin-git` packages were also fixed ([mirage/irmin#439](https://github.com/mirage/irmin/pull/439) [@samoht](https://github.com/samoht)).

**Functionality improvements:**

- DataKit now allows the use of `/` in branch names ([moby/datakit#533](https://github.com/moby/datakit/pull/533) [@samoht](https://github.com/samoht)).
- The GitHub bridge was simplified by removing the VFS layer, which was intended to be useful for debugging but rarely used in practise ([moby/datakit#535](https://github.com/moby/datakit/pull/535) [@samoht](https://github.com/samoht)).
- The Continuous Integration subsystem was improved to make the required GitHub scopes configurable to make it easier to deploy on public GitHub repositories with less privilege ([moby/datakit#534](https://github.com/moby/datakit/pull/534) [@talex5](https://github.com/talex5) [@avsm](https://github.com/avsm)) and to accept SSH keys automatically when autocloning state repositories ([moby/datakit#536](https://github.com/moby/datakit/pull/536) [@talex5](https://github.com/talex5)).
- A regression in Irmin 1.0 in the `Irmin.Tree.diff` function, where nested diffs were reported with the wrong path was fixed ([mirage/irmin#438](https://github.com/mirage/irmin/pull/438) [@samoht](https://github.com/samoht)).
- Irmin can now specify branches in URLs for the `fetch` function ([mirage/irmin#432](https://github.com/mirage/irmin/pull/432) [@samoht](https://github.com/samoht)).

Thanks to [@samoht](https://github.com/samoht) [@avsm](https://github.com/avsm) and [@djs55](https://github.com/djs55) for handling all the release activity this week in [moby/datakit#538](https://github.com/moby/datakit/pull/538) [moby/datakit#540](https://github.com/moby/datakit/pull/540), [moby/datakit#541](https://github.com/moby/datakit/pull/541), [moby/datakit#542](https://github.com/moby/datakit/pull/542) and [mirage/ocaml-9p#120](https://github.com/mirage/ocaml-9p/pull/120).

### PRs closed this week without merge

- [@kayceesrk](https://github.com/kayceesrk) and [@samoht](https://github.com/samoht) have been discussing the semantics of merging `option` types in [mirage/irmin#421](https://github.com/mirage/irmin/issues/421), and concluded that Irmin 1.0.1 has the right semantics. [mirage/irmin#422](https://github.com/mirage/irmin/pull/422) was closed as a result.

- [@yomimono](https://github.com/yomimono) and [@samoht](https://github.com/samoht) did a lot of work in the runup to the MirageOS 3.0 release a few months ago to prepare DataKit for support, and all of that functionality is now present in DataKit 0.10, so [moby/datakit#433](https://github.com/moby/datakit/pull/433) and [moby/datakit#459](https://github.com/moby/datakit/pull/459) were closed.

- While building the [mirage-ci](https://github.com/avsm/mirage-ci), [@avsm](https://github.com/avsm) had reported that invalid job names from the CI resulted in inscrutable errors in the web interface. After some discussion, [@talex5](https://github.com/talex5) decided that [moby/datakit#492](https://github.com/moby/datakit/pull/492) wasn’t worth the complexity, and that it should be addressed elsewhere.

- [@samoht](https://github.com/samoht) had put together a design for a `dkt` CLI tool in [moby/datakit#146](https://github.com/moby/datakit/pull/146). While there were many good ideas in the design, it has never been implemented fully, so the PR is closed. Get in touch with [@samoht](https://github.com/samoht) if you’d like to build it!

## Ongoing activity

- The Golang DataKit bindings need to handle larger values better, so [@simonferquel](https://github.com/simonferquel) and [@samoht](https://github.com/samoht) are discussing merging [moby/datakit#292](https://github.com/moby/datakit/pull/292).
- The C# bindings to DataKit in [moby/datakit#334](https://github.com/moby/datakit/pull/334) are also awaiting review from [@dgageot](https://github.com/dgageot) and [@simonferquel](https://github.com/simonferquel). If you are interested in other language bindings, please create an issue.
- Irmin 1.1.0 is being improved by [@samoht](https://github.com/samoht) and [@talex5](https://github.com/talex5) to make it possible to distinguish between empty trees and nonexistent sub-trees in [mirage/irmin#431](https://github.com/mirage/irmin/pull/431) and [mirage/irmin#433](https://github.com/mirage/irmin/pull/433). It will also add some new features such as `Irmin.Merge.idempotent` ([mirage/irmin#435](https://github.com/mirage/irmin/pull/435)) and an `S.repo` type exposed ([mirage/irmin#436](https://github.com/mirage/irmin/pull/436)).
- [@talex5](https://github.com/talex5) has opened a WIP PR in [mirage/ocaml-9p#118](https://github.com/mirage/ocaml-9p/pull/118) to upstream some of the 9P patches used by DataKit in production, such as logging exceptions and keep-alive pings.
- The documentation is being improved by fixing the build instructions in [moby/datakit#537](https://github.com/moby/datakit/issues/537) ([@samoht](https://github.com/samoht) [@talex5](https://github.com/talex5)). More patches to improve the instructions are welcome!

## External Links or Blogs

- [Introducing the Moby Project](https://blog.docker.com/2017/04/introducing-the-moby-project/) to explain DataKit’s new home.

Other reports in this series can be browsed directly in the repository at [moby/datakit:reports](https://github.com/moby/datakit/tree/master/reports/).

---

<div class="post-metadata">

**Author:** ![samoht](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/samoht/32/81_2.png) [@samoht](https://discuss.ocaml.org/u/samoht)\
**Post date:** [May 18, 2017, 11:14pm UTC](https://discuss.ocaml.org/t/datakit-dev-reports/224/3 "2017-05-18T23:14:15Z")

</div>

# Weekly DataKit dev report for 2017-05-01 to 2017-05-07 (week 18)

This report covers weekly developments in the [moby/datakit](https://github.com/moby/datakit), [mirage/irmin](https://github.com/mirage/irmin), [mirage/ocaml-git](https://github.com/mirage/ocaml-git) and [mirage/ocaml-9p](https://github.com/mirage/ocaml-9p) repositories.

**TL;DR:** It’s been a quiet week after the previous week’s slew of releases, with mainly packaging fixes and debugging deployments of DataKit CI in [LinuxKit](https://github.com/linuxkit/linuxkit).

## PRs merged

**Build and Packaging:**

- The automated release infrastructure was improved to support Jbuilder ([moby/datakit#543](https://github.com/moby/datakit/pull/543) [@samoht](https://github.com/samoht)).
- The GitHub bridge version constraints were fixed to improve OPAM installation ([moby/datakit#544](https://github.com/moby/datakit/pull/544) [@samoht](https://github.com/samoht)).
- The test harnesses were also extended to test on OCaml 4.03.0, which is the new minimum supported version of the compiler as of MirageOS 3.0 ([moby/datakit#546](https://github.com/moby/datakit/pull/546) [@samoht](https://github.com/samoht)).

**Functionality improvements:**

- The example ‘self-ci’ was updated to use the latest release of the GitHub bridge ([moby/datakit#545](https://github.com/moby/datakit/pull/545) [@samoht](https://github.com/samoht)).

### Ongoing activity

- [@samoht](https://github.com/samoht) updated the development tree to support the latest versions of the 9P and Lwt libraries ([moby/datakit#547](https://github.com/moby/datakit/pull/547) [@samoht](https://github.com/samoht)).
- [@djs55](https://github.com/djs55) has fixed the linking of the `protocol-9p-unix` subpackage to correctly link against `io-page.unix` ([mirage/ocaml-9p#121](https://github.com/mirage/ocaml-9p/pull/121) [@djs55](https://github.com/djs55)).
- [@talex5](https://github.com/talex5) has been debugging a possible regression in the latest releases by investigating why Irmin is reporting all remote branches as having invalid names ([mirage/irmin#440](https://github.com/mirage/irmin/issues/440) [@talex5](https://github.com/talex5)).

## External Links or Blogs

- Some of the code that powers DataKit is going to be [launched into space](https://twitter.com/avsm/status/860058980676141056)!
- A Slack channel is now available to chat about DataKit over in [#datakit at Docker Community Slack](https://community.docker.com/registrations/groups/4316).

Other reports in this series can be browsed directly in the repository at [moby/datakit:/reports](https://github.com/moby/datakit/tree/master/reports/).

---

<div class="post-metadata">

**Author:** ![samoht](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/samoht/32/81_2.png) [@samoht](https://discuss.ocaml.org/u/samoht)\
**Post date:** [May 18, 2017, 11:14pm UTC](https://discuss.ocaml.org/t/datakit-dev-reports/224/4 "2017-05-18T23:14:34Z")

</div>

# Weekly DataKit report for 2017-05-08 to 2017-05-14 (week 19)

This report covers weekly developments in the [moby/datakit](https://github.com/moby/datakit), [mirage/irmin](https://github.com/mirage/irmin), [mirage/ocaml-git](https://github.com/mirage/ocaml-git) and [mirage/ocaml-9p](https://github.com/mirage/ocaml-9p) repositories.  
This week also saw [@dinosaure](https://github.com/dinosaure) and [@eyyub](https://github.com/eyyub) also release two new libraries that replace functionality formerly written in C with pure OCaml implementations:

- **[Digestif](https://github.com/mirage/digestif)** is a standalone cryptographic hashing library.
- **[Decompress](https://github.com/mirage/decompress)** is a zlib implementation in pure OCaml.

Both are now in the Mirage GitHub organisation and will be integrated into Irmin and DataKit over the next few months. The first step is to replace the use of `camlzip` fully, and then switch ocaml-git from using Nocrypto and GMP to the pure OCaml digestif instead. Irmin and DataKit support for both will follow after that.  
There is also an _experimental_ new Git packfile encoder and decoder called [Sirodepac](https://github.com/dinosaure/sirodepac) to let ocaml-git perform compression of repositories more easily, also by [@dinosaure](https://github.com/dinosaure).

There has also been significant progress this week on having a filesystem that Irmin and DataKit can use when compiled as unikernels (and hence only have access to a raw block device):

- [Tom Ridge](http://www.tom-ridge.com) [announced](https://lists.xenproject.org/archives/html/mirageos-devel/2017-05/msg00003.html) the [initial release](https://github.com/tomjridge/tjr_btree) of a formally verified btree filesystem. It still needs to be integrated with the MirageOS filesystem interfaces, so volunteers who want to glue things together are welcome to participate.
- [Gabriel de Perthuis](https://github.com/g2p) also responded to note that he is continuing to work on a flash-optimised pure OCaml filesystem (using [hitchhiker trees](https://lists.xenproject.org/archives/html/mirageos-devel/2017-01/msg00002.html)). It is not quite ready to open-source yet, but he anticipates doing so in the next month or so.

Between both of these efforts, it looks like a fully unikernel-aware, type-safe DataKit that persists onto a block storage device is not too far away.

Finally, we welcome [David Udelson](http://www.davidudelson.com), a junior from Cornell who has been selected as a Google Summer of Code student to work on a REST API for Irmin!

## Releases

This week saw a minor [moby/datakit:0.10.1](https://github.com/moby/datakit/releases/tag/0.10.1) release of DataKit to support the latest versions of the 9P and Lwt libraries. Both of these have some backwards-imcompatible changes, so the 0.10.1 release of DataKit lets us use the latest features.

## PRs merged

- DataKit support for the latest 9P versions was added, including upstreaming the local changes to the 9P library that were only present in DataKit before. ([datakit#549](https://github.com/moby/datakit/pull/549) [datakit#550](https://github.com/moby/datakit/pull/550) [datakit#552](https://github.com/moby/datakit/pull/552) [@talex5](https://github.com/talex5) [@djs55](https://github.com/djs55) [@samoht](https://github.com/samoht) [@avsm](https://github.com/avsm)).
- Irmin no longer logs errors on invalid branch names. ([irmin#441](https://github.com/mirage/irmin/pull/441) [irmin#440](https://github.com/mirage/irmin/issues/440) [@samoht](https://github.com/samoht)).
- Irmin support for Cstruct contents was fixed to pretty print the raw contents, not the hexdump. ([irmin#442](https://github.com/mirage/irmin/pull/442) [@samoht](https://github.com/samoht)).
- `Irmin.Hash.of_string` now does not raise an exception on invalid hash ([irmin#443](https://github.com/mirage/irmin/pull/443) [@samoht](https://github.com/samoht)).
- OCaml Git got a `commit-add-parent` command ([git#208](https://github.com/mirage/ocaml-git/pull/208) [@samoht](https://github.com/samoht)).
- Git also switched to using the new Decompress library ([git#211](https://github.com/mirage/ocaml-git/pull/211) [git#145](https://github.com/mirage/ocaml-git/pull/145) [git#145](https://github.com/mirage/ocaml-git/pull/145) [git#210](https://github.com/mirage/ocaml-git/pull/210) [@dinosaure](https://github.com/dinosaure) [@samoht](https://github.com/samoht)).

The Jbuilder build porting journey continues to be successful:

- Irmin was ported to Jbuilder, speeding it up a lot. ([irmin#444](https://github.com/mirage/irmin/pull/444) [@samoht](https://github.com/samoht)).
- OCaml-Git was also ported to Jbuilder ([git#209](https://github.com/mirage/ocaml-git/pull/209) [@samoht](https://github.com/samoht)).
- The build now also supports Mtime 1.0, which had some API incompatiblities from previous versions ([git#212](https://github.com/mirage/ocaml-git/pull/212) [datakit#554](https://github.com/moby/datakit/issues/554) [@dinosaure](https://github.com/dinosaure) [@samoht](https://github.com/samoht)).
- The `ogit` binary was temporarily missing, now fixed ([git#213](https://github.com/mirage/ocaml-git/pull/213) [@samoht](https://github.com/samoht)).

### Ongoing activity

- Work continues on switching to Digestif instead of Nocrypto (which has C stubs) ([git#214](https://github.com/mirage/ocaml-git/pull/214) [@dinosaure](https://github.com/dinosaure) [@samoht](https://github.com/samoht)).
- The 9P interfaces are being made safer against leaking exceptions on read/write ([9p#126](https://github.com/mirage/ocaml-9p/pull/126) [9p#125](https://github.com/mirage/ocaml-9p/issues/125)).
- The tests, client and server code has been refactored to make transport layer abstract. The default is still to use 9p but this is the first step to replace it by gRPC: [datakit#551](https://github.com/moby/datakit/pull/551) [@samoht](https://github.com/samoht).

Other reports in this series can be browsed directly in the repository at [moby/datakit:/reports](https://github.com/moby/datakit/tree/master/reports/).

---

<div class="post-metadata">

**Author:** ![thomas\_ridge](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/thomas_ridge/32/1478_2.png) [@thomas\_ridge](https://discuss.ocaml.org/u/thomas_ridge)\
**Post date:** [May 19, 2017, 9:09am UTC](https://discuss.ocaml.org/t/datakit-dev-reports/224/5 "2017-05-19T09:09:20Z")

</div>

Thomas - thank you for posting these. Very useful, and much appreciated.

---

<div class="post-metadata">

**Author:** ![avsm](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/avsm/32/6_2.png) [@avsm](https://discuss.ocaml.org/u/avsm)\
**Post date:** [May 19, 2017, 9:47am UTC](https://discuss.ocaml.org/t/datakit-dev-reports/224/6 "2017-05-19T09:47:46Z")

</div>

@samoht this is really excellent, thanks for posting. Regarding the format of these reports, perhaps a good structure would be for the main post topic to briefly explain what DataKit and the associated reposirtories are (from the [README](https://github.com/moby/datakit/blob/master/README.md)), and then you can just add a weekly comment with the latest dev report?

I’ll also look into generating [MirageOS](https://mirage.io) dev reports in the same style, and perhaps we can add a category for updates so that other projects can also participate with their own.

---

<div class="post-metadata">

**Author:** ![samoht](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/samoht/32/81_2.png) [@samoht](https://discuss.ocaml.org/u/samoht)\
**Post date:** [May 20, 2017, 12:28am UTC](https://discuss.ocaml.org/t/datakit-dev-reports/224/7 "2017-05-20T00:28:02Z")

</div>

That makes sense, I’ve edited the first post to add a bit of context.

---

<div class="post-metadata">

**Author:** ![samoht](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/samoht/32/81_2.png) [@samoht](https://discuss.ocaml.org/u/samoht)\
**Post date:** [May 30, 2017, 10:47am UTC](https://discuss.ocaml.org/t/datakit-dev-reports/224/8 "2017-05-30T10:47:50Z")

</div>

# Weekly Datakit dev report for 2017-05-15 to 2017-05-28 (week 20 and 21)

This report covers two weeks of developments in the [moby/datakit](https://github.com/moby/datakit) [mirage/irmin](https://github.com/mirage/irmin)  
[mirage/ocaml-git](https://github.com/mirage/ocaml-git) [mirage/ocaml-9p](https://github.com/mirage/ocaml-9p) repositories.

This week saw DataKit add direct client bindings that use Git directly, and  
thus do not require a 9P server to be running. This improves performance and  
deployment simplicity of DataKit, but removes the intermediate flexibility that  
a 9P filesystem offers.

Datakit also features a local filesystem bridge, in addition to the remote  
GitHub one, to faciliate desktop development with local repositories. This  
filesystem bridge was improved this week by using the latest Irmin release,  
with better watch support [moby/datakit#577](https://github.com/moby/datakit/pull/577).

Most of these improvements are still in the various `master` branches of  
Irmin and DataKit, and will appear in a release near you soon.

If you are interested in contributing to any of these repositories, then  
[mirage/irmin#415](https://github.com/mirage/irmin/issues/415#issuecomment-304460556) has a thread about coding standards and getting started  
with the Irmin REST API.

## PRs merged

Client API:

- **Golang:** separate user config from defaults in the database ([moby/datakit#523] [@djs55] [@talex5](https://github.com/talex5)).
- **Golang:** add `List` func in the `Snapshot` module to get files in a directory ([moby/datakit#578] [@ebriney](https://github.com/ebriney))
- **OCaml/9P** : Remove `rename` API calls in the client, as there were no users and it doesn’t work with 9P directories ([moby/datakit#563](https://github.com/moby/datakit/pull/563) [@talex5](https://github.com/talex5) [@samoht](https://github.com/samoht))
- **OCaml/Git:** Fix the `exists` function in the new Git client API ([moby/datakit#576](https://github.com/moby/datakit/pull/576) [@samoht](https://github.com/samoht)).
- **OCaml:** Renamed `Datakit_path` into `Datakit_client.Path` and `Datakit_S` into `Datakit_client.S`, to create a `Datakit_client` namespace ([moby/datakit#558](https://github.com/moby/datakit/pull/558) [@samoht](https://github.com/samoht))
- **OCaml:** Added client bindings that use Git directly without a 9P server process ([moby/datakit#559](https://github.com/moby/datakit/pull/559) [@samoht](https://github.com/samoht))
- Use `Irmin.Merge.idempotent` instead of re-implementing our own ([moby/datakit#564](https://github.com/moby/datakit/pull/564) [@samoht](https://github.com/samoht))
- `DK.commit` now fails if the commit does not exist ([moby/datakit#565](https://github.com/moby/datakit/pull/565) [@samoht](https://github.com/samoht))
- Build the `9pmount` helper in the the `datakit-client-9p` library instead of `datakit-client) ([moby/datakit#573](https://github.com/moby/datakit/pull/573) [@samoht](https://github.com/samoht))

Bridges:

- Use latest Irmin to fix issues with sync of local Git bridge ([moby/datakit#561](https://github.com/moby/datakit/pull/561) [@samoht](https://github.com/samoht))
- Fix regression in Irmin watches ([mirage/irmin#446](https://github.com/mirage/irmin/pull/446) [@samoht](https://github.com/samoht)) that was causing the local filesystem bridge to fail ([moby/datakit#553](https://github.com/moby/datakit/issues/553) [@samoht](https://github.com/samoht) [@avsm](https://github.com/avsm))
- The CI now uses the dev version of `protocol-9p-unix` to avoid `End_of_file` exceptions ([moby/datakit#569](https://github.com/moby/datakit/pull/569) [mirage/ocaml-9p#126](https://github.com/mirage/ocaml-9p/pull/126) [mirage/ocaml-9p#125](https://github.com/mirage/ocaml-9p/issues/125) [@samoht](https://github.com/samoht))

Packaging:

- Update to latest Irmin ([moby/datakit#556](https://github.com/moby/datakit/pull/556) [@samoht](https://github.com/samoht))
- Use mtime 1.0 ([mirage/irmin#445](https://github.com/mirage/irmin/pull/445) [moby/datakit#560](https://github.com/moby/datakit/pull/560) [moby/datakit#554](https://github.com/moby/datakit/issues/554) [@samoht](https://github.com/samoht))
- Clean up jbuild files after the build system port ([moby/datakit#568](https://github.com/moby/datakit/pull/568) [@samoht](https://github.com/samoht)) and add missing dependency towards `str` ([moby/datakit#572](https://github.com/moby/datakit/pull/572) [@samoht](https://github.com/samoht))
- Stop using camlzip in Datakit in favour of the ocaml-git implementation ([moby/datakit#570](https://github.com/moby/datakit/pull/570) [@samoht](https://github.com/samoht))
- Update CI to latest version of Redis ([moby/datakit#567](https://github.com/moby/datakit/pull/567) [@samoht](https://github.com/samoht))

Tests:

- Run the 9p tests in Datakit by default ([moby/datakit#566](https://github.com/moby/datakit/pull/566) [@samoht](https://github.com/samoht)) and fix warnings in the tests ([moby/datakit#571](https://github.com/moby/datakit/pull/571) [@samoht](https://github.com/samoht))
- Fix the collection of build artefacts on Windows and OSX ([moby/datakit#574](https://github.com/moby/datakit/pull/574) [@guillaumerose](https://github.com/guillaumerose) [@samoht](https://github.com/samoht))
- add opam cache for appveyor [moby/datakit#575](https://github.com/moby/datakit/pull/575) [@samoht](https://github.com/samoht)
- Remove warnings in tests [mirage/irmin#447](https://github.com/mirage/irmin/pull/447) [@samoht](https://github.com/samoht)
- Remove duplicated (and unused) file [mirage/ocaml-git#215](https://github.com/mirage/ocaml-git/pull/215) [@samoht](https://github.com/samoht)
- Remove warnings in tests [mirage/ocaml-git#216](https://github.com/mirage/ocaml-git/pull/216) [@samoht](https://github.com/samoht)

Other reports in this series can be browsed directly in the repository at [moby/datakit:/reports](https://github.com/moby/datakit/tree/master/reports/).

---

<div class="post-metadata">

**Author:** ![samoht](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/samoht/32/81_2.png) [@samoht](https://discuss.ocaml.org/u/samoht)\
**Post date:** [June 6, 2017, 8:44am UTC](https://discuss.ocaml.org/t/datakit-dev-reports/224/9 "2017-06-06T08:44:11Z")

</div>

# Weekly DataKit dev report for 2017-05-29 to 2017-06-04 (week 22)

This report covers weekly developments in the [datakit](https://github.com/moby/datakit) [irmin](https://github.com/mirage/irmin) [git](https://github.com/mirage/ocaml-git) [9p](https://github.com/mirage/ocaml-9p) repositories.

This week saw a significant simplication in how the GitHub bridge is deployed, as it can now run without a server and use a local Git repository directly ([datakit#577](https://github.com/moby/datakit/pull/577) [@talex5](https://github.com/talex5) [@samoht](https://github.com/samoht)). This in turn simplifies the deployment of DataKit as part of various CI services use it now, such as the [LinuxKit](https://github.com/linuxkit/linuxkit) Moby project.

The GitHub bridge also adds an an `owner` file to identify the PR creator, which can be used by DataKitCI plugins ([datakit#587](https://github.com/moby/datakit/pull/587) [@samoht](https://github.com/samoht)).

For those getting started with DataKit and Irmin, [@nickbetteridge](https://github.com/nickbetteridge) started an issue on how to get started ([irmin#450](https://github.com/mirage/irmin/issues/450)).  
Our GSoC intern [@dudelson](https://github.com/dudelson) is also active on [irmin#415](https://github.com/mirage/irmin/issues/415) about the HTTP REST API ([@dbuenzli](https://github.com/dbuenzli) [@dudelson](https://github.com/dudelson) [@samoht](https://github.com/samoht)), and has started  
a tracking issue on contributing documentation improvements ([irmin#451](https://github.com/mirage/irmin/issues/451)). This is massively appreciated!

As part of the move to standardise interfaces across Moby components, there is working on adding building GRPC and CaPnP RPC libraries so that DataKit can make use of these. Check in with [@talex5](https://github.com/talex5) if you are interested in contributing.

Releases are now being cut with this functionality, starting with [git:1.11.0](https://github.com/mirage/ocaml-git/releases/tag/1.11.0).

## PRs this week

- Git client: fix the `exists*` functions ([datakit#576](https://github.com/moby/datakit/pull/576) [@samoht](https://github.com/samoht))
- Datakit: only the binary is using `datakit_io` ([datakit#582](https://github.com/moby/datakit/pull/582) [datakit#580](https://github.com/moby/datakit/issues/580) [@samoht](https://github.com/samoht))
- Refactor the library to wrap everything under the Datakit namespace ([datakit#583](https://github.com/moby/datakit/pull/583) [@samoht](https://github.com/samoht))
- Simplify the creation of DataKit stores using an Irmin backend ([datakit#584](https://github.com/moby/datakit/pull/584) [@nickbetteridge](https://github.com/nickbetteridge) [@samoht](https://github.com/samoht))
- Fix ocamldoc ([datakit#585](https://github.com/moby/datakit/pull/585) [@samoht](https://github.com/samoht))
- Various build fixes ([datakit#588](https://github.com/moby/datakit/pull/588) [@talex5](https://github.com/talex5)) and jbuilder improvements ([git#218](https://github.com/mirage/ocaml-git/pull/218) [@samoht](https://github.com/samoht)).

## External Links or Blogs

- [“A quick tour of the Tezos code base, and the state of its development”](https://medium.com/tezos/a-quick-tour-of-the-tezos-code-base-and-the-state-of-its-development-c4e5fcb34b8a)

Other reports in this series can be browsed directly in the repository at [datakit:/reports](https://github.com/moby/datakit/tree/master/reports/). There is also a [Docker Community Slack channel](https://docs.docker.com/opensource/get-help/) in `#datakit`.
