Fsync'd database

I’m trying to write a distributed database currently. For the algorithms I’m working with several values must be written to disk before it can commit.

Before I disappear down the rabbithole of writing a durable (fsync’d etc) database, is there anything which already does that?

I know about irmin, however I can’t see whether it fsyncs to disk or not…

I’m assuming you want as-close-to-100%-ocaml as possible, in which case, I know of nothing. But if you’re willing to wrapper C++, you might want to look at the “recordio” code at the bottom of leveldb. It’s … well-designed and -implemented. And does what you asked for (in addition to more).

Neither of the Irmin file-system backends (irmin-fs and irmin-pack) currently fsync to disk, but we have an open issue to add an option for this in irmin-pack.

This would likely not be difficult, as the work has already been done in mirage/index, which forms the core of the irmin-pack backend. We’d need to lift the with_fsync option to the sync function on the Pack.S signature. Contributions are very welcome.

1 Like

I’ll have a look at that at some point in the future, I haven’t worked out yet how irmin works yet tho :slight_smile:

For now I’m just going to fudge a WAL.

In opam: arakoon and redis could do what you are looking for.
I guess arakoon is mostly OCaml while redis is just some bindings.

1 Like

Perhaps you’re asking “is there a durable fsync’ed database accessible from Ocaml?” If so (and setting aside stuff like ODBC drivers, on the assumption that you’re looking for something single-process) I’d suggest leveldb/rocksdb, as well as sqlite. I don’t know about sqlite, but for sure the former two do the job right.

Aerospike in Strong Consistency mode (enteprise only) has a commit-to-device option. More information is available at: https://www.aerospike.com/blog/developers-understanding-aerospike-transactions/.