some ppx magic on records so that record array becomes a table; where each field of the record is a column
the ppx magic is some typesafe way to expose the fields of the record to the SQL repl
we now expose some array-of-records; so that the runtime can (1) query and (2) update the array of records [then causing things to get triggered OCaml side]
why not use an existing db ?
Much of DB complexity arises from: durability (storage to disk), recovery (from write ahead log), query optimization (given disk latency); we don’t have any of these problems since we are purely in memory.
By going with a pure-ocaml solution, we also get the benefit of ocaml’s type system (instead of akward encodings of variants)
@dbuenzli : If you have time, I am interested in hearing what your original aspirations were with rel, what the reality turned out to be, and why are you now considering phasing it out (i.e. better alternative, or giving up on idea).
I tried to dig thru the paper for examples of their proposal, but … got exhausted. 66 pages! And well past halfway, it was still “preliminary discussion”! Whew! grin
For small datastructures, this is true. But as they get larger, even as they remain in-memory, both query optimization and even just efficient use of the memory hierarchy become important considerations. With modern processors, “main memory is effectively at infinity”. For sufficiently complex datastructures, atomicity of complex updates becomes important. If they’re big enough, and your system is valuable, you mgiht end up needing to support multithreaded access, so then you might need serializability guarantees. And with GC, it becomes important to manage bulky data as if it were offline anyway. So it’s possible that it’s worth using an existing DB, and putting its storage on a memory-backed FS.
Have a nice interface for dealing with databases in OCaml. In particular along this line of papers, which enable composable queries (in contrast to SQL) that efficiently compile to a single select statement (no query avalanche problems, the line of work is related to the work on LINQ @Chet_Murthy mentions).
I also kind of struggled with the last paper, but I met by chance Oleg in the street last year in Ljubljana and he pointed me to more material I didn’t have the time to look into yet.
Turned out quite well but I had to ship software to users so I didn’t have time to finish it and so the query language is too limited for now (no group by and aggregates) and you have to resort to raw (but typed) SQL (see this project if you are interested to see it both in action and missing in action).
I’m not considering phasing rel out but I’m considering phasing out the in memory queries on lists or arrays – because it uses Obj and I didn’t think people would be interested by it.
That being said another thing you might be interested in looking into is µKanren, you can find a list of OCaml implementations on this page. I also tried one here but AFAIR it mainly succeeded in producing infinite loops.