[ANN] BAP 2.1.0 Release

[ivg]

@Jonathanks

I couldn’t find a simple embedded key-value store

I hope at least one of them will match your definitions of simple and embedded

$ opam search 'key-value' 
# Packages matching: match(*key-value*)
# Name                # Installed # Synopsis
arakoon               --          A distributed key-value store that guarantees consistency above anything else.
bap-primus-dictionary --          BAP Primus Lisp library that provides dictionaries
cryptodbm             --          Encrypted layer over the dbm library: access to serverless, key-value databases with symmetric encryption.
dokeysto              --          The dumb OCaml key-value store
dokeysto_camltc       --          The dumb OCaml key-value store w/ tokyocabinet backend
dokeysto_lz4          --          The dumb OCaml key-value store w/ LZ4 compression
gmap                  0.3.0       Heterogenous maps over a GADT
index                 --          A platform-agnostic multi-level index for OCaml
kyotocabinet          --          OCaml bindings for Kyoto Cabinet DBM
mirage-kv             --          MirageOS utilities for interfacing with key-value stores.
mirage-kv-lwt         --          MirageOS utilities for interfacing with key-value stores.
mirage-kv-unix        --          Key-value store for MirageOS backed by Unix filesystem
podge                 --          Shortcuts and helpers for common tasks in OCaml ecosystem
redis                 --          Bindings for the key-value cache and store redis.
ringo                 --          Caches (bounded-size key-value stores) and other bounded-size stores
string_dict           --          Efficient static string dictionaries
tezos-storage         --          Tezos: low-level key-value store for `tezos-node`
trakeva               --          Transactions, Keys, and Values; with Postgresql and/or Sqlite.

[Yaron_Minsky]

@ivg

OK, so after recompiling everything with -O3, I’ve got about 40% improvement in such simple task as binary disassembling, which is fantastic, nearly twice as fast. On more complex benchmarks the results are less impressive, something between 10% to 20%.

That’s great. We’ve seen lots of 10-20% improvements, but I can’t think of a case where things went 40% faster.

The cons. It took a lot of time to compile everything (I can’t tell how much, because in the middle I run out of disk space and had to revive my system) and the binaries are now roughly 4-5 times larger:

du -hs ~/.opam/*
3.6G    /home/ivg/.opam/4.09.0
15G     /home/ivg/.opam/4.09.0+flambda

Interesting. We definitely see a big blowup in total artifact size, but less so in the final executables. I’m curious what’s causing that. For what it’s worth, there’s been some tantalizing progress towards better whole-program dead-code elimination. Not sure where that will end up, but I’m hoping we’ll have something to announce in the coming months.

[ivg]

@Yaron_Minsky

Interesting. We definitely see a big blowup in total artifact size, but less so in the final executables. I’m curious what’s causing that.

The binary itself is four times larger (25 vs 100 Mb), which is partially because we have the -linkall option to enable loading plugins in runtime. When I started to count our dependencies, i.e., the dependencies that are linked into the binary, I found out that roughly 30Mb is core_kernel, bap libraries together comprise less than 5 megabytes, and the runtime is about 1Mb. And yes it doesn’t sum up to 100 Mb.

When I used readelf to examine the structure of the binary, I discovered that the rela.dyn section (the section with relocations is 20Mb large. Funny, it is even smaller than the .text section that contains the actual code, which is only 17Mb large. And yes, we also have 20Mb of dwarf information. To summarize, out of 100Mb only 17Mbytes is the actual code (17%) the rest 83% is some auxiliary information.

[XVilka]

Core Theory and Knowledge Base are much more interesting concept. BIL is so XX century, CT is the future! See:

• http://binaryanalysisplatform.github.io/knowledge-intro-1
• http://binaryanalysisplatform.github.io/bap-2-release

[Chet_Murthy]

Re: k/v store, there’s also “orocksdb”. Wrapper for “RocksDB”, which should be plennty powerful.

[Drup]

There are also lmdb bindings, for the basics.

[XVilka]

There are two draft PRs to discuss these options as global opam flags:

• https://github.com/ocaml/opam-repository/pull/16619
• https://github.com/ocaml/opam-repository/pull/16753

Was there anything decided about that? Enabling these optimizations for the whole switch would be a tremendous help for the OCaml-written services deployment.