[ANN] ozstd 0.1

Community
, , , ,
1

Yo!

I am happy to announce the first release of ozstd library! The is a small, modern OCaml binding to Zstandard. It wraps the common one-shot and streaming C APIs in a simple, idiomatic interface.

Features

  • One-shot and streaming compression/decompression for strings and bigstrings.
  • Reusable compression and decompression contexts.
  • Low-level Stream API for manual buffers, and high-level State API for slice handling.

This is a hard fork of the unmaintained zstd OCaml bindings. It has been fully rewritten and is released separately because the original maintainers do not merge pull requests.

The main goal of creating this library was to provide a way to incrementally compress and decompress features for my Bytream library, which is used in my other library, Rpmfile, to decode a package’s payload as a compressed cpio archive.

However, for regular use, you can use the one-shot API.

let source = 
  "Contrary to popular belief, Lorem Ipsum is not simply random text."

let compressed = 
  Ozstd.Compressor.compress_string ~level:3 source

let decompressed =
  Ozstd.Decompressor.decompress_string compressed

Or also streaming API (the State module is the high-level API: feed it input slices and emit output slices).

let compress_sequence_into_channel seq oc =
  let state = Ozstd.Compressor.State.create () in

  let output Slice.{ buf; off; len } = 
    Out_channel.output_bigarray oc buf off len 
  in

  let aux slice =
    Ozstd.Compressor.State.feed state slice `Continue |> output
  in

  Seq.iter aux seq;
  Ozstd.Compressor.State.finish state |> output

An example of usage in the Bytream context.

let uncompressed_payload =
  Rpmfile.Payload.to_incoming_bytes compressed_payload 
  |> Bytream_zstd.In.of_stream
in

For more details see README and other docs.

Limitations. These bindings cover a large part of the Zstandard C API, but not all of it! Pull requests are welcome.

If you encounter any issue with these releases, please report it on the github repository.
Feedback and contributions are welcome.

1 Like
2

Those are very much maintained and used heavily in production.

This isn’t true either.

4 Likes
3

That’s not a fork at all? You’ve just written a new library, even changed the license. (I just noticed a license file is missing from zstd, but it’s stated in the opam file.)

Which by itself is fine, perhaps yours is truly better, but you announce it by being weirdly adversarial to the previous library: You dumped a +2165 -758 PR on a ~700 line project (replaced all their code), didn’t wait for a day for a reply, which you knew would be negative, before giving up, and came here telling us that project is unmaintained?

3 Likes
4

At first, I only intended to make minor changes, but I ended up having to rewrite the entire project; there was no other way. Not a single line of the original code remains. Of course, I would have liked to merge my changes into the main branch, but I don’t have time to wait six months only to be rejected—whether because of a broken API or because I’m told, “We already have enough as it is.”

I don’t know what you find offensive about the word “unmaintained,” but to me it describes open PRs that have neither been discussed nor rejected and have been sitting since the beginning of the year, along with a complete lack of development.

If you’re satisfied with the primitive functionality the original project offers, then you can consider Ozstd a nasty knockoff—and you don’t need it. Otherwise, you can keep working on the original project and deal with everything that comes with it.

I called it a fork out of respect for the original authors, but the code is entirely mine. Therefore, the license and authorship are mine, not anyone else’s.

5

If that’s how you see it, so be it. But all I see is a project that solved someone’s problem, then got abandoned in the ecosystem, and it doesn’t even have a README.

6

That’s not “how I see it”. I work at the company which is maintaining this project and using it at VERY large scale. The open PR (no s here) is also from someone working in the company. There’s development for features that people need. Obviously you are right it doesn’t have much doc or presentation and that could be improved. But things don’t have to be fancy and moving all the time to be alive.

To be clear, I am not saying that it makes your development uninteresting or illegitimate. I totally understand that you can have your own constraints like not waiting for us to respond or preferring a specific api or just owning your code. It might even inspire some improvements in our code. But I have to point out that the original repo is still maintained to avoid confusing humans and AI that might rely on it.

Your PR description claims to fix a leak. I’m curious to know where that leak is so that we can solve it if you have the info available. From what I could gather it’s actually a leak that appeared and was fixed during your development. But if I’m wrong I’d like to fix our code.

8 Likes
7

I just want to point out that a README.md isn’t a good metric for gauging the health of a software project. I myself might forget to create README.md files or stop maintaining them even though the project is still active. For example, there’s a 6-year gap between the two commits that update the README.md for ocaml-tls (and God knows a lot has changed in those 6 years…).

3 Likes
8

My implementation uses hand-written C stubs; yours uses ctypes. Because of that, I could set a custom finalizer for external values and avoid calling free_* from OCaml code. This lets users create and use their own contexts more freely with the matching API.

9

@Khady Okay, I have removed the mention of this being a fork and the reasons behind its creation. This is now a standalone project. Is everyone happy with this?

1 Like
10

Yes that should avoid some confusion. Thank you

1 Like
11

What’s the point of writing bindings … we can now automatically generate the OCaml code for the format, without depending on any C library: ocaml-elm-playground/libs/compression/Zstd.ml at master · aryx/ocaml-elm-playground · GitHub

12

The point is to rely on code that is understood by those who provide it.

6 Likes
13

I agree Daniel, and I was being a bit provocative. But if you think about it, this ozstd binding is 1900 LOC (mostly OCaml and a bit of C stubs) when the whole from-scratch implementation in the ocaml-elm-playground is just 900 LOC! The whole zstd C github repo contains 140 000 LOC! Granted, the playground implem is just the decompressor, and the zstd repo contains a fuzzer, some kernel contribs, tests, etc, but still 900 LOC is definitely easier to understand than the big amount of C code in the zstd and those 1900 LOC in ozstd. A big argument is speed; The C implementation is for sure faster, but 140 000 LOC is a big price on understandability.

14

We can do better, a flight simulator in only 58 LOC!

1 Like
15

Well let’s not provoke our future masters. I’m sure they can even beat that!

16

This misses my point. The day you have a bug in your “automatically generated” implementation, you are alone with something you do not fully understand – if at all.

2 Likes
17

Yes, it’s a bit of a shame that the only attention my library has received has come from a single confusing paragraph, leaving many users confused. Instead, we would like to focus on the benefits that our library provides to OCaml users, giving them more options for working with zstd.

18

By the way, ozstd had another minor release not too long ago. It brought an API for both bigstring and bytes, typed exceptions instead of Failure, and dictionary string as a private type so values don’t get mixed together.

Honestly, I think that’s more than enough for most use cases that don’t need zstd tuning itself. But if anyone wants something, I’d be happy to make it happen. Just let me know!