After the discussion on another thread (Advantages of OCaml over Rust), I have started some work towards exploring the types needed to start an implementation of HTTP2. This might not be the best code as I’m still making my way through learning OCaml. Feedback and/or contributions are very welcome. For now i’ll be reading the HTTP2 specs (https://httpwg.org/specs/rfc7540.html) and looking at some existing implementations. (Rust, Dart and Haskell are a few that have existing implementations)
4 Likes
I recommend also looking at the architecture of libraries like httpaf. The protocol is a bit different but you would probably want to be able to fit any implementation into an existing library like that.
That is definitely the eventual goal. My current goal is to get more familiar with serialization/deserialization with Angstrom and Faraday, and then move forward from there.
A thing to keep in mind about HTTP2 is that messages are composed with frames using an octet-counting binary data encoding. Using a parser combinator like Angstrom for that may not be the best choice.
I don’t want to derail you, but HTTP2 is one of the motivations I had in mind when I started my Orsetto project. It provides two different mechanisms for scanning and emitting messages in various formats, mainly informed by the discussion on Protocol Mechanisms in RFC 3117.
For binary encoded octet-counted framing schemes, I implemented imperative scanners and emitters that work with encoding and decoding schemes comprising primitives for portable primitive data types and combinators for composing header and packet framing formats.
As Orsetto is still very unstable, I can’t recommend you use it for this project. Instead, if I were in your position, then I think I would dispense with the parser combinator and just code an HTTP data structure library directly over the OCaml standard library.
2 Likes
I forgot to mention, if you’re interested in how one might use Orsetto’s core foundation scanners and emitters to implement a fairly complicated binary data format, you could look at my recently committed work on the Concise Binary Object Representation [RFC7049].
@jhw Thanks for the reply! Looking at angstrom and faraday today I found that it has helpers to work with 8 bit integers. That being said, I initially wanted to explore Angstrom, mostly to learn enough to play around with the current httpaf library.
For just starting with a proof of concept serialization/deserialization I was leaning towards bitstring for now, as it feels very similar to the bit pattern matching I was used to in erlang/elixir, and would provide a really convenient means of slicing and extracting bits.
I’ll definitely take a look at Orsetto. It looks really interesting at a first glance. Specially since my first goal is more about exploring the types and learn more about the protocol itself I wouldn’t mind working with a library that isn’t stable yet.
3 Likes
I implemented Hpack compression (https://github.com/314eter/ocaml-hpack). The interface is maybe not ideal yet, but the implementation works en performs well.
6 Likes
That’s really cool! I had to put this on the back burner for a while because of other work commitments. (mostly exploring the js_of_ocaml landscape) but I’d definitely like to revisit this soon.
Nice job functorizing over unix/lwt!
I’ve added an entry to OCamlverse dealing with this topic. Feel free to keep it up to date.
I’m looking at HTTP2 now. The (de)serialization can be done the same way as I did for hpack, but is more straightforward. The logic of the protocol itself is more complex.
1 Like
From my experience, it’s not the best to functorize over I/O abstraction. Indeed, semantics of implementations are different and it’s better to provide a pure OCaml implementation (like ocaml-tls, httpaf/cohttp or decompress) and let I/O logic outside the scope of the protocol/format implementation.
Then, as ocaml-tls or httpaf/cohttp, the developer can provide sub-packages which implement I/O logic in the way of unix/lwt/async - see cohttp-lwt/cohttp-lwt-unix/etc.
4 Likes
I tried to avoid an extra layer of buffering because http2/hpack can be implemented “pass-though” without lookahead or backtracking. All your examples seem to copy data to a buffer, and pass this buffer to the implementation.
Is this possible with your approach too? Or is it better to use a buffer anyway?
This question is a bit complex because it depends on your protocol or your format, your deal between memory consumption or your speed, and by the end some security issues.
For example, decompress expects 2 buffers (and it does not care about the size of these buffers). It never asks to the client to grow the input buffer and just compute what you fill (from a recv syscall for example - but, again, this is outside the scope of decompress) and tell you what it consumes. Finally, it writes on your fixed-size output buffer and tell you to flush it if it does not terminate the inflation of the flow.
By this way, the client can just read and write over what decompress do - wrap it on a an async Pipe abstraction, a lwt Stream abstraction or something else. The client (and it’s the case of the easy example) can have a buffer which grows automatically too (but it’s a security issue) and just output an completely inflated string. Finally, on this approach, we let the user (and it’s the case of ocaml-git) to control precisely memory needed to inflate or deflate any flow.
However, this is not necessary the case for any others protocols. angstrom, by the alteration operation, needs a backtracking for example (and an implementation of HTTP1.1 needs to use this alteration specifically for the LWS token for example to extend - and grow - a value of an header field). ocaml-tls/TLS 1.2 has the same problem when each packets are variable length - so internally, we need to keep entirely each packets to start to decode them. Finally, you can arbitrary set a (controversial?) limit depending on the context of where the protocol or the format was developed - limits of SMTP are intrinsic with MTU for example.
ocaml-git/Git has this kind of limit where we can surely say than if the backtracking needs more than 80 characters, we should leave up and return an error.
So my response is: it depends on what you implement and what you want to do. decompress was developed to be used on unikernel-context (and ocaml-git too) as a long lived process, so memory consumption of any computation should be predictable (to avoid an Out_of_memory). But it’s not necessary what you want - constraints are strong, and arbitrary limits appear like a white rabbit.
2 Likes
Considering all this, I think my current approach is flexible enough to work with buffered/unbuffered/asynchronous I/O, leaving the control to the user, with predictable memory usage. But I added an Async implementation as an experiment and quickly ran in some problems (e.g. different error handling), so I believe functorized I/O is a bad idea now.
But implementing the core algorithm in pure ocaml and leaving I/O to the client is not ideal too, because then you still have to choose something for your input - probably some kind of buffer - which is less flexible and uses more memory.
Another option is to give up flexibility, and just use Lwt.
An update for https://github.com/anuragsoni/h2
I’ve had some time to start taking a look at this again this week. I spent some time reading up on Angstrom and decided to continue with it.
Current state: Parsing of the frame types (at the very least, the success cases look correct)
Next steps:
- I’ve been testing with data from existing implementations in python/Go etc. I would like to set up a proper test harness using: https://github.com/http2jp/http2-frame-test-case. I would also like to setup automated comparisons with other existing implementations that are known to be spec compliant. (I might need help here)
- Spending time with
faradaybefore starting on the serialization - clean-up the parser implementation/types and add documentation cross-referencing the relevant sections of the RFC
8 Likes