Library/framework for message-queue-based architecture?

Learning
1

I have a hunch that large systems benefit from using message queues to communicate: messages permit asynchronous communication and the details of a queue (length, throughput) give insight into the workload of a system. Are there OCaml libraries or frameworks that support such an architecture, possibly including RPC mechanisms? I believe Lwt uses queues in its implementation but I believe they are not meant to be observed by client code. Sorry if this sounds vague - I’m just trying to explore the landscape.

6 Likes
2

You mean things like ZeroMQ? There are ZeroMQ bindings for OCaml. (More than one IIRC.)

2 Likes
3

That looks interesting. I was also thinking about something with an Erlang flavour but implemented as an OCaml framework.

4

I’m unaware of anything that implements the OCaml message passing paradigm but I agree that such a library would be interesting.

5

There’s also https://github.com/essdotteedot/distributed — I haven’t used it yet but it looks interesting.

6

I don’t know if you saw this thread from several days ago, but I am too in search of this grail :smiley:

Still haven’t settled on a solution though :frowning:

7

So the closest things you’re going to find are RPC parallel, which in erlang would sort of be like working with nodes and rpc in distributed erlang and https://github.com/owlbarn/actor which is like working with Actors in distributed erlang.

As much as I like Actors, they don’t suit typed Functional Languages well, from an API perspective.

If you’re willing to implement something from scratch, that’s closer to Ocaml’s natural semantics, look through the code of Async and the STDLibs ConcurrentML implementation, and figure out how to make it network based on top of ZMQ, or stick to the RPC model.

2 Likes
8

I am curious about

As much as I like Actors, they don’t suit typed Functional Languages well, from an API perspective.

Can you explain why?

Thanks

9

See also https://github.com/andersfugmann/amqp-client for AMQP messaging bus support, it works well with RabbitMQ for example.

2 Likes
10

I highly recommend ZMQ for distributed programming in OCaml.
https://github.com/UnixJunkie/daft used it.
Forget about RPC. Design your protocol: which agent generate which messages, receives which messages and what are the message types.
Then, let the type system tell you where to fill the holes.

4 Likes
11

Yeah, I am also curious if there’s any OCaml’s framework using message queues to communicate.

As far as I know, there’s a 1 Billion DAU app which is called “WeChat” uses that framework.

Relate refer:

Some of the highlights:

  1. Each micro-service maintains its own priority thresholds for admitting requests, and monitors its own load status by checking the system-level resource indicator such as the average waiting time of requests in the pending queue.

  2. IO threads and worker threads use in-memory queue to communicate.

I’m not sure the thread mechanisms of OCaml, for example, does OCaml support worker thread and io thread? I know ZMQ is amazing, but I think threads share the same memory is much efficient.

12

Sorry for the very late response, but basically it’s not so much functional languages, as the ones that rely very heavily on the type system,

Basically an actor in most languages is implemented the following way it can receive any type which afaik ocaml doesn’t support reflection and any actor in the actor system can communicate with eachother, and there is no output type, so even if you get around it by forcing an input type on actors your flexibility is diminished when you implement it in ocaml, and it’s hard to compose.

You can get around some limitations with objects, but who the fuck uses objects in Ocaml, without alarm bells going off.

You can’t pass around the nice convenient {op, arg_a, arg_b} style tuples like you can in erlang.

Not to mention implementing the actor system itself is a bit of a headache in Ocaml.

GenServer is an entirely different story.

1 Like