2

I have a particle simulation program and want to split it up onto multiple machines in a LAN. Each server node calculates a number of particles' positions and needs to send the updated positions (i.e. the same data) to all other server nodes. So each server node needs to be connected to all the others.

This reliable all-to-all communication should work without a central server/message broker and should have few latency. Also sending same data multiple times over the network should be avoided, if possible (as with multicast).

I looked around and ZeroMQ (supports Bus, Pub/Sub and PGM) as well as Nanomsg (supports Bus and Pub/Sub, but no PGM) look like great libraries fitting to the problem.

Which networking technique would be suited best in this case, a Message Bus, Publish/Subscribe or Multicast/PGM?

1 Answer 1

2

Pub / Sub is implemented as multicast, the use of PGM as a method of transport is transparent to a ZeroMQ client. ZeroMQ works hard to hide implementation details from the client, which is one of the reasons why it works so well. But don't let the simplicity fool you, ZeroMQ is an expertly engineered messaging solution that is very powerful and flexible. It is fast, efficient, and can handle really huge numbers of messages with ease.

ZeroMQ is used in some impressively large deployments, at my workplace we use ZeroMQ to handle over 200k messages per second. We have found ZeroMQ to scale effortlessly, the library is well engineered and optimised (no memory leaks, good performance), and has proven itself to work very well regardless of what we throw at it.

In ZeroMQ, publish / subscribe is done on user-defined topics, which dictate what data is sent to which connected clients. So if I have 10 clients connected to my publish socket, and 9 subscribe to a topic called "A", and 1 client subscribes to a topic "B", and I send a message with the topic of "B", then only the client subscribed to the topic "B" will be sent the message. ZeroMQ performs the filtering of pub/sub messages at both the point of transmission (to avoid wasting bandwidth), and at the point of receipt (to avoid possible race conditions when a topic is unsubscribed). It is also possible to subscribe to more than one topic.

To implement the mesh messaging system you describe, I recommend creating two sockets on each node in the cluster, one for receiving messages from all the other nodes, and one for sending messages to all the other nodes. If you do not need topics, then a subscription to the "*" topic will allow that client to receive all messages.

7
  • Thanks for providing very detailed information and suggesting a concrete solution. When I use a message bus instead (e.g. with 4 nodes, node 1 connects to node 2&3&4, node 2 to 3&4, node 3 to 4), what would be different to Pub/Sub, if I do not need topic filtering? Do ZeroMQ and Nanomsg handle Pub/Sub equivalently, apart from frontend implementation?
    – jellysheep
    Mar 29, 2014 at 22:04
  • ZeroMQ and nanomsg handle pub/sub and bus equivalently, bus is just a convenience for the client. Mar 30, 2014 at 9:13
  • Thanks for the explanation. I tried out Pub/Sub using nanomsg and two sockets per node. That works fine, so I will use it. I will upvote your answer as soon as I am allowed to.
    – jellysheep
    Mar 30, 2014 at 9:25
  • This nanomsg blog entry about bus states "Note that as with any other broadcasting pattern, the message transfer is not reliable." So is Pub/Sub messaging in nanomsg/ZeroMQ not reliable either? I need reliable communication for my application.
    – jellysheep
    Apr 1, 2014 at 15:26
  • Here's a more in-depth explanation of the problem (and potential solutions): zguide.zeromq.org/php:chapter5#Pros-and-Cons-of-Pub-Sub Apr 1, 2014 at 19:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.