4

I'm implementing a performance heavy two-party protocol in C++14 utilising multithreading and am currently using ZeroMQ as a network layer.

The application has the following simple architecture:

  • One main server-role,
  • One main client-role,
  • Both server and client spawn a fixed number n of threads
  • All n parallel concurrent thread-pairs execute some performance and communication heavy mutual, but exclusive, protocol exchange, i.e. these run in n fixed pairs and should not mix / interchange any data but with the pairwise fixed-opponent.

My current design uses a single ZeroMQ Context()-instance on both server and client, that is shared between all n-local threads and each respective client/server thread-pair creates a ZMQ_PAIR socket ( I just increment the port# ) on the local, shared, context for communication.

My question

Is there is a smarter or more efficient way of doing this?

i.e.: is there a natural way of using ROUTERS and DEALERS that might increase performance?

I do not have much experience with socket programming and with my approach the number of sockets scales directly with n ( a number of client-server thread-pairs ). This might go to the couple of thousands and I'm unsure if this is a problem or not.

I have control of both the server and client machines and source code and I have no outer restrictions that I need to worry about. All I care about is performance.

I've looked through all the patterns here, but I cannot find anyone that matches the case where the client-server pairs are fixed, i.e. I cannot use load-balancing and such.

2
  • Some questions: 1) Are there any latency requirements? 2) Can you guarantee that you have thousands of adjacent free ports? (or in other words how much control do you have over the machines this is running on?) 3) How will you know when you need to spawn more threads, is it a run time decision?
    – David
    May 30, 2016 at 18:40
  • Hi David, thanks for taking an interest. 1) I don't understand what you mean exactly, but latency is definitely a priority. 2) Yes, I can guarantee that 3) Yes, it depends on the input to the program how many threads are spawned
    – Fulnir
    May 31, 2016 at 20:45

1 Answer 1

2

Happy man!

ZeroMQ is a lovely and powerful tool for highly scaleable, low-overheads, Formal Communication ( behavioural, yes emulating some sort of the peers mutual behaviour "One Asks, the other Replies" et al ) Patterns.

Your pattern is quite simple, behaviourally-unrestricted and ZMQ_PAIR may serve well for this.


Performance

There ought be some more details on quantitative nature of this attribute.

  • a process-to-process latency [us]
  • a memory-footprint of a System-under-Test (SuT) architecture [MB]
  • a peak-amount of data-flow a SuT can handle [MB/s]

Performance Tips ( if quantitatively supported by observed performance data )

  • may increase I/O-performance by increasing Context( nIOthreads ) on instantiation

  • may fine-tune I/O-performance by hard-mapping individual thread# -> Context.IO-thread# which is helpfull for both distributed workload and allows one to keep "separate" localhost IOthread(s) free / ready for higher-priority signalling and others such needs.

  • shall setup application-specific ToS-labeling of prioritised types of traffic, so as to allow advanced processing on the network-layer alongside the route-segments between the client and server

  • if memory-footprint hurts ( ZeroMQ is not Zero-copy on TCP-protocol handling at operating-system kernel level ) one may try to move to a younger sister of ZeroMQ -- authored by Martin SUSTRIK, a co-father of ZeroMQ -- a POSIX compliant nanomsg with similar motivation and attractive performance figures. Worth to know about, at least.


Could ROUTER or DEALER increase an overall performance?

No, could not. Having in mind your stated architecture ( declared to be communication heavy ), other, even more sophisticated Scaleable Formal Communication Patterns behaviours that suit some other needs, do not add any performance benefit, but on the contrary, would cost you additional processing overheads without delivering any justifying improvement.

While your Formal Communication remains as defined, no additional bells and whistles are needed.

One point may be noted on ZMQ_PAIR archetype, some sources quote this to be rather an experimental archetype. If your gut sense feeling does not make you, besides SuT-testing observations, happy to live with this, do not mind a slight re-engineering step, that will keep you with all the freedom of un-pre-scribed Formal Communication Pattern behaviour, while having "non"-experimental pipes under the hood -- just replace the solo ZMQ_PAIR with a pair of ZMQ_PUSH + ZMQ_PULL and use messages with just a one-way ticket. Having the stated full-control of the SuT-design and implementation, this would be all within your competence.


How fast could I go?

There are some benchmark test records published for both the ZeroMQ or nanomsg performance / latency envelopes for un-loaded network transports across the traffic-free route-segments ( sure ).

If your SuT-design strives to go even faster -- say under some 800 ns end-to-end, there are other means to achieve this, but your design will have to follow other distributed computing strategy than a message-based data exchange and your project budget will have to adjust for additional expenditure for necessary ultra-low-latency hardware infrastructure.

It may surprise, but definitely well doable and pretty attractive for systems, where hundreds of nanoseconds are a must-have target within a Colocation Data Centre.

1
  • 1
    Thank you for your detailed answer! I ended up going to the Pull/Push pattern following you recommendations. No difference in my timings though. But it's good to know that I didn't miss something obvious
    – Fulnir
    Jun 1, 2016 at 16:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.