20

I started using ZeroMQ this week, and when using the Request-Response pattern I am not sure how to have a worker safely "hang up" and close his socket without possibly dropping a message and causing the customer who sent that message to never get a response. Imagine a worker written in Python who looks something like this:

import zmq
c = zmq.Context()
s = c.socket(zmq.REP)
s.connect('tcp://127.0.0.1:9999')
while i in range(8):
    s.recv()
    s.send('reply')
s.close()

I have been doing experiments and have found that a customer at 127.0.0.1:9999 of socket type zmq.REQ who makes a fair-queued request just might have the misfortune of having the fair-queuing algorithm choose the above worker right after the worker has done its last send() but before it runs the following close() method. In that case, it seems that the request is received and buffered by the ØMQ stack in the worker process, and that the request is then lost when close() throws out everything associated with the socket.

How can a worker detach "safely" — is there any way to signal "I don't want messages anymore", then (a) loop over any final messages that have arrived during transmission of the signal, (b) generate their replies, and then (c) execute close() with the guarantee that no messages are being thrown away?

Edit: I suppose the raw state that I would want to enter is a "half-closed" state, where no further requests could be received — and the sender would know that — but where the return path is still open so that I can check my incoming buffer for one last arrived message and respond to it if there is one sitting in the buffer.

Edit: In response to a good question, corrected the description to make the number of waiting messages plural, as there could be many connections waiting on replies.

2
  • 2
    I found this thread, which is related-but-not-really: mail-archive.com/[email protected]/msg02338.html . Have you made any progress on this?
    – Jack Kelly
    Sep 23, 2010 at 3:37
  • No, I have not yet made further progress, and it appears that ZeroMQ expertise is a bit thinner on the ground than I had expected. Thanks for the link — it seems to involve message safety in the other direction, when a message is outbound, but brings up many of the same issues. Thanks! Sep 23, 2010 at 12:01

6 Answers 6

11

You seem to think that you are trying to avoid a “simple” race condition such as in

... = zmq_recv(fd);
do_something();
zmq_send(fd, answer);
/* Let's hope a new request does not arrive just now, please close it quickly! */
zmq_close(fd);

but I think the problem is that fair queuing (round-robin) makes things even more difficult: you might already even have several queued requests on your worker. The sender will not wait for your worker to be free before sending a new request if it is its turn to receive one, so at the time you call zmq_send other requests might be waiting already.

In fact, it looks like you might have selected the wrong data direction. Instead of having a requests pool send requests to your workers (even when you would prefer not to receive new ones), you might want to have your workers fetch a new request from a requests queue, take care of it, then send the answer.

Of course, it means using XREP/XREQ, but I think it is worth it.

Edit: I wrote some code implementing the other direction to explain what I mean.

2
  • 1
    You are quite correct; in a more complex example than the one originally described in my message, where there might be many clients, there might be many waiting messages. I have updated the text to reflect this possibility. Thanks! Dec 8, 2010 at 13:40
  • I updated my answer with some new code I wrote to show how to deal with your issue. Dec 10, 2010 at 7:33
3

I think the problem is that your messaging architecture is wrong. Your workers should use a REQ socket to send a request for work and that way there is only ever one job queued at the worker. Then to acknowledge completion of the work, you could either use another REQ request that doubles as ack for the previous job and request for a new one, or you could have a second control socket.

Some people do this using PUB/SUB for the control so that each worker publishes acks and the master subscribes to them.

You have to remember that with ZeroMQ there are 0 message queues. None at all! Just messages buffered in either the sender or receiver depending on settings like High Water Mark, and type of socket. If you really do need message queues then you need to write a broker app to handle that, or simply switch to AMQP where all communication is through a 3rd party broker.

1
  • Good explanation — thanks for a bit of philosophy about why I was trying to do things the wrong way. I was thinking that my worker should be sent work, and then that there would be some magic way for it to say “stop” — but in fact the solution is for the worker to start the conversation each time, which puts it in control instead of leaving the sender in control. Thanks! Jul 31, 2011 at 12:13
2
+100

I've been thinking about this as well. You may want to implement a CLOSE message which notifies the customer that the worker is going away. You could then have the worker drain for a period of time before shutting down. Not ideal, of course, but might be workable.

2
  • Yes, that would be possible. But my hope was to use a REQ customer that did not even know how many servers I had set up to load-balance his requests. Having to move to XREQ and implement my own subscription and de-subscription actions is something I hope to avoid! Oct 28, 2010 at 18:00
  • Not a great solution, but it doesn't seem like any real solutions exist at this time :S.
    – Jack Kelly
    Nov 2, 2010 at 0:30
0

There is a conflict of interest between sending requests as rapidly as possible to workers, and getting reliability in case a worked crashes or dies. There is an entire section of the ZeroMQ Guide that explains different answers to this question of reliability. Read that, it'll help a lot.

tl;dr workers can/will crash and clients need a resend functionality. The Guide provides reusable code for that, in many languages.

1
  • This must be a newer part of the Guide than existed when I wrote this question! I will take a look and see what their advice now is. Thanks! Jan 23, 2012 at 4:33
-1

Wouldn't the simplest solution be to have the customer timeout when waiting for the reply and then retry if no reply is received?

3
  • How would you specify a timeout to the ØMQ recv() function? I think it blocks forever without any chance of escaping. Nov 22, 2010 at 4:30
  • @Brandon: You could do this with zmq_poll(). Not having to write out this sort of time-out-and-recover code is meant to be one of ZMQ's strengths, though. :(
    – Jack Kelly
    Nov 22, 2010 at 5:12
  • Retries are a problem because they duplicate work in the case where a particular job simply takes longer than normal. And note that in these cases, it is the longer jobs that get submitted several times, increasing the load offered to the servers. The issue here is that unlike TCP, which takes an unreliable network and lets you pretty much forget about the issue of whether packets are ever lost, ØMQ takes TCP and makes you start worrying about lost messages again. In this REQ-REP use case it seems to be a big step backwards in application complexity. Nov 23, 2010 at 7:00
-3

Try sleeping before the call to close. This is fixed in 2.1 but not in 2.0 yet.

3
  • 6
    Synchronisation-by-sleeping scares me. Do you know how this was fixed in 2.1? Did they add an option for half-closed sockets, or does closing tell the sender of unprocessed messages that they need to retransmit elsewhere?
    – Jack Kelly
    Oct 1, 2010 at 22:28
  • Trey, I see that in 2.1 close() will not destroy queued outgoing messages. But I see nothing about incoming messages, at least not on the little summary page I am looking at. Could you point us to the changelog or docs at the right place? Oct 2, 2010 at 12:54
  • Sorry guys I have the same info you do. I'm not a contributor to 0mq. As far as I know 2.1 just flushes on close instead of immediately closing.
    – Trey Stout
    Oct 4, 2010 at 16:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.