I have many celery nodes running on different Linux boxes. A simplified cluster looks like: Celery cluster with 3 nodes

I have a long-running taskA which downloads a (huge) file from a remote file-server, and I want to trigger taskB once the taskA finishes its job on the same machine because that is where the taskA downloaded the file which is basically an input to the taskB. I know Celery is supposed to be distributed, but in some cases, like this one having a true distributed solution would be expensive.

Notice in the diagram that taskA on nodeA triggers taskB on the same node, while taskA that runs on nodeB triggers taskB on nodeB (same node that runs the "parent" task).

apply() does not work because I want to queue the taskB() to run when there is a spare worker process (on the same node!) for it.

I started exploring possibility of having a unique queue on each node (so nodeA has queueA, nodeB has queueB, etc), but I do not know how to get available queue names from taskA so I can queue task on the queue on the same node. (This would probably solve my problem.)

I was trying to find solution in Celery documentation, but I could not find anything that helps me. Any idea will greatly appreciated.

up vote 1 down vote accepted
+100
  1. You could trigger taskB from in withing taskA directly using the apply method.

    This method will execute the task locally without sending to the broker.

    Note that this is equivalent to have only one task with a more advanced retry policy.

  2. First task could be published to generic queue while you could schedule second task from within taskA running on serverABC and passing a specific queue name like "FILE_PROCESSING_SERVER_ABC" where only the worker running on serverABC is configured as consumer. In this way the taskA will run on whatever server picks it up while B will be forced tu run on the specific queue belonging only to the worker where taskA was running. More on manual routing here.

    EDIT

    As suggested by Aaron answer you could use an environment variable where you could store the queue for the machine X and retrieve it from within the taskA when scheduling task B.

  • Mauro, is it going to run the taskB in a separate worker process? (That is what I need - to basically queue the task on local server) – DejanLekic May 9 '16 at 15:15
  • I edited the answer to after your clarification. – Mauro Rocco May 9 '16 at 15:38
  • Mauro, if you refine your answer, I am willing to reward it with the bounty. You deserve it! – DejanLekic May 11 '16 at 17:03
  • Hi Dejan, what you want me to add more :-) ? point 2 is not what you was looking for ? You want more details ? Thanks for clarify it. – Mauro Rocco May 12 '16 at 5:45
  • For an example, your answer does not say how a task (taskA) knows where to queue taskB? After discussion on IRC, I have decided to set HOST_QUEUE variable in my celery config file for each server, so tasks can use this value when they want to queue new tasks on the same server they are running on. – DejanLekic May 12 '16 at 8:22

If you're OK with a queue per node. I'd use a per host setting to hold the name of that queue. That way TaskA can look up the correct queue to run taskB on.

  • That is most likely how I am going to solve this problem, Aaron. Thanks for the help here and on IRC! – DejanLekic May 11 '16 at 15:48

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.