I have a question about retrying celery tasks within a chunk. My python version is 2.7.9, and my celery version is 3.1.18

When an individual task inside a chunk fails, it doesn't seem to be retried, and instead the whole chunk fails together.

First, the task that will fail sometimes:

@app.task(bind=True, default_retry_delay=30, max_retries=5)
def sometimes_add(self, x, y):
    try:
        if random.randint(0, 10) < 8:
            return x + y
        else:
            raise Exception("I'm failing!")
    except Exception as e:
        raise self.sometimes_add.retry(exc=e)

And the task called 100 times, and chunked into 5 pieces and therefore 20 tasks:

sometimes_add.chunks(zip(range(100), range(100)), 5).apply_async()

I would expect the failed task within the chunk to be retried once it has failed, but instead no tasks seem to be retried at all, and the whole chunk of 5 tasks fails as a unit.

I'm very new to Celery, so it definitely could be that I am missing something rather fundamental to the best way to use chunking of tasks. If so I'd appreciate any feedback. Thanks so much!

Do you have a results backend set up? From the docs:

Tasks used within a chord must not ignore their results. In practice this means that you must enable a CELERY_RESULT_BACKEND in order to use chords. Additionally, if CELERY_IGNORE_RESULT is set to True in your configuration, be sure that the individual tasks to be used within the chord are defined with ignore_result=False. This applies to both Task subclasses and decorated tasks.

  • Ah, thanks for your comment! I do indeed have a results backend setup (amqp). Also, it looks like this particular part from the docs is referring to a chord, not to chunks. Are they somehow related? – erinspace Jun 8 '15 at 17:16

Looks like there is a syntactical error here:

sometimes_add.chunks(zip(range(100), range(100)), 5).apply_async()

You are closing the zip() function before adding the third parameter -- the 5. I think it should look like this:

sometimes_add.chunks(zip(range(100), range(100), 5)).apply_async()

  • No the chunk function takes 2 parameters, an iterable of 'work' and the number of chunks to divide it into. – skolsuper Jun 7 '15 at 1:56
  • The example here seems to disagree... celery.readthedocs.org/en/latest/userguide/canvas.html#chunks – samskeller Jun 7 '15 at 2:12
  • You are correct that 2 of the examples there disagree, but I think they are typos. The documentation for chunks says the signature is (it, n): celery.readthedocs.org/en/latest/reference/… . Maybe one of us should file an issue with celery to fix the docs? – skolsuper Jun 7 '15 at 2:18
  • To generate this example, I simply replaced the add task with the sometimes_add task, taken directly from the docs on celery. It seems to run just fine locally - some of the chunks are successful. I was mainly just curious about how to retry tasks that are chunked. Thanks for the comment! – erinspace Jun 8 '15 at 17:20

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.