I have a question about retrying celery tasks within a chunk. My python version is 2.7.9, and my celery version is 3.1.18
When an individual task inside a chunk fails, it doesn't seem to be retried, and instead the whole chunk fails together.
First, the task that will fail sometimes:
@app.task(bind=True, default_retry_delay=30, max_retries=5)
def sometimes_add(self, x, y):
try:
if random.randint(0, 10) < 8:
return x + y
else:
raise Exception("I'm failing!")
except Exception as e:
raise self.sometimes_add.retry(exc=e)
And the task called 100 times, and chunked into 5 pieces and therefore 20 tasks:
sometimes_add.chunks(zip(range(100), range(100)), 5).apply_async()
I would expect the failed task within the chunk to be retried once it has failed, but instead no tasks seem to be retried at all, and the whole chunk of 5 tasks fails as a unit.
I'm very new to Celery, so it definitely could be that I am missing something rather fundamental to the best way to use chunking of tasks. If so I'd appreciate any feedback. Thanks so much!