22

I am pretty new to celery and django in general so please excuse my lack of knowledge. I am trying to run a test to do some calculations and wait for the test to finish so that I can make sure it is done for the correct answers.

Here is what i have:

In app/tests.py

from tasks import *


c = calculate.apply_async(args=[1])

# wait until the task is done
while not calculate.AsyncResult(c.id).status == "SUCCESS":
    print c.state
    pass

in app/tasks.py

from celery import shared_task

@shared_task
def calculate(proj_id):

    #some calculations followed by a save of the object

The state never changes from pending even though in the celery log it says that the task was completed successfully

[2014-06-10 17:55:11,417: INFO/MainProcess] Received task: app.tasks.calculate[1f11e7ab-0add-42df-beac-3d94c6868aac]
[2014-06-10 17:55:11,505: INFO/MainProcess] Task app.tasks.calculate[1f11e7ab-0add-42df-beac-3d94c6868aac] succeeded in 0.0864518239978s: None

I have also put CELERY_IGNORE_RESULT = False in the mainapp/settings.py, but this did not seem to do anything.

3
  • Test code you have is correct, problem has to be somewhere in the settings or the overall setup I presume, where exactly I can't tell without more info.
    – lehins
    Jun 11, 2014 at 10:23
  • 1
    @AlexeyKuleshevich Here are the settings I am aware of, if I miss something please let me know. celery.py and _init.py follow the tutorial conventions and here is what I have in mainapp/settings.py: BROKER_URL = "amqp://the ipadress" CELERY_RESULT_BACKEND='djcelery.backends.database:DatabaseBackend' CELERY_ACCEPT_CONTENT = ['json','pickle','msgpack','yaml'] CELERY_IGNORE_RESULT = False Jun 11, 2014 at 19:15
  • @AlexeyKuleshevich As you have answered my original question I have moved the other question to a more appropriate place in another post. If you wish to continue helping you can find it here Jun 17, 2014 at 19:31

3 Answers 3

15

Straight from doc: Result backend does not work or tasks are always in PENDING state.

All tasks are PENDING by default, so the state would have been better named "unknown". Celery does not update any state when a task is sent, and any task with no history is assumed to be pending (you know the task id after all).

  1. Make sure that the task does not have ignore_result enabled.

    Enabling this option will force the worker to skip updating states.

  2. Make sure the CELERY_IGNORE_RESULT setting is not enabled.

  3. Make sure that you do not have any old workers still running.

    It’s easy to start multiple workers by accident, so make sure that the previous worker is properly shutdown before you start a new one.

    An old worker that is not configured with the expected result backend may be running and is hijacking the tasks.

    The –pidfile argument can be set to an absolute path to make sure this doesn’t happen.

  4. Make sure the client is configured with the right backend.

If for some reason the client is configured to use a different backend than the worker, you will not be able to receive the result, so make sure the backend is correct by inspecting it:

>>> result = task.delay(…)
>>> print(result.backend)
3
  • Cool, I am glad they finally added that to documentation for the new version.
    – lehins
    Oct 23, 2015 at 18:33
  • I switched the backend from "rpc://" to database, and then the task state is no longer "PENDING".
    – zhanxw
    Dec 30, 2015 at 19:50
  • Is there a reason why Tast States dont work with multiple workers? can I circumvent this? Jul 18, 2018 at 11:48
8

So, your settings are wrong. :) You also need to setup a broker for celery to work.

First of all, djcelery is deprecated, everything is included in celery for it to work with django.

Second of all, do not set all content to be accepted, it can be a potential security risk. Use pickle only in case that simple json is not enough (let's say you pass functions or objects as arguments to tasks, or return from tasks)

So my guess is, you are just trying celery out, that's why you are trying to use database backend, which is fine, but for production use I would recommend using RabbitMQ.

In any case, give it a try with those settings:

BROKER_URL = 'django://'
INSTALLED_APPS = (
    ...
    'kombu.transport.django', 
    ...
)
CELERY_RESULT_BACKEND = 'db+scheme://user:password@host:port/dbname' 
CELERY_ACCEPT_CONTENT = ['json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_IGNORE_RESULT = False # this is less important

then run python manage.py syncdb

Just to let you know, I have not used database as a broker or result backend, hence setup might be incomplete, or even incorrect, but give it a try anyways.

more CELERY_RESULT_BACKEND setting for database examples

In case you want to setup RabbitMQ as a broker backend, which I would recommend and I know for sure it will work:

if on ubuntu run:

sudo apt-get install rabbitmq-server
sudo rabbitmqctl add_user <username> <password>
sudo rabbitmqctl add_vhost <vhost, use project name for example>
sudo rabbitmqctl set_permissions -p <vhost> <username"> ".*" ".*" ".*"

Then configure celery in settings.py:

BROKER_URL = 'amqp://<user>:<password>@localhost:5672/<vhost>'
CELERY_TIMEZONE = TIME_ZONE
CELERY_RESULT_BACKEND = 'amqp'
# thats where celery will store scheduled tasks in case you restart the broker:
CELERYD_STATE_DB = "/full/path/data/celery_worker_state" 
CELERY_ACCEPT_CONTENT = ['json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'

Let me know how it goes.

7
  • it no longer pends forever, but it is now failing. The error is telling me that I need task args to be a list or a tuple though when I make it a list or tuple it tells me it needs to be an int(). So very confusing. Jun 13, 2014 at 16:03
  • Well, that's good, that means celery is working. Post your task code and error, I'll tell you where the problem is coming from.
    – lehins
    Jun 14, 2014 at 5:41
  • Also you can use c = calculate.delay(1, foo=2) instead of c = calculate.apply_async(args=[1], kwargs={'foo':2}) if you just want to put it on queue instead of scheduling at specific time or need to specify any custom behavior.
    – lehins
    Jun 14, 2014 at 5:44
  • I can't post the code in its entirety but I can post some of it. @shared_task def calculate(project_id): project = Project.objects.get(project=project_id) for part in project.part_set.all(): partCalculate(part.id) p = Part.objects.get(id=part.id) #add various things to the project project.save() def partCalculate(part_id): part = Part.objects.get(id=part_id) #initalize various lists for p in part.subpart_set.all(): # put stuff in lists if any(of the values above): #some statistics stuff part.save() Jun 16, 2014 at 17:08
  • [INFO/MainProcess] Received task: measuring.tasks.calculate[id] [ERROR/MainProcess] Task measuring.tasks.calculate[id] raised unexpected: ValueError('task args must be a list or tuple',) Traceback: File "/celery/app/trace.py", line 240, in trace_task R = retval = fun(*args, **kwargs) File "/celery/app/trace.py", line 437, in __protected_call__ return self.run(*args, **kwargs) File "/calculate/tasks.py", line 28, in calculate p = Part.objects.get(id=part.id) File "/python2.7/site-packages/celery/app/task.py", line 555, in apply_async Jun 16, 2014 at 17:21
0

If you are using old django-celery and RabbitMQ as result backend, then these settings may help:

# Mostly, all settings are the same as in other answers

CELERY_RESULT_BACKEND = 'rpc://'
CELERY_ACCEPT_CONTENT = ['json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_IGNORE_RESULT = False

# This line is what I needed
CELERY_TRACK_STARTED = True

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.