5

I'm trying to run Celery as service in Ubuntu 18.04, using Django 2.1.1 and Celery 4.1.1 that is installed in env and Celery 4.1.0 installed systemwide. I'm going forward with this tutorial to run Celery as service. Here is my Django's project tree:

    hamclassy-backend
                 ├── apps
                 ├── env
                 │   └── bin
                 │        └── celery
                 ├── hamclassy
                 │   ├── celery.py
                 │   ├── __init__.py
                 │   ├── settings-dev.py
                 │   ├── settings.py
                 │   └── urls.py
                 ├── wsgi.py
                 ├── manage.py
                 └── media

Here is celery.py :

from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'hamclassy.settings')

app = Celery('hamclassy')

app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()

Here is /etc/systemd/system/celery.service:

[Unit]
Description=Celery Service
After=network.target

[Service]
Type=forking
User=classgram
Group=www-data
EnvironmentFile=/etc/conf.d/celery
WorkingDirectory=/home/classgram/www/hamclassy-backend
ExecStart=/bin/sh -c '${CELERY_BIN} multi start ${CELERYD_NODES} \
  -A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \
  --logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}'
ExecStop=/bin/sh -c '${CELERY_BIN} multi stopwait ${CELERYD_NODES} \
  --pidfile=${CELERYD_PID_FILE}'
ExecReload=/bin/sh -c '${CELERY_BIN} multi restart ${CELERYD_NODES} \
  -A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \
  --logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}'

[Install]
WantedBy=multi-user.target


and /etc/conf.d/celery:

CELERYD_NODES="celery-worker"

CELERY_BIN="/home/classgram/www/env/bin/celery"

# App instance to use
CELERY_APP="hamclassy"


CELERYD_MULTI="multi"
# Extra command-line arguments to the worker
CELERYD_OPTS="--time-limit=300 --concurrency=8"

# %n will be replaced with the first part of the nodename.
CELERYD_LOG_FILE="/var/log/celery/%n%I.log"
CELERYD_PID_FILE="/var/run/celery/%n.pid"
CELERYD_LOG_LEVEL="INFO"

When I run service with systemctl start celery.service the following error appears:

Job for celery.service failed because the control process exited with error code. See "systemctl status celery.service" and "journalctl -xe" for detail

When I run sudo journalctl -b -u celery the below log appears:

Starting Celery Service... celery.service: Control process exited, code=exited status=200 celery.service: Failed with result 'exit-code'. Failed to start Celery Service.

For more information:
1) When I use groups classgram this appears: www-data sudo.
2) I can run Celery with celery -A hamclassy worker -l info after activating environment very well.

Thank you

2
  • 2
    What I suggest you to do is to log in as user specified in your systemd service file, source the environment file, and run the Celery worker exactly like systemd would run it and see why it fails.
    – DejanLekic
    Jul 30, 2019 at 8:55
  • thank you for reply. the error is about module ModuleNotFoundError: No module named 'hamclassy' . when i run celery -A hamclassy worker -l info without activation virtual environment this error too so i think service can't run because i can't activate virtual environment before this command. how i can do this?
    – mehdi
    Jul 30, 2019 at 10:40

1 Answer 1

7

OK, now that we know what was the error (from your comment), the following few steps will most likely solve your problem:

  1. Create a virtual environment for your Celery: python3 -m venv /home/classgram/venv

  2. Your enviroment file either has to export PYTHONPATH=/home/classgram/www/hamclassy-backend, which is the simplest solution, good for testing, and development. However, for production I recommend you to build a Python package (wheel), and install it in your virtual environment.

  3. Modify your systemd service file (and possibly the environment file too) so that you simply execute Celery directly from the Python virtual environment we created above. It should run Celery this way (or similar): /home/classgram/venv/bin/celery multi start ${CELERYD_NODES} -A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}. Sure, you can make it shorter by setting CELERY_BIN to /home/classgram/venv/bin/celery.

2
  • god bless you master. you save my several days. that is working now. i have env myself for django so not need to create new one but i change systemd as you told me and its working now.
    – mehdi
    Jul 30, 2019 at 13:13
  • I gave up on using celery multi, and now I run single worker in detached mode.
    – DejanLekic
    Jan 10, 2022 at 10:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.