How to run celery worker on Windows without creating Windows Service? Is there any analogy to $ celery -A your_application worker
?
13 Answers
Celery 4.0+
does not officially support window already. But it still works on window for some development/test purpose.
Use eventlet
instead as below:
pip install eventlet
celery -A <module> worker -l info -P eventlet
It works for me on window 10
+ celery 4.1
+ python 3
.
This solution solved the following exception:
[2017-11-16 21:19:46,938: ERROR/MainProcess] Task handler raised error: ValueError('need more than 0 values to unpack',)
Traceback (most recent call last):
File "c:\users\wchen8\work\venv\weinsta\lib\site-packages\billiard\pool.py", line 358, in workloop
result = (True, prepare_result(fun(*args, **kwargs)))
File "c:\users\wchen8\work\venv\weinsta\lib\site-packages\celery\app\trace.py", line 525, in _fast_trace_task
tasks, accept, hostname = _loc
ValueError: need more than 0 values to unpack
===== update 2018-11 =====
Eventlet has an issue on subprocess.CalledProcessError:
https://github.com/celery/celery/issues/4063
https://github.com/eventlet/eventlet/issues/357
https://github.com/eventlet/eventlet/issues/413
So try gevent
instead.
pip install gevent
celery -A <module> worker -l info -P gevent
This works for me on window 10
+ celery 4.2
+ python 3.6
-
1this also works with gevent and solo concurrency pool implementations (tested with Windows 10 and Celery 4.2.1): See github.com/ZoomerAnalytics/celery-4-windows for example code Commented Sep 3, 2018 at 16:23
-
2nice! works (sort of) under windows server 12, python 3.7.1, celery 4.2.1, gevent 1.3.7. The only issue for me is monitoring of these workers.
celery *** inspect active
works initially after workers are turned on, and then after about 5 minutes it becomes unresponsive, not sure why yet (this was not an issue under python 3.3.5, celery 4.1.0 eventlet 0.22.1). Commented Jan 3, 2019 at 17:46 -
1for future readers, the issue that I am encountering is discussed here: github.com/celery/celery/issues/4817 seems like the work around is setting
CELERY_BROKER_HEARTBEAT = 0
. Commented Jan 3, 2019 at 18:15 -
Does this work for concurrent requests scenario to celery worker? Commented Sep 17, 2020 at 19:01
yes:
celery -A your_application -l info
also note Celery have dropped support for Windows(since v4), so best to
pip install celery==3.1.25
3.1.25 was the last version that works on windows(just tested on my win10 machine). Didn't need to downgrade flower(browser monitor for celery) though.
See also the FAQ for Windows
-
-
using 3.1.25 i got ValueError: not enough values to unpack (expected 3, got 0)– pelosCommented Jul 7, 2020 at 1:21
-
It works for celery version 4.0 and above as well for windows OS, you just have to enable multiprocessing environment variable before creating celery instance. Commented Sep 17, 2020 at 19:03
-
Cannot install version 3.1.25, gives an error:
error in anyjson setup command: use_2to3 is invalid.
during installation Commented Jun 3, 2022 at 14:08
Compile Celery with --pool=solo argument.
Example:
celery -A your-application worker -l info --pool=solo
-
3
-
2@Fusion Basically python only allows one thread to use the interpreter at a time. There is a package that allows python to sidestep this global interpreter lock, called billiard. Celery relies on billiard when it uses the default pool type of --pool=prefork, which lets the workers fork (copy entire state, variables, etc) to utilize different threads. Windows doesn't allow process forking, it only allows process spawning, which means the worker has to start a new python interpreter and import pkgs, etc. So you have to use a solo pool and run the task in the worker's own process. Commented May 29 at 1:05
There are two workarounds to make Celery 4 work on Windows:
- use eventlet, gevent or solo concurrency pool (if your tasks as I/O and not CPU-bound)
- set the environment variable FORKED_BY_MULTIPROCESSING=1 (this is what actually causes the underlying billiard package to to fail under Windows since version 4)
See https://www.distributedpython.com/2018/08/21/celery-4-windows for more details
-
What are the ramifications for selecting the second option? I still can't successfully
revoke()
a task after its been executed...– ShmackCommented Oct 11, 2021 at 3:17
I have run celery task using RabbitMQ server. RabbitMq is better and simple than redis broker
while running celery use this command "celery -A project-name worker --pool=solo -l info" and avoid this command "celery -A project-name worker --loglevel info"
-
For -P solo option, if the task takes longer time like around 2 hours, connection reset error will occur on windows. Commented Sep 17, 2020 at 19:04
It's done the same way as in Linux. Changing directory to module containing celery task and calling "c:\python\python" -m celery -A module.celery worker
worked well.
-
1There are typos you need to fix. More so, you don't need to call
python
before using celery. Doingcelery -A tasks worker -l info
should be sufficient if you have yourPYTHONPATH
correctly set. Commented May 19, 2016 at 9:05 -
-
1pyton (
python
), woker (worker
). I just thought it would better serve those who will have the same question Commented May 19, 2016 at 10:22 -
3@MosesKoledoye Isn't
PYTHONPATH
just for theimport
statements? How could the Celery binary be found without havingPython\Scripts
directory inPATH
or callingpython -m
? Commented Sep 26, 2016 at 15:20
Celery 4.0+ does not officially support window already. But it still works on window for some development/test purpose. you can use any of this:
celery worker --app=app.app --pool=eventlet --loglevel=INFO
celery worker --app=app.app --pool=gevent --loglevel=INFO
celery worker --app=app.app --pool=solo --loglevel=INFO
or using other format:
celery -A <app> worker --loglevel=info -P eventlet
celery -A <app> worker --loglevel=info -P gevent
celery -A <app> worker --loglevel=info -P solo
if you have get context
error then upgrade gevent 20.6.2 and eventlet to 0.26.1 or use solo
https://www.distributedpython.com/2018/08/21/celery-4-windows/
-
if multiprocessing environment variable is enabled, no need to specify pool to eventlet or gevent or solo. It will work with default pool on windows as well for celery version 4.0 and above. Commented Sep 17, 2020 at 19:06
You can run celery on windows without an extra library by using threads
celery -A your_application worker -P threads
-
2This disables multiple processes. All workers would run within the same process. It is good if you want a quick bit of concurrency but your tasks would hinder eachother if they require a lot of CPU.– nurettinCommented Oct 8, 2021 at 12:50
You can still use celery 4 0+ with Windows 10+ Just use this command "celery -A projet worker - -pool=solo - l info" instead of "celery - A project worker -l info
-
For -P solo option, if the task takes longer time like around 2 hours, connection reset error will occur on windows Commented Sep 17, 2020 at 19:04
For using celery 4.4 on windows, I think I can answer this question.
For celery version 4.0 and above, first set following environment variable in python code before creation of celery instance.
os.environ.setdefault('FORKED_BY_MULTIPROCESSING', '1')
Then run celery worker command with default pool option.
celery worker -A <celery_file> -l info
This will run celery worker concurrently with multiple child processes.
Note: When you run celery with gevent or eventlet pool, it will work but it won't run concurrent processes on windows.
-
What are the ramifications for selecting the second option? I still can't successfully
revoke()
a task after its been executed...– ShmackCommented Oct 11, 2021 at 3:17
To run celery on windows for development. you have two methods.
- Use Windows as Host Machine.
- Install Redis Server from https://github.com/dmajkic/redis/downloads
- Run redis_server.exe
- Install Celery using
Pip install Celery
. - Run Celery commands.
Limitations:
- This method is only supported for Python 3.6 or lower.
- Celery 4 does not provide support for windows.
- will be using Old Redis-server Version.
- Hard to manage
4. Using WSL for running Celery and Redis-server
- Install WSL using this guide https://www.windowscentral.com/install-windows-subsystem-linux-windows-10.
- Install redis-server using
sudo apt install redis-server
on the WSL terminal. - Install Celery and Execute all celery-related commands on WSL.
WSL would be a friend for you on windows if you have already familiar with Linux. With few limitations including support of Docker on WSL. you can perform most of the Development tasks using WSL.
Celery is testing it with windows in it's CI again from version 5.x and you can try using it with windows if any issue occurs try set following environment variable in python code before creation of celery instance
os.environ.setdefault('FORKED_BY_MULTIPROCESSING', '1')
But as far as I know celery now officially support running on windows for development purpose.
-
How is this answer different than this one, where this suggestion was already made? stackoverflow.com/a/63944021/3250829– rayryengCommented Mar 30, 2023 at 5:31
-
that doesn't mention that celery restarted support for windows again in v5 but I can edit my post– auvipyCommented Mar 30, 2023 at 6:02
I've made a .bat file besides my manage.py file with this code:
title CeleryTask
::See the title at the top.
cd
cmd /k celery -A MainProject worker -l info
so each time I want to run celery, I just double-click this batch file and it runs perfectly. And the fact that you can't use celery 4 on windows is true.
-
you can use celery 4 on windows: celery -A MainProject worker -l info - -pool=solo Commented Apr 17, 2020 at 11:52
-
@rogers.wang, -P solo also has issues. When you run long running task for around 2 hours, connection reset error will occur between celery worker and client Commented Sep 17, 2020 at 19:07