How to run celery worker on Windows without creating Windows Service? Is there any analogy to $ celery -A your_application worker
?
12 Answers
Celery 4.0+
does not officially support window already. But it still works on window for some development/test purpose.
Use eventlet
instead as below:
pip install eventlet
celery -A <module> worker -l info -P eventlet
It works for me on window 10
+ celery 4.1
+ python 3
.
This solution solved the following exception:
[2017-11-16 21:19:46,938: ERROR/MainProcess] Task handler raised error: ValueError('need more than 0 values to unpack',)
Traceback (most recent call last):
File "c:\users\wchen8\work\venv\weinsta\lib\site-packages\billiard\pool.py", line 358, in workloop
result = (True, prepare_result(fun(*args, **kwargs)))
File "c:\users\wchen8\work\venv\weinsta\lib\site-packages\celery\app\trace.py", line 525, in _fast_trace_task
tasks, accept, hostname = _loc
ValueError: need more than 0 values to unpack
===== update 2018-11 =====
Eventlet has an issue on subprocess.CalledProcessError:
https://github.com/celery/celery/issues/4063
https://github.com/eventlet/eventlet/issues/357
https://github.com/eventlet/eventlet/issues/413
So try gevent
instead.
pip install gevent
celery -A <module> worker -l info -P gevent
This works for me on window 10
+ celery 4.2
+ python 3.6
-
1this also works with gevent and solo concurrency pool implementations (tested with Windows 10 and Celery 4.2.1): See github.com/ZoomerAnalytics/celery-4-windows for example code Sep 3, 2018 at 16:23
-
2nice! works (sort of) under windows server 12, python 3.7.1, celery 4.2.1, gevent 1.3.7. The only issue for me is monitoring of these workers.
celery *** inspect active
works initially after workers are turned on, and then after about 5 minutes it becomes unresponsive, not sure why yet (this was not an issue under python 3.3.5, celery 4.1.0 eventlet 0.22.1). Jan 3, 2019 at 17:46 -
1for future readers, the issue that I am encountering is discussed here: github.com/celery/celery/issues/4817 seems like the work around is setting
CELERY_BROKER_HEARTBEAT = 0
. Jan 3, 2019 at 18:15 -
Does this work for concurrent requests scenario to celery worker? Sep 17, 2020 at 19:01
yes:
celery -A your_application -l info
also note Celery have dropped support for Windows(since v4), so best to
pip install celery==3.1.25
3.1.25 was the last version that works on windows(just tested on my win10 machine). Didn't need to downgrade flower(browser monitor for celery) though.
See also the FAQ for Windows
-
-
using 3.1.25 i got ValueError: not enough values to unpack (expected 3, got 0)– pelosJul 7, 2020 at 1:21
-
It works for celery version 4.0 and above as well for windows OS, you just have to enable multiprocessing environment variable before creating celery instance. Sep 17, 2020 at 19:03
-
Cannot install version 3.1.25, gives an error:
error in anyjson setup command: use_2to3 is invalid.
during installation Jun 3, 2022 at 14:08
Compile Celery with --pool=solo argument.
Example:
celery -A your-application worker -l info --pool=solo
There are two workarounds to make Celery 4 work on Windows:
- use eventlet, gevent or solo concurrency pool (if your tasks as I/O and not CPU-bound)
- set the environment variable FORKED_BY_MULTIPROCESSING=1 (this is what actually causes the underlying billiard package to to fail under Windows since version 4)
See https://www.distributedpython.com/2018/08/21/celery-4-windows for more details
-
What are the ramifications for selecting the second option? I still can't successfully
revoke()
a task after its been executed...– ShmackOct 11, 2021 at 3:17
I have run celery task using RabbitMQ server. RabbitMq is better and simple than redis broker
while running celery use this command "celery -A project-name worker --pool=solo -l info" and avoid this command "celery -A project-name worker --loglevel info"
-
For -P solo option, if the task takes longer time like around 2 hours, connection reset error will occur on windows. Sep 17, 2020 at 19:04
It's done the same way as in Linux. Changing directory to module containing celery task and calling "c:\python\python" -m celery -A module.celery worker
worked well.
-
1There are typos you need to fix. More so, you don't need to call
python
before using celery. Doingcelery -A tasks worker -l info
should be sufficient if you have yourPYTHONPATH
correctly set. May 19, 2016 at 9:05 -
-
1pyton (
python
), woker (worker
). I just thought it would better serve those who will have the same question May 19, 2016 at 10:22 -
3@MosesKoledoye Isn't
PYTHONPATH
just for theimport
statements? How could the Celery binary be found without havingPython\Scripts
directory inPATH
or callingpython -m
? Sep 26, 2016 at 15:20
You can still use celery 4 0+ with Windows 10+ Just use this command "celery -A projet worker - -pool=solo - l info" instead of "celery - A project worker -l info
-
For -P solo option, if the task takes longer time like around 2 hours, connection reset error will occur on windows Sep 17, 2020 at 19:04
You can run celery on windows without an extra library by using threads
celery -A your_application worker -P threads
-
2This disables multiple processes. All workers would run within the same process. It is good if you want a quick bit of concurrency but your tasks would hinder eachother if they require a lot of CPU.– nurettinOct 8, 2021 at 12:50
Celery 4.0+ does not officially support window already. But it still works on window for some development/test purpose. you can use any of this:
celery worker --app=app.app --pool=eventlet --loglevel=INFO
celery worker --app=app.app --pool=gevent --loglevel=INFO
celery worker --app=app.app --pool=solo --loglevel=INFO
or using other format:
celery -A <app> worker --loglevel=info -P eventlet
celery -A <app> worker --loglevel=info -P gevent
celery -A <app> worker --loglevel=info -P solo
if you have get context
error then upgrade gevent 20.6.2 and eventlet to 0.26.1 or use solo
https://www.distributedpython.com/2018/08/21/celery-4-windows/
-
if multiprocessing environment variable is enabled, no need to specify pool to eventlet or gevent or solo. It will work with default pool on windows as well for celery version 4.0 and above. Sep 17, 2020 at 19:06
I've made a .bat file besides my manage.py file with this code:
title CeleryTask
::See the title at the top.
cd
cmd /k celery -A MainProject worker -l info
so each time I want to run celery, I just double-click this batch file and it runs perfectly. And the fact that you can't use celery 4 on windows is true.
-
you can use celery 4 on windows: celery -A MainProject worker -l info - -pool=solo Apr 17, 2020 at 11:52
-
@rogers.wang, -P solo also has issues. When you run long running task for around 2 hours, connection reset error will occur between celery worker and client Sep 17, 2020 at 19:07
After feeling like killing myself by using celery 4.4 on windows, I think I can answer this question.
For celery version 4.0 and above, first set following environment variable in python code before creation of celery instance.
os.environ.setdefault('FORKED_BY_MULTIPROCESSING', '1')
Then run celery worker command with default pool option.
celery worker -A <celery_file> -l info
This will run celery worker concurrently with multiple child processes.
Note: When you run celery with gevent or eventlet pool, it will work but it won't run concurrent processes on windows.
-
What are the ramifications for selecting the second option? I still can't successfully
revoke()
a task after its been executed...– ShmackOct 11, 2021 at 3:17
To run celery on windows for development. you have two methods.
- Use Windows as Host Machine.
- Install Redis Server from https://github.com/dmajkic/redis/downloads
- Run redis_server.exe
- Install Celery using
Pip install Celery
. - Run Celery commands.
Limitations:
- This method is only supported for Python 3.6 or lower.
- Celery 4 does not provide support for windows.
- will be using Old Redis-server Version.
- Hard to manage
4. Using WSL for running Celery and Redis-server
- Install WSL using this guide https://www.windowscentral.com/install-windows-subsystem-linux-windows-10.
- Install redis-server using
sudo apt install redis-server
on the WSL terminal. - Install Celery and Execute all celery-related commands on WSL.
WSL would be a friend for you on windows if you have already familiar with Linux. With few limitations including support of Docker on WSL. you can perform most of the Development tasks using WSL.