52

How to run celery worker on Windows without creating Windows Service? Is there any analogy to $ celery -A your_application worker?

13 Answers 13

76

Celery 4.0+ does not officially support window already. But it still works on window for some development/test purpose.

Use eventlet instead as below:

pip install eventlet
celery -A <module> worker -l info -P eventlet

It works for me on window 10 + celery 4.1 + python 3.

This solution solved the following exception:

[2017-11-16 21:19:46,938: ERROR/MainProcess] Task handler raised error: ValueError('need more than 0 values to unpack',)
Traceback (most recent call last):
  File "c:\users\wchen8\work\venv\weinsta\lib\site-packages\billiard\pool.py", line 358, in workloop
    result = (True, prepare_result(fun(*args, **kwargs)))
  File "c:\users\wchen8\work\venv\weinsta\lib\site-packages\celery\app\trace.py", line 525, in _fast_trace_task
    tasks, accept, hostname = _loc
ValueError: need more than 0 values to unpack

===== update 2018-11 =====

Eventlet has an issue on subprocess.CalledProcessError:

https://github.com/celery/celery/issues/4063

https://github.com/eventlet/eventlet/issues/357

https://github.com/eventlet/eventlet/issues/413

So try gevent instead.

pip install gevent
celery -A <module> worker -l info -P gevent

This works for me on window 10 + celery 4.2 + python 3.6

4
  • 1
    this also works with gevent and solo concurrency pool implementations (tested with Windows 10 and Celery 4.2.1): See github.com/ZoomerAnalytics/celery-4-windows for example code Commented Sep 3, 2018 at 16:23
  • 2
    nice! works (sort of) under windows server 12, python 3.7.1, celery 4.2.1, gevent 1.3.7. The only issue for me is monitoring of these workers. celery *** inspect active works initially after workers are turned on, and then after about 5 minutes it becomes unresponsive, not sure why yet (this was not an issue under python 3.3.5, celery 4.1.0 eventlet 0.22.1).
    – pangyuteng
    Commented Jan 3, 2019 at 17:46
  • 1
    for future readers, the issue that I am encountering is discussed here: github.com/celery/celery/issues/4817 seems like the work around is setting CELERY_BROKER_HEARTBEAT = 0.
    – pangyuteng
    Commented Jan 3, 2019 at 18:15
  • Does this work for concurrent requests scenario to celery worker? Commented Sep 17, 2020 at 19:01
29

yes:

celery -A your_application -l info

also note Celery have dropped support for Windows(since v4), so best to

pip install celery==3.1.25

3.1.25 was the last version that works on windows(just tested on my win10 machine). Didn't need to downgrade flower(browser monitor for celery) though.

See also the FAQ for Windows

4
  • the link goes to a non existent page.
    – Renier
    Commented Jul 24, 2018 at 6:29
  • using 3.1.25 i got ValueError: not enough values to unpack (expected 3, got 0)
    – pelos
    Commented Jul 7, 2020 at 1:21
  • It works for celery version 4.0 and above as well for windows OS, you just have to enable multiprocessing environment variable before creating celery instance. Commented Sep 17, 2020 at 19:03
  • Cannot install version 3.1.25, gives an error: error in anyjson setup command: use_2to3 is invalid. during installation
    – GooDeeJAY
    Commented Jun 3, 2022 at 14:08
22

Compile Celery with --pool=solo argument.

Example: celery -A your-application worker -l info --pool=solo

2
  • 3
    Can you please explain why this solves the problem?
    – Fusion
    Commented Jul 1, 2023 at 10:56
  • 2
    @Fusion Basically python only allows one thread to use the interpreter at a time. There is a package that allows python to sidestep this global interpreter lock, called billiard. Celery relies on billiard when it uses the default pool type of --pool=prefork, which lets the workers fork (copy entire state, variables, etc) to utilize different threads. Windows doesn't allow process forking, it only allows process spawning, which means the worker has to start a new python interpreter and import pkgs, etc. So you have to use a solo pool and run the task in the worker's own process. Commented May 29 at 1:05
8

There are two workarounds to make Celery 4 work on Windows:

  • use eventlet, gevent or solo concurrency pool (if your tasks as I/O and not CPU-bound)
  • set the environment variable FORKED_BY_MULTIPROCESSING=1 (this is what actually causes the underlying billiard package to to fail under Windows since version 4)

See https://www.distributedpython.com/2018/08/21/celery-4-windows for more details


1
  • What are the ramifications for selecting the second option? I still can't successfully revoke() a task after its been executed...
    – Shmack
    Commented Oct 11, 2021 at 3:17
8

I have run celery task using RabbitMQ server. RabbitMq is better and simple than redis broker

while running celery use this command "celery -A project-name worker --pool=solo -l info" and avoid this command "celery -A project-name worker --loglevel info"

1
  • For -P solo option, if the task takes longer time like around 2 hours, connection reset error will occur on windows. Commented Sep 17, 2020 at 19:04
7

It's done the same way as in Linux. Changing directory to module containing celery task and calling "c:\python\python" -m celery -A module.celery worker worked well.

4
  • 1
    There are typos you need to fix. More so, you don't need to call python before using celery. Doing celery -A tasks worker -l info should be sufficient if you have your PYTHONPATHcorrectly set. Commented May 19, 2016 at 9:05
  • what typos do you mean?
    – nix
    Commented May 19, 2016 at 10:20
  • 1
    pyton (python), woker (worker). I just thought it would better serve those who will have the same question Commented May 19, 2016 at 10:22
  • 3
    @MosesKoledoye Isn't PYTHONPATH just for the import statements? How could the Celery binary be found without having Python\Scripts directory in PATH or calling python -m? Commented Sep 26, 2016 at 15:20
7

Celery 4.0+ does not officially support window already. But it still works on window for some development/test purpose. you can use any of this:

celery worker --app=app.app --pool=eventlet --loglevel=INFO

celery worker --app=app.app --pool=gevent --loglevel=INFO

celery worker --app=app.app --pool=solo --loglevel=INFO

or using other format:

celery -A <app> worker --loglevel=info -P eventlet

celery -A <app> worker --loglevel=info -P gevent 

celery -A <app> worker --loglevel=info -P solo

if you have get context error then upgrade gevent 20.6.2 and eventlet to 0.26.1 or use solo

https://www.distributedpython.com/2018/08/21/celery-4-windows/

1
  • if multiprocessing environment variable is enabled, no need to specify pool to eventlet or gevent or solo. It will work with default pool on windows as well for celery version 4.0 and above. Commented Sep 17, 2020 at 19:06
7

You can run celery on windows without an extra library by using threads

celery -A your_application worker -P threads

1
  • 2
    This disables multiple processes. All workers would run within the same process. It is good if you want a quick bit of concurrency but your tasks would hinder eachother if they require a lot of CPU.
    – nurettin
    Commented Oct 8, 2021 at 12:50
5

You can still use celery 4 0+ with Windows 10+ Just use this command "celery -A projet worker - -pool=solo - l info" instead of "celery - A project worker -l info

1
  • For -P solo option, if the task takes longer time like around 2 hours, connection reset error will occur on windows Commented Sep 17, 2020 at 19:04
2

For using celery 4.4 on windows, I think I can answer this question.

For celery version 4.0 and above, first set following environment variable in python code before creation of celery instance.

os.environ.setdefault('FORKED_BY_MULTIPROCESSING', '1')

Then run celery worker command with default pool option.

celery worker -A <celery_file> -l info

This will run celery worker concurrently with multiple child processes.

Note: When you run celery with gevent or eventlet pool, it will work but it won't run concurrent processes on windows.

1
  • What are the ramifications for selecting the second option? I still can't successfully revoke() a task after its been executed...
    – Shmack
    Commented Oct 11, 2021 at 3:17
0

To run celery on windows for development. you have two methods.

  1. Use Windows as Host Machine.

Limitations:

  • This method is only supported for Python 3.6 or lower.
  • Celery 4 does not provide support for windows.
  • will be using Old Redis-server Version.
  • Hard to manage

4. Using WSL for running Celery and Redis-server

WSL would be a friend for you on windows if you have already familiar with Linux. With few limitations including support of Docker on WSL. you can perform most of the Development tasks using WSL.

0

Celery is testing it with windows in it's CI again from version 5.x and you can try using it with windows if any issue occurs try set following environment variable in python code before creation of celery instance

os.environ.setdefault('FORKED_BY_MULTIPROCESSING', '1')

But as far as I know celery now officially support running on windows for development purpose.

2
  • How is this answer different than this one, where this suggestion was already made? stackoverflow.com/a/63944021/3250829
    – rayryeng
    Commented Mar 30, 2023 at 5:31
  • that doesn't mention that celery restarted support for windows again in v5 but I can edit my post
    – auvipy
    Commented Mar 30, 2023 at 6:02
-2

I've made a .bat file besides my manage.py file with this code:

title CeleryTask
::See the title at the top.
cd 
cmd /k celery -A MainProject worker -l info

so each time I want to run celery, I just double-click this batch file and it runs perfectly. And the fact that you can't use celery 4 on windows is true.

2
  • you can use celery 4 on windows: celery -A MainProject worker -l info - -pool=solo Commented Apr 17, 2020 at 11:52
  • @rogers.wang, -P solo also has issues. When you run long running task for around 2 hours, connection reset error will occur between celery worker and client Commented Sep 17, 2020 at 19:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.