Questions tagged [celery]
Celery is a distributed task queue framework for Python, used for asynchronous and parallel execution.
9,027
questions
0
votes
0
answers
2
views
Celery task executed always but only returning if condition is met?
I'm working with Python 3 and Celery (v5.3.6).
I have a workflow that looks something like...
part1_task
part2_task
Both Part1 and Part2 hit different external APIs, and the time they take to finish ...
0
votes
0
answers
28
views
Celery Beat schedules multiple tasks but only runs the first one
I'm facing an issue with Celery where I have multiple tasks scheduled in Celery Beat, but only the one task(notify_after_sobes) is being executed. In the Celery Beat logs, I can see that all the tasks ...
0
votes
0
answers
10
views
Celery Python - Run A Chord Callback Only After All Tasks In Body Have Finished All Retries Or Succeeded
I'm currently using celery with rabbitMQ to handle decentralization of message queues.
I'm facing a bit of an issue, where sometimes the network im on may fall for an hour or so.
I want to be able to ...
0
votes
0
answers
7
views
Delayed execution of task causing empty task getting redelivered in queue
I have celery verison 5.1.2 (sun-harmonics) and using rabbitmq for broker. Celery task are getting redelivered to queue if there is delay in processing celery task. Celery queues are getting build up ...
1
vote
0
answers
33
views
Connecting to Amazon SQS key (SSE-SQS) Encrypted SQS Queue via Django Celery [sqs]
I am unable to get my Celery working running while connecting to an SQS Encrypted Queue
This is my settings.py for my django project
SQS_AWS_ACCESS_KEY_ID = 'xxxx'
SQS_AWS_SECRET_ACCESS_KEY = 'xxxx'
...
0
votes
0
answers
13
views
django celery tasks are executed before establishing a websocket connection
The user uploads the file to the form, after which the form with the file is sent to the server, and upon successful validation, I perform the following actions:
def form_valid(self, form):
...
1
vote
1
answer
59
views
db.session.commit inside celery task: "Outside of request context"
I am trying to use celery to modify an SQLalchemy table.
The task correctly prints into the console the target record, but upon committing I get the below error.
Any ideas what I am missing?
Many ...
0
votes
0
answers
27
views
Celery returning error that task is not registered
I have been trying to solve this issue for two days now. Everything seems to be working when I run the task manually, and also celery beat and worker are working, but the issue is every time the ...
0
votes
0
answers
37
views
Celery instead of Async views with asyncio in django
I am new to django! (So basically i am seeking suggestions for this approach).
In my project i had so many views and each view had at least 1 database i/o and as all my views are synchronous i find ...
0
votes
0
answers
13
views
Django Progress bar and results backend not updating even though the celery worker shows updates?
I'm new to using Celery with Django and using RabbitMQ as a message broker, and I'm having trouble being able to get a progress bar of sorts working.
I have a function in the views.py, called ...
0
votes
0
answers
22
views
how to deal with resources for celery docker
I am building python web app using dash and for long running tasks, I am using celery.The application is dockerized with following services:
docker-compose.yaml
db: # postgres image
redis: # redis ...
0
votes
0
answers
16
views
Celery SQS Consumer Issue: Visibility Timeout and Message Handling
I've implemented a Celery SQS consumer application with a visibility timeout of 3600 seconds configured in the broker_transport_options. The consumer processes messages from an SQS queue produced by ...
1
vote
1
answer
52
views
Can't start django project in docker
At startup Redis does not work, therefore caching does not work, as I realized there is no connection to the database. I can't make loaddata fixtures and Celery can't connect to Redis either. In ...
-1
votes
0
answers
15
views
Problem reading State in DASH @long_callback
I am struggling to read State values in the @long_callback. Getting errors missing 2 required positional arguments
@app.long_callback(
output=[
Output('O_A', 'figure')
],
inputs=[...
0
votes
0
answers
12
views
Can't delete Celery Task
I'm having a very weird Issue, I mounted a Django app with Docker, and added a Redis DB to implement Celery tasks, now, yesterday everything was working perfectly, but today I created a new task to ...
0
votes
0
answers
36
views
Celery progress bar isn`t being updated
i am developing some Django app, which generates reports. I am using celery to enable progress bar after hitting 'submit' button. The problem is that I can't figure out what i am missing and why ...
0
votes
0
answers
32
views
LocalExecutor or CeleryExecutor for Airflow on a single machine?
My question may sound dumb, but we have a ETL process running in a remote machine (EC2) and as we only use it internally in the company and there are only 5 of us, we set up Airflow with Docker ...
0
votes
1
answer
23
views
How can I solve the error "ImportError: cannot import name ‘_request_ctx_stack’ from ‘flask’"?
When I am running a flask app with celery and redis, an importerror in appearing on the command prompt screen.
ImportError: cannot import name ‘_request_ctx_stack’ from ‘flask
I updated the versions ...
0
votes
1
answer
27
views
AttributeError: 'Logger' object has no attribute 'warn'
I'm trying to connect celery to my django project via docker.
But when starting the worker container, I get the following error -
File "/usr/local/lib/python3.13/site-packages/kombu/transport/...
0
votes
0
answers
39
views
How to keep scheduled jobs of celery worker in docker container? (django, celery, redis, docker)
I deploy my django app with docker. When I update my deployment, I stop current running container and start new container with new image. I start my celery worker in the created container. (exec -> ...
0
votes
0
answers
30
views
Wait until subtask is ready (celery + unittest)
I have a task which called by parent task.
@shared_task
def task1():
if condition:
task2.apply_async()
@shared_task
def task2():
pass
I want to test all flow starting by calling ...
0
votes
0
answers
21
views
the .delay() method fails as celery is unable to connect to mongodb(aws document db)in kubernates
Python Version: 3.7
Celery Version: 5.2
below is the code being used
# app.py (Flask application)
from flask import Flask, jsonify
from celery import Celery
import db_connectors
import os
from ...
0
votes
0
answers
12
views
What is difference between Celery Beats, Celery worker, Threads and Multiprocessors?
What is the difference between:-
Celery beats and Celery Worker, and if Celery worker works just like threads or multiprocessors, why do we use celery worker especially?
Thank you.
I tried working ...
0
votes
0
answers
9
views
Unable to Terminate Django 2.1 Project Running on Ubuntu 20.04.2 LTS with NGINX, UWSGI, and Celery
I have a test project (a 'zombie' project) on the same server as the production version of my app. Both projects use Celery for background tasks, but they share the same RabbitMQ port. I noticed that ...
0
votes
0
answers
33
views
How to retry failed tasks in celery chunks or celery starmap
I would like to know how to configure these tasks correctly so that the failed tasks can be retried. One thing I have noted is if one of the tasks in the chunk fails, the whole chunk (I assume ...
0
votes
0
answers
87
views
Celery Flower UNAUTHENTICATED_API not working
I am working on a project using celery workers and Flower to monitor those workers. Both the worker and Flower are running as images inside one container. I want to change attributes of the worker ...
0
votes
0
answers
21
views
Celery Emails and Accessing Mailbox
I'm in the process of moving my emails over to Celery tasks to run asychronously.
Porting over my existing tests I'm struggling with an AssertionError when I attempt to access the mail.outbox.
I ...
0
votes
0
answers
19
views
Celery and Flask - RuntimeError('Working outside of application context)
In my flask application, I have app/celery_utils.py:
from celery import Celery
celery = Celery(__name__,
broker='redis://localhost:6379/0',
include=['backend....
0
votes
0
answers
26
views
how to use redis in cpanel for python (django) module celery for sending email asynchrnously
I have used celery for sending email asynchronous task I use Redis in localhost for change password but when I want use it for send email asynchronous task in server Cpanel it gives error like :-
...
0
votes
0
answers
16
views
Making Downloaded Files a Parameter for Subsequent Functions in Django, Celery, and Digital Ocean Spaces Integration
How can I utilize Django with Celery to handle a scenario where a function downloads a file from Digital Ocean Spaces, and once the download is complete, I want to pass the downloaded file as a ...
0
votes
0
answers
28
views
Celery Task ECS Termination Issue - Need Help Updating Decorator for Handling ProtectionEnabled State Changes
Explanation:
I have a Django application where I am running multiple Celery tasks on AWS Elastic Container Service (ECS), using SQS as the broker. I am encountering an issue where the Celery tasks are ...
2
votes
0
answers
31
views
Celery task starts running before the flask app starts
The celery task is starting even before the flask app is running.
I want to start a celery task when I make API call to a certain endpoint (eg: "/report"). So I started using celery for this....
0
votes
0
answers
28
views
Celery task result persistent strategy
I have a web server, a rabbitMQ server and two task processing servers in the system. Using Celery to manage the tasks. The web server needs to query the status of the tasks. Based on Celery's doc, I ...
0
votes
0
answers
36
views
Celery Tasks not Running on Mac M2 (Process 'ForkPoolWorker' exited with 'signal 11 (SIGSEGV)')
I am encountering an issue while trying to run Celery tasks on my Mac M1 machine. The error message I'm getting is as follows:
The process has forked and you cannot use this CoreFoundation ...
0
votes
1
answer
46
views
Celery with postgresql giving error "can't adapt type 'AsyncResult'"
I am using postgresql as my backend for Celery (v5.3.5).
Celery is returning an SQL error when I call ready() on the task ASyncResult:
sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) can'...
1
vote
1
answer
24
views
Do celery have backfill and catch up feature built in if not than how can we implement these features with celery?
Do celery have backfill and catch up feature built in if not than how can we implement these features with celery?
Can we execute our job in past date using celery or can we catch up using celery?
...
0
votes
1
answer
38
views
How to configure celery with SQS as backend?
I'm trying to setup a SQS broker with a celery app, configured in a django project.
Here's my setup:
celery.py:
import os
from celery import Celery
os.environ.setdefault("DJANGO_SETTINGS_MODULE&...
0
votes
0
answers
18
views
How to prevent overlapping tasks, that executing to schedule, in django celery beat?
I'm constructing some weather reminder API, where users can register, and take JWT token, for using api endpoints, such create subscriptions, with period of notifications, and adding cities to that ...
0
votes
0
answers
23
views
Celery redis reduce freqeuncy of BRPOP messages
I'm running my worker with the following arguments --without-gossip --without-mingle --without-heartbeat with Redis as my celery broker
I can see my redis instance is frequently receiving messages (...
0
votes
0
answers
19
views
How to restart celery beat container or resolve KeyError(b'entries')?
My ecs fargate celery beat container is running well during some hours after deployment.
But some times later, error raised like below
Traceback (most recent call last):
File "/usr/local/lib/...
0
votes
1
answer
62
views
Load and unload model on celery worker
I currently have a system that initiates tasks on whisper AI model using Celery. However, the existing setup involves loading the model inside each task, which is suboptimal due to the repeated ...
0
votes
0
answers
49
views
How to run celery workers with sequential task execution with "per-child" bounds
The context is that I'm using kubernetes for parallelism and workload balancing, similar to this. Given this, it is desirable to have no concurrency within the celery workers -- we get that by k8s ...
0
votes
0
answers
15
views
FileNotFoundError: [Errno 2] No such file or directory in celery container while exists in django application container
I am developing a django application in which I have to manipulate excel files. I have two containers: one for my main django application and another for celery where I process my files. These two ...
0
votes
1
answer
85
views
Adding Celery settings - what does it do?
We upgrade Celery and started to get a warning:
CPendingDeprecationWarning: The broker_connection_retry configuration
setting will no longer determine whether broker connection retries are
made ...
1
vote
2
answers
47
views
Django Celery Results Table Missing periodictaskname and taskname Fields
I have set up Django with Celery, using django_celery_results and django_celery_beat to schedule tasks and store the results. The tasks are running successfully, but I've encountered an issue with the ...
0
votes
1
answer
26
views
Celery concurrency configuration for long tasks
I'm looking for the right configuration of celery to handle long tasks executed in django.
These are the keys of the case:
the tasks take several minutes to complete, but the processing is done in a ...
0
votes
0
answers
16
views
Apache Superset 3.0.1 Sending Alert/Report in GMT TimeZone
In Apache Superset 3.0.1, I want to scheduled Alert/Report and get the email in basis of GMT timezone. Currently, it is working on UTC format only.
Package Version
Apache Superset 3.0.1
Celery 5.3.5
...
0
votes
0
answers
12
views
Celery "unregistered task" error only on Heroku
My Flask app with celery works well locally on Mac. But on Heroku, the celery worker raises the error Received unregistered task of type 'app.task_func'. I am sure that the same commit of Git has been ...
0
votes
0
answers
40
views
Celery inspect remove 10 items limit
I need to get the worker queue size using python and Celery. The code works fine, but result is limited to 10 items. Is there any way to get entire list of queued tasks? I get 100+ items, but only 10 ...
-1
votes
1
answer
33
views
Problem communicating from celery container to redis
I'm Dockerizing my old project, but I came across this error:
> Traceback (most recent call last):File
> "/venv/lib/python3.10/site-packages/redis/connection.py", line 624, in
> ...