Questions tagged [python-rq]
RQ is a simple, lightweight, Python library for creating background jobs, and processing them.
157
questions
1
vote
0answers
11 views
Django-RQ having a hard time adding items to Django Model
I'm using django-rq to setup background tasks using Redis. Redis is working great and the code is sending the request to Redis - but it is throwing an exception.
Exception:
Traceback (most recent call ...
0
votes
0answers
18 views
What is the ideal number of workers in django-rq and python-rq?
I have a question regarding django-rq. It is pip-installed library that functions as a small layer on top of python-rq, which runs on a redis instance. Currently, I run all of the jobs on the default ...
0
votes
0answers
35 views
+100
Python-rq takes unreasonably high CPU usage
I'm new to python-rq, this is how I enqueue my jobs:
After I execute ~rq worker~, everything runs normally except that rq trying to use many of my CPUs:
Normally the job I executed only takes 100% ...
0
votes
1answer
43 views
Why is importing `rq` much slower than importing `celery`?
I am unsure if I should use "celery" or "rq".
I am looking for a light-weight solution and my gut feeling told me that importing celery will be much slower than importing rq.
But ...
0
votes
1answer
174 views
RQ Job queued but not run
I'm having issues enqueuing jobs with Python-RQ, jobs seems well enqueued but they don't run, crash or whatever they have to do.
The process I'm doing is the following:
Run redis server on localhost:
...
0
votes
1answer
14 views
Python rq: handle success or failure of jobs
I have a fairly basic (so far) queue set up in my app:
Job 1 (backup): back up the SQL table I'm about to replace
Job 2 (update): do the actual table drop/update
very simplified code:
from rq import ...
2
votes
1answer
45 views
Executing an awaitable / async function in Python RQ
My jobs are all a series of requests that need to be made per object. Ie, its a User with several data points (potentially hundreds) that need to be added to that user with requests. I had originally ...
0
votes
0answers
145 views
Queuing two step process with RQ queue
I have the following function which I want to parallelize or run --
def func():
a = f1()
b = f2(a)
Say f1 takes t1 seconds and f2 takes t2 seconds (say t2>t1). I wish to use python rq and ...
0
votes
0answers
291 views
Django rq-scheduler: jobs in scheduler doesnt get executed
In my Heroku application I succesfully implemented background tasks. For this purpose I created a Queue object at the top of my views.py file and called queue.enqueue() in the appropriate view.
Now I'...
0
votes
0answers
293 views
flask_rq, redis, rq Moving job to FailedJobRegistry Work-horse process was terminated unexpectedly (waitpid returned None)
Working on flask app, with Flask-RQ2, redis, and rq libraries. Queue works, but with larger datasets i get the error:
Moving job to FailedJobRegistry Work-horse process was terminated unexpectedly (...
0
votes
0answers
181 views
Use Redis and python-rq to create jobs in flask and obtain in another service
I am looking to create a flask app that will add tasks to a radis-queue using python-rq and to consume job results in another microservice that runs on another docker.
I am bit confused if it's ...
0
votes
0answers
117 views
Python rq loading flask app for every request added to queue
Am hosting apis through flask app, along with -
running on top are 2 gunicorn workers
flask tasks are queued using 2 rq workers
First I start 2 Gunicorn workers, and I get this message -
2020-10-29 ...
2
votes
0answers
167 views
Redis RQ Python - jobs not registering
I need your help on figuring out how to get RQ working with Flask. I have a simple Flask page with a bunch of links to trigger scripts which are / should enter RQ. It seems that only one "...
0
votes
1answer
191 views
Redis Queue Kill Job using JOB-ID
I used Python Flask+Redis and I queued the jobs at redis queue using the below code:
with Connection(redis.from_url("redis://localhost:6379")):
queue = Queue()
task =...
0
votes
0answers
25 views
RQ Worker Processing Jobs in a Batch
Say you have a RQ Queue with lots of jobs, which gets filled from various sources.
Those Jobs would be more efficiently processed in batches, eg pulling and processing 100 Jobs at a time.
How would ...
0
votes
0answers
33 views
How to use the web process to add a task to a job queue for Flask App deployment to Heroku
Getting this error when running Flask app in Heroku:
UnicodeError: label empty or too long
2020-08-31T04:37:51.710930+00:00 app[web.1]:
2020-08-31T04:37:51.710931+00:00 app[web.1]: The above ...
1
vote
0answers
69 views
Heroku + Redis Cloud: Temporary failure in name resolution
I am using the Redis Cloud add-in on my Heroku application and I keep getting this error sporadically. I have tried flushing the redis DB and restarting dynos and that seems to fix it but I am curious ...
0
votes
0answers
78 views
How can I ensure that this Python RQ task on a Redis server is being executed?
In a Flask app, I have a function that takes a list of lists and adds each as a row to an Excel file. This function works fine on a development server when the request won't time out. Here it is ...
0
votes
0answers
34 views
“Expected string or bytes-like object” only when using RQ
This code can generate and upload a PDF to S3 using Boto and Python (Flask), but when a RQ worker is used, the exception "Expected string or bytes-like object" is returned.
generator.py
...
1
vote
0answers
198 views
python-rq: worker is busy but current_job doesn't exist
I'm using python-rq to enqueue background tasks and then attempting to check their status in my web app.
First I grab all the workers attached to the queue:
workers = rq.Worker.all(queue=queue)
...
-1
votes
1answer
136 views
JavaScript while loop + setInterval doesn't work in Django
I'm developing a website using Django. I have about 50 functions. And it takes about 1 minute to get the result for all of them.
The problem is that I use heroku for my app. And heroku has a limit of ...
0
votes
0answers
27 views
rq - Job creating not correctly resolve namespace
To reproduce the problem enter REPL and type:
from rq.job import Job
from redis import Redis
conn = Redis()
# now create a job
j = Job.create(func=AsyncJob.upload, connection=conn)
, where I also ...
1
vote
1answer
158 views
Can a Python RQ job reschedule itself and keep depending jobs around?
I have a Python RQ job that downloads a resource from a webserver.
In case of a non-responding webserver, can the download-job reschedule itself and retry the download after a certain interval?
...
1
vote
0answers
200 views
How to obtain the return value from a function enqueued in rq?
From the documentation of redis queue https://python-rq.org/docs, I came to know that the worker can return results only after a certain time and till then return None.
Is there any way to find out ...
0
votes
0answers
36 views
Inconsistent timing of tasks using RQ workers with py-redis
I am running a Python program where each instance of the program uses a unique redis-server, and RQ workers are used to run tasks remotely. The program is run on a PBS cluster and the RQ workers are ...
0
votes
0answers
27 views
Problem with rq background jobs in Django Heroku
I have Django app which makes lots of different analysis
Here is the structure of the view:
def analysis(request):
making analysis 1
...
making analysis 2
...
making analysis 3
...
1
vote
1answer
243 views
Python rq error while putting a class method on an RQ queue
I need to put a class method to on an RQ queue. But it gives an error
Here is the worker.py
import os
import redis
from rq import Worker, Queue, Connection
listen = ['high', 'default', 'low']
...
0
votes
2answers
214 views
Issue Deploying Heroku App - RQ Worker: wrong number of arguments for 'HSET' command
Modifying an app based on this example, I am having a problem when deploying my updated Heroku app during initialization of the web.1 dyno.
Everything is working fine in my development envr; when I ...
1
vote
1answer
426 views
Redis queue worker crashes in utcparse
I'm trying to get a basic rq working following the tutorial at https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-xxii-background-jobs.
I'm running on windows 10 WSL1 ubuntu 20.04.
I ...
3
votes
0answers
292 views
How to start multiple workers for only one of rq Queues?
I need to start multiple workers for only one of my queues (the "high" priority one below). How can I do this in the context of a worker script that I am using to start my workers?
from config import ...
0
votes
2answers
500 views
I am getting this error The DNS operation timed out after 20.000144004821777 seconds when running my app in cygwin instead of my windows terminal
so I am trying to do some background job to run in my flask app , and in order to run the worker, I needed to use cygwin instead of my windows terminal. My application use MongoDb Atlas for documents ...
0
votes
1answer
36 views
Specifying which processes implement RQ worker using mpirun
I am using RQ to implement a job queue on a cluster, where the jobs are managed and run with Python. The RQ workers are launched via mpirun, followed by a Python program which adds jobs to the queue.
...
2
votes
0answers
31 views
Flask App with RQ worker: perform imports only on worker side
I have a Flask app that sends tasks to an RQ worker which will uses Pykaldi. I am installing Pykaldi via conda and the package is quite large.
At the moment, I have my Flask app and an RQ worker ...
1
vote
0answers
148 views
Stopping rq worker doesn't stop underlying ffmpeg process
I am fairly new to python and rq, and have come to a point I can't solve by myself.
I am using ffmpeg-python to encode livestreams, this is distributed in rq workers and displayed on a web app using ...
3
votes
1answer
109 views
How to propagate errors in python rq worker tasks to Sentry
I have a Flask app with Sentry error tracking. Now I created some tasks with rq, but their errors do not show up in Sentry Issues stream. I can tell the issues aren't filtered out, because the number ...
1
vote
1answer
314 views
Trying to get supervisor to create a worker for python-rq
I am trying to get supervisor to spawn a worker following this pattern using python-RQ, much like what is mentioned in this stackoverflow question. I can start workers manually from the terminal as ...
3
votes
0answers
217 views
Do python-rq workers support multiprocessing module?
I currently have multiple python-rq workers executing jobs from a queue in parallel. Each job also utilizes the python multiprocessing module.
Job execution code is simply this:
from redis import ...
0
votes
1answer
268 views
python-rq scheduler count the number of times job executed
I'm using Django-rq which has the functionality of Scheduling the jobs with specified interval.
https://github.com/rq/django-rq#support-for-rq-scheduler
task = scheduler.schedule(
scheduled_time=...
1
vote
1answer
164 views
Python rq-scheduler: enqueue a failed job after some interval
I am using python RQ to execute a job in the background. The job calls a third-party rest API and stores the response in the database. (Refer the code below)
@classmethod
def fetch_resource(cls, ...
0
votes
1answer
546 views
Remote workers on multiple servers with python-rq (redis)
I am using redis and python-rq to manage a data processing task. I wish to distribute the data processing across multiple servers (each server would manage several rq workers) but I would like to keep ...
1
vote
1answer
620 views
Python rq.worker not honoring Django's LOGGING config
I am trying to run a standalone script outside of a Django app that interfaces with a Redis queue and uses the Python rq module. Everything is working fine but somewhere in either upgrading to Python ...
3
votes
1answer
119 views
Best approach to tackle long polling in server side
I have a use case where I need to poll the API every 1 sec (basically infinite while loop). The polling will be initiated dynamically by user through an external system. This means there can be ...
0
votes
0answers
569 views
Python Redis Queue
I would like to implement a queue in redis + Flask and Python.
I already implemented a query like this with RQ and it works fine if you have the Flask app and the task working on the same server. I am ...
0
votes
1answer
864 views
RQ Python “Working outside of application context”
I'm trying to use Redis & RQ to set the task of sending an email, however, the "RQ Worker" is returning runtime error while using the function to send emails outside q.enqueue works fine.
app/...
5
votes
1answer
450 views
How to create a ``depends_on`` relationship between scheduled and queued jobs in python-rq
I have a web service (Python 3.7, Flask 1.0.2) with a workflow consisting of 3 steps:
Step 1: Submitting a remote compute job to a commercial queuing system (IBM's LSF)
Step 2: Polling every 61 ...
0
votes
1answer
859 views
rq in Docker: “Could not resolve a Redis connection” with an otherwise working redis connection
I feel like I'm doing what every other question on this has an accepted answer suggesting. But I am unable to get a Flask app using rq to successfully connect to redis despite successfully connecting ...
1
vote
1answer
91 views
How to reuse Mongoengine connection in python rq?
I need to sync a lot of data on the back-ground using mongoengine through python rq and i would like to reuse connection. Syncing will be done all the time and i don't wanna open connection to mongo ...
1
vote
0answers
118 views
How to implement image.read() for multiple images in flask request.files.getlist('images') in a separate job or asynchronously
I have an API to upload multiple images. How do I implement image.read() for multiple files asynchronously or in a separate job, such that I can respond back as soon as possible without having to wait ...
0
votes
1answer
229 views
Redis Queue: How to prevent chained Jobs from running asynchronously
I'm trying to get 3 Jobs to run after each other in the sequence:
Job1 -> Job2 -> Job3
These 3 jobs are defined in operations.py:
def Job1(x):
return x
def Job2(x):
return x * x
def ...
1
vote
0answers
169 views
RQ job times out
I am trying to accomplish a task by using rq and redis. However the job is sometimes getting stuck importing a module.
JobTimeoutException: Task exceeded maximum timeout value (100 seconds)
File "...