Questions tagged [grequests]
GRequests allows you to use Requests with Gevent to make asyncronous HTTP Requests easily.
139
questions
0
votes
0
answers
43
views
Grequests on AWS Lambda "Gevent is required" Error
I want to use grequests on AWS Lambda. I created venv and reqs.txt and then pip install in a folder named python, then zipped the python folder and uploaded it to a Lambda Layer. After that, I am ...
0
votes
0
answers
29
views
using websockets like python grequests
Sorry if the problem seems obvious for most of you but I am quite new in using websockets and I can't get my way around this issue.
I want to receive multiple data at the same time on python via a ...
0
votes
1
answer
39
views
How to send thousands of HTTP Requests using grerequests?
I need to request all the review pages for a company on Glassdoor, and in certain cases, there can be thousands of pages. I am trying to use grequests to do this, but I found that when I sent more ...
0
votes
1
answer
31
views
How to get JSON data from a post in grequests library (async-requests) python
so im trying to make an async-requests thingy and i cant get the json data from a post
import grequests as requests
headers = {SOME HEADERS}
data = {
SOME DATA...
}
r = requests.post(
"...
0
votes
0
answers
32
views
using grequest for many request
I'm using the grequest library for translating api requests, I need to quickly send about 2000 requests, but in this case I get an error: response 429, too many requests. How is it possible to set a ...
0
votes
1
answer
164
views
part of the requests delay the whole process in grequests
I have 600 urls to request, when I use grequests, I found that sometimes it finishes so fast within 10 secs, but sometimes just stuck there(can't reach the statement of printing 'done').
Here is my ...
0
votes
1
answer
138
views
why do I can't extract data from response using grequests?
import grequests
import time
start_time = time.time()
sites = ['https://facebook.com' for x in range(5)]
data = {'a': 'b'}
responses = [grequests.get(u, data=data) for u in sites]
for response in ...
1
vote
1
answer
359
views
How to avoid 429 error when using grequests?
I want to make a parser using "grequests", but as expected, due to many requests, i get an error 429.
import grequests
sites = [f"https://fantlab.ru/work{i}" for i in range(1, 10)]...
0
votes
0
answers
100
views
Difference between async, aiohttp and grequests - python
Could someone explain me, in a simple way, the difference between using grequests on one side, aiohttp / asyn on the other side ?
I'm not sure to be clear about the exact meaning between asynchronous, ...
0
votes
1
answer
394
views
Enabling gevent debugging in Google Colab
I want to used the grequests library in a Google Colab notebook, however upon importing it (and patching it using gevent.monkey), the program spits out a random amount of the following warning:
It ...
0
votes
0
answers
75
views
How to block requests while analyzing 1000 links?
How i can block "grequests" when i found "target file"? I need to analyze like 1000 links, but just one is that good.
If I have found that link after e.g. 10 URLs, I need to block ...
0
votes
1
answer
169
views
MODUL grequests, How GET simple print URL and Response?
anyone can please explain me how i can split results for get just simple url and response?
I have try so many time but nothing, for now i can print just like:
50
0.4110674999999999
........, [<...
1
vote
1
answer
206
views
How to Scrape Product Pages using Python grequests and BeautifulSoup
from bs4 import BeautifulSoup
import grequests
import pandas as pd
# STEP 1: Create List of URLs from main archive page
def get_urls():
urls = []
for x in range(1,3):
urls.append(...
0
votes
0
answers
143
views
Asynchronous API Requests
I am trying to send multiple requests to an API using grequests, but I am getting no response, when I map the list. The API has a limit of about 1000 requests so it cannot be that I am getting denied.
...
1
vote
1
answer
182
views
How to parse the response from Grequests faster?
I want to webscraping multiple urls and parse quick as possible but the for loop is not too faster for me, have a way to do this maybe with asynchronous or multiprocessing or multithreading?
import ...
0
votes
1
answer
136
views
S3 upload fails due to import modules collision
I am using a python script to upload a file to s3 bucket. However I was facing an error like
S3 Upload Failed with exception : An HTTP Client raised an unhandled exception: cannot switch to a ...
0
votes
1
answer
373
views
grequests not having the attributes response.status_code
Can I get a tip for how to do an http request via a VOLTTRON agent using grequests? From what I know about VOLTTRON I think grequests are required for an asynchronous methods.
Small snip from my ...
1
vote
0
answers
55
views
How to use grequest to automatically reconnect after grequest timeout [duplicate]
How does grequests do something like'requests' time out auto reconnect.
import time
import requests
from requests.adapters import HTTPAdapter
for i in range(1,1000):
s = requests.Session()
s....
-1
votes
1
answer
343
views
How to send multiple HTTP requests using Python
I'm trying to create a script that checks all possible top level domain combinations based on a given root domain name.
What I've done is generate a list of all possible TLD combinations and what I ...
0
votes
2
answers
395
views
Mapbox API PUT Datasets Feature return "Provide a single Feature to insert"
I am trying to add a feature to the Dataset via Mapbox API using Python. I'm following this instruction https://docs.mapbox.com/api/maps/#update-a-dataset but keep getting this error:
{'message': '...
2
votes
1
answer
4k
views
Passing async function as argument
I am trying to use use grequests to make a single HTTP call asynchronously.
All goes find until I try to call a function (handle_cars) to handle the response.
The problem is that the function is an ...
1
vote
1
answer
63
views
Why does my for loop only check the last item?
I've built a async'ed program thats going to check if a element exists on multiple paths of a website.
The program has a base url, that will get different paths of the domain to check, which are ...
2
votes
1
answer
606
views
Extracting Text From Asynchronus Request Using grequests
I am trying to extract the text part from the request that I made through grequest library but I am unable to figure out how can I do so.
If we use Requests Library I would do
r = requests.get('www....
2
votes
1
answer
2k
views
Python Webscraping: grequests vs. mult-threaded requests?
I'm trying to make web-scraper I'm writing in Python faster.
Currently I fire up a set amount of scraper threads, create a queue with a list of URLs I want to scrape and let them dequeue entries so ...
0
votes
2
answers
697
views
Unable to import grequest in AWS Lambda (Python 3.X)
When I zip "grequest" and attempt to use it in a AWS Lambda function I get this error:
[ERROR] RuntimeError: Gevent is required for grequests.
Traceback (most recent call last):
File "...
1
vote
1
answer
402
views
Asynchronous Webscraping with beautiful soup, grequests
I am trying to webscrape some information from a list of urls, and would like to do it asynchronously to save time. I am able to get the information I would like when I do so synchronously, and the ...
1
vote
0
answers
44
views
Define a variable once in an asynchronous loop? Python grequests
I have this code that I am trying to make asynchronous
def m1():
links={
#my list of links
}
rs = (grequests.get(u) for u in links)
res = grequests.map(rs)
for r in res:
...
0
votes
1
answer
241
views
How to retrieve html with grequests
When I was just doing some research on python web scraping I got to know of a package named grequests, it was said that this can send parallel HTTP requests thus gaining more speed than the normal ...
1
vote
2
answers
372
views
Asynchronous URL un-shortening in Python
I am currently trying to implement a feature into my program that will detect and unshorten any URL shorteners, including bit.ly and old goo.gl links (now no longer existent). I have found a few ...
1
vote
0
answers
1k
views
"pip install gevent" on windows gives IndexError
I'm trying to pip install grequests on Windows 10, but it errors out when it attempts to install gevent.
I'll be grateful for any clues on how to resolve this. (FWIW, I also tried pip --no-cache-dir ...
0
votes
1
answer
105
views
API + grequests
I have a script that sends lots of numbers, almost 27k, to our server through API then I get some info. However, I would like to send a few requests at once, is it possible through grequests? Because ...
-2
votes
1
answer
402
views
Difference between ZeroMQ asynchronous http requests and Messages?
How is using asynchronous HTTP Requests different from using Messages when it comes to sending data in ZeroMQ?
0
votes
0
answers
226
views
Python issues scraping certificate data from a list of URLs
I've made a get_certificate method to scrape all certificate data from a list of URLs, but my issue is that the 5 URLs get added to the database, but the certificate information from 1 of the URLs are ...
0
votes
1
answer
65
views
Python SQL not inserting data for the whole list of URLs
I am using the grequest module to use multiple URLs.
However, my issue here is that I'm only getting data inserted into my database from 1 of the URLs and not each one in the List.
Question:
How ...
0
votes
1
answer
80
views
Extracting links from website with selenium bs4 and python
Okay so.
The heading might seem like this question has already been asked but I had no luck finding an answer for it.
I need help with making link extracting program with python.
Actually It works....
2
votes
1
answer
2k
views
Even the gevent and monkey patching added top of the imports its giving error
Even the gevent and monkey patching added top of the imports its giving error: MonkeyPatchWarning: Patching more than once will result in the union of all True parameters being patched curious_george....
2
votes
1
answer
933
views
Why does grequests.get() return None?
I'm trying to fetch automatically the MAC addresses for some vendors in Python. I found a website that is really helpful, but I'm not being able to access its information from Python. When I run this:
...
0
votes
2
answers
423
views
Making Array Of Requests Using Requests Library In Python
I am using the requests library to map an array of requests after I get an array from another API request. I am using a loop for the requests but I sure there is a better way to do this because this ...
9
votes
2
answers
10k
views
GRequests monkey patch warning
I get the following warning each time , though the module works as expected :
/usr/local/lib/python3.7/site-packages/grequests.py:21: MonkeyPatchWarning: Monkey-patching ssl after ssl has already ...
0
votes
1
answer
162
views
Multithreading requests without a URL list
I'm currently trying to make my requests faster by multithreading them but I'm not sure how to do it the way I want. I know about grequests but they seem to require a URL list. I have code with a ...
1
vote
0
answers
119
views
Difference in using grequests with chunks of addresses or all at once?
I try to download the /home.htm file of around 6000 http addresses. For speed I tried to use grequests to send them all at once but I only get around 200 answers, most of them give a connection ...
0
votes
2
answers
310
views
How to follow redirect in grequests?
How to follow redirect in grequests ?
In requests library we can simply use : r = requests.get(url, allow_redirects=True) to allow redirect .
is there anything in grequests ?
i have checked https:/...
1
vote
2
answers
4k
views
How to map grequests response with request?
I wrote a python code to send async requests using grequests, how to check which request's response is this ? are responses are in same order as they are sent ?
Here is my code :
import grequests
...
2
votes
2
answers
2k
views
Why does 'API' requests return 'None' in grequests while 'normal web scraping' gives a 200 response
EXAMPLE:
The below code is capable of getting multiple web results with grequests but why on earth does it fail with "making simple api calls"
CODE:
import grequests
links = [
'https://api....
-1
votes
1
answer
966
views
How to send 10000 POST requests fast in python?
import requests
for i in range(100):
data = ("test=test")
header = {"header":"dasdasddasd"}
x = requests.post("http://example.com", data=data, headers=header)
print(x)
I want send ...
0
votes
1
answer
165
views
Additional arguments in Flask grequests hook
I am having issue in passing additional parameter to grequests using a hook, Its working in a standalone app (non flask) but its not with flask (flask integrated server) Here is my code snippet.
...
1
vote
1
answer
609
views
How to throttle GET requests from a list of URLs
I have a list of ~250000 urls, that I need to get data from an API.
I have created a class using the grequests library to make asynchronous calls. However the API limit is 100 calls per second, ...
2
votes
3
answers
2k
views
How to slow down asynchrounous API calls to match API limits?
I have a list of ~300K URLs for an API i need to get data from.
The API limit is 100 calls per second.
I have made a class for the asynchronous but this is working to fast and I am hitting an ...
0
votes
1
answer
722
views
How to stagger asynchronous API calls to prevent Max retries with grequests library?
I have a list of ~250K urls for an API that I need to retrieve.
I have made a class using grequests that works exactly how I want it to except, I think it is working too fast because after running ...
0
votes
1
answer
547
views
How to send batches of URLs into grequest?
I have a list of ~300K API URLs that I want to call and get data from:
lst = ['url.com','url2.com']
If I subset my list down to say 5 urls grequest handles the request perfectly. However when I pass ...