Questions tagged [grequests]

GRequests allows you to use Requests with Gevent to make asyncronous HTTP Requests easily.

Filter by
Sorted by
Tagged with
0 votes
0 answers
43 views

Grequests on AWS Lambda "Gevent is required" Error

I want to use grequests on AWS Lambda. I created venv and reqs.txt and then pip install in a folder named python, then zipped the python folder and uploaded it to a Lambda Layer. After that, I am ...
  • 152
0 votes
0 answers
29 views

using websockets like python grequests

Sorry if the problem seems obvious for most of you but I am quite new in using websockets and I can't get my way around this issue. I want to receive multiple data at the same time on python via a ...
  • 1
0 votes
1 answer
39 views

How to send thousands of HTTP Requests using grerequests?

I need to request all the review pages for a company on Glassdoor, and in certain cases, there can be thousands of pages. I am trying to use grequests to do this, but I found that when I sent more ...
  • 59
0 votes
1 answer
31 views

How to get JSON data from a post in grequests library (async-requests) python

so im trying to make an async-requests thingy and i cant get the json data from a post import grequests as requests headers = {SOME HEADERS} data = { SOME DATA... } r = requests.post( "...
0 votes
0 answers
32 views

using grequest for many request

I'm using the grequest library for translating api requests, I need to quickly send about 2000 requests, but in this case I get an error: response 429, too many requests. How is it possible to set a ...
  • 3
0 votes
1 answer
164 views

part of the requests delay the whole process in grequests

I have 600 urls to request, when I use grequests, I found that sometimes it finishes so fast within 10 secs, but sometimes just stuck there(can't reach the statement of printing 'done'). Here is my ...
  • 103
0 votes
1 answer
138 views

why do I can't extract data from response using grequests?

import grequests import time start_time = time.time() sites = ['https://facebook.com' for x in range(5)] data = {'a': 'b'} responses = [grequests.get(u, data=data) for u in sites] for response in ...
1 vote
1 answer
359 views

How to avoid 429 error when using grequests?

I want to make a parser using "grequests", but as expected, due to many requests, i get an error 429. import grequests sites = [f"https://fantlab.ru/work{i}" for i in range(1, 10)]...
  • 53
0 votes
0 answers
100 views

Difference between async, aiohttp and grequests - python

Could someone explain me, in a simple way, the difference between using grequests on one side, aiohttp / asyn on the other side ? I'm not sure to be clear about the exact meaning between asynchronous, ...
  • 996
0 votes
1 answer
394 views

Enabling gevent debugging in Google Colab

I want to used the grequests library in a Google Colab notebook, however upon importing it (and patching it using gevent.monkey), the program spits out a random amount of the following warning: It ...
0 votes
0 answers
75 views

How to block requests while analyzing 1000 links?

How i can block "grequests" when i found "target file"? I need to analyze like 1000 links, but just one is that good. If I have found that link after e.g. 10 URLs, I need to block ...
0 votes
1 answer
169 views

MODUL grequests, How GET simple print URL and Response?

anyone can please explain me how i can split results for get just simple url and response? I have try so many time but nothing, for now i can print just like: 50 0.4110674999999999 ........, [<...
1 vote
1 answer
206 views

How to Scrape Product Pages using Python grequests and BeautifulSoup

from bs4 import BeautifulSoup import grequests import pandas as pd # STEP 1: Create List of URLs from main archive page def get_urls(): urls = [] for x in range(1,3): urls.append(...
  • 179
0 votes
0 answers
143 views

Asynchronous API Requests

I am trying to send multiple requests to an API using grequests, but I am getting no response, when I map the list. The API has a limit of about 1000 requests so it cannot be that I am getting denied. ...
1 vote
1 answer
182 views

How to parse the response from Grequests faster?

I want to webscraping multiple urls and parse quick as possible but the for loop is not too faster for me, have a way to do this maybe with asynchronous or multiprocessing or multithreading? import ...
  • 17
0 votes
1 answer
136 views

S3 upload fails due to import modules collision

I am using a python script to upload a file to s3 bucket. However I was facing an error like S3 Upload Failed with exception : An HTTP Client raised an unhandled exception: cannot switch to a ...
  • 199
0 votes
1 answer
373 views

grequests not having the attributes response.status_code

Can I get a tip for how to do an http request via a VOLTTRON agent using grequests? From what I know about VOLTTRON I think grequests are required for an asynchronous methods. Small snip from my ...
  • 3,152
1 vote
0 answers
55 views

How to use grequest to automatically reconnect after grequest timeout [duplicate]

How does grequests do something like'requests' time out auto reconnect. import time import requests from requests.adapters import HTTPAdapter for i in range(1,1000): s = requests.Session() s....
-1 votes
1 answer
343 views

How to send multiple HTTP requests using Python

I'm trying to create a script that checks all possible top level domain combinations based on a given root domain name. What I've done is generate a list of all possible TLD combinations and what I ...
0 votes
2 answers
395 views

Mapbox API PUT Datasets Feature return "Provide a single Feature to insert"

I am trying to add a feature to the Dataset via Mapbox API using Python. I'm following this instruction https://docs.mapbox.com/api/maps/#update-a-dataset but keep getting this error: {'message': '...
  • 409
2 votes
1 answer
4k views

Passing async function as argument

I am trying to use use grequests to make a single HTTP call asynchronously. All goes find until I try to call a function (handle_cars) to handle the response. The problem is that the function is an ...
  • 327
1 vote
1 answer
63 views

Why does my for loop only check the last item?

I've built a async'ed program thats going to check if a element exists on multiple paths of a website. The program has a base url, that will get different paths of the domain to check, which are ...
user avatar
2 votes
1 answer
606 views

Extracting Text From Asynchronus Request Using grequests

I am trying to extract the text part from the request that I made through grequest library but I am unable to figure out how can I do so. If we use Requests Library I would do r = requests.get('www....
  • 493
2 votes
1 answer
2k views

Python Webscraping: grequests vs. mult-threaded requests?

I'm trying to make web-scraper I'm writing in Python faster. Currently I fire up a set amount of scraper threads, create a queue with a list of URLs I want to scrape and let them dequeue entries so ...
  • 387
0 votes
2 answers
697 views

Unable to import grequest in AWS Lambda (Python 3.X)

When I zip "grequest" and attempt to use it in a AWS Lambda function I get this error: [ERROR] RuntimeError: Gevent is required for grequests. Traceback (most recent call last): File "...
1 vote
1 answer
402 views

Asynchronous Webscraping with beautiful soup, grequests

I am trying to webscrape some information from a list of urls, and would like to do it asynchronously to save time. I am able to get the information I would like when I do so synchronously, and the ...
  • 59
1 vote
0 answers
44 views

Define a variable once in an asynchronous loop? Python grequests

I have this code that I am trying to make asynchronous def m1(): links={ #my list of links } rs = (grequests.get(u) for u in links) res = grequests.map(rs) for r in res: ...
0 votes
1 answer
241 views

How to retrieve html with grequests

When I was just doing some research on python web scraping I got to know of a package named grequests, it was said that this can send parallel HTTP requests thus gaining more speed than the normal ...
1 vote
2 answers
372 views

Asynchronous URL un-shortening in Python

I am currently trying to implement a feature into my program that will detect and unshorten any URL shorteners, including bit.ly and old goo.gl links (now no longer existent). I have found a few ...
1 vote
0 answers
1k views

"pip install gevent" on windows gives IndexError

I'm trying to pip install grequests on Windows 10, but it errors out when it attempts to install gevent. I'll be grateful for any clues on how to resolve this. (FWIW, I also tried pip --no-cache-dir ...
0 votes
1 answer
105 views

API + grequests

I have a script that sends lots of numbers, almost 27k, to our server through API then I get some info. However, I would like to send a few requests at once, is it possible through grequests? Because ...
user avatar
-2 votes
1 answer
402 views

Difference between ZeroMQ asynchronous http requests and Messages?

How is using asynchronous HTTP Requests different from using Messages when it comes to sending data in ZeroMQ?
0 votes
0 answers
226 views

Python issues scraping certificate data from a list of URLs

I've made a get_certificate method to scrape all certificate data from a list of URLs, but my issue is that the 5 URLs get added to the database, but the certificate information from 1 of the URLs are ...
0 votes
1 answer
65 views

Python SQL not inserting data for the whole list of URLs

I am using the grequest module to use multiple URLs. However, my issue here is that I'm only getting data inserted into my database from 1 of the URLs and not each one in the List. Question: How ...
0 votes
1 answer
80 views

Extracting links from website with selenium bs4 and python

Okay so. The heading might seem like this question has already been asked but I had no luck finding an answer for it. I need help with making link extracting program with python. Actually It works....
2 votes
1 answer
2k views

Even the gevent and monkey patching added top of the imports its giving error

Even the gevent and monkey patching added top of the imports its giving error: MonkeyPatchWarning: Patching more than once will result in the union of all True parameters being patched curious_george....
2 votes
1 answer
933 views

Why does grequests.get() return None?

I'm trying to fetch automatically the MAC addresses for some vendors in Python. I found a website that is really helpful, but I'm not being able to access its information from Python. When I run this: ...
  • 1,102
0 votes
2 answers
423 views

Making Array Of Requests Using Requests Library In Python

I am using the requests library to map an array of requests after I get an array from another API request. I am using a loop for the requests but I sure there is a better way to do this because this ...
  • 439
9 votes
2 answers
10k views

GRequests monkey patch warning

I get the following warning each time , though the module works as expected : /usr/local/lib/python3.7/site-packages/grequests.py:21: MonkeyPatchWarning: Monkey-patching ssl after ssl has already ...
  • 1,736
0 votes
1 answer
162 views

Multithreading requests without a URL list

I'm currently trying to make my requests faster by multithreading them but I'm not sure how to do it the way I want. I know about grequests but they seem to require a URL list. I have code with a ...
  • 18
1 vote
0 answers
119 views

Difference in using grequests with chunks of addresses or all at once?

I try to download the /home.htm file of around 6000 http addresses. For speed I tried to use grequests to send them all at once but I only get around 200 answers, most of them give a connection ...
0 votes
2 answers
310 views

How to follow redirect in grequests?

How to follow redirect in grequests ? In requests library we can simply use : r = requests.get(url, allow_redirects=True) to allow redirect . is there anything in grequests ? i have checked https:/...
1 vote
2 answers
4k views

How to map grequests response with request?

I wrote a python code to send async requests using grequests, how to check which request's response is this ? are responses are in same order as they are sent ? Here is my code : import grequests ...
2 votes
2 answers
2k views

Why does 'API' requests return 'None' in grequests while 'normal web scraping' gives a 200 response

EXAMPLE: The below code is capable of getting multiple web results with grequests but why on earth does it fail with "making simple api calls" CODE: import grequests links = [ 'https://api....
-1 votes
1 answer
966 views

How to send 10000 POST requests fast in python?

import requests for i in range(100): data = ("test=test") header = {"header":"dasdasddasd"} x = requests.post("http://example.com", data=data, headers=header) print(x) I want send ...
0 votes
1 answer
165 views

Additional arguments in Flask grequests hook

I am having issue in passing additional parameter to grequests using a hook, Its working in a standalone app (non flask) but its not with flask (flask integrated server) Here is my code snippet. ...
1 vote
1 answer
609 views

How to throttle GET requests from a list of URLs

I have a list of ~250000 urls, that I need to get data from an API. I have created a class using the grequests library to make asynchronous calls. However the API limit is 100 calls per second, ...
2 votes
3 answers
2k views

How to slow down asynchrounous API calls to match API limits?

I have a list of ~300K URLs for an API i need to get data from. The API limit is 100 calls per second. I have made a class for the asynchronous but this is working to fast and I am hitting an ...
0 votes
1 answer
722 views

How to stagger asynchronous API calls to prevent Max retries with grequests library?

I have a list of ~250K urls for an API that I need to retrieve. I have made a class using grequests that works exactly how I want it to except, I think it is working too fast because after running ...
0 votes
1 answer
547 views

How to send batches of URLs into grequest?

I have a list of ~300K API URLs that I want to call and get data from: lst = ['url.com','url2.com'] If I subset my list down to say 5 urls grequest handles the request perfectly. However when I pass ...