22

Is it possible to limit the download rate of GET requests using the requests Python library? For instance, with a command like this:

r = requests.get('https://stackoverflow.com/')

...is it possible to limit the download rate? I'm hoping for something similar to this wget command:

wget --limit-rate=20k https://stackoverflow.com/

I know it's possible with urllib2. I'm asking specifically about the requests library.

2 Answers 2

9

There are several approaches to rate limiting; one of them is token bucket, for which you can find a recipe here and another one here.

Usually you would want to do throttling or rate limiting on socket.send() and socket.recv(). You could play with socket-throttle and see if it does what you need.


This is not to be confused with x-ratelimit rate limiting response headers, which are related to a number of requests rather than a download / transfer rate.

2
  • 9
    Since the OP asked about doing this with the requests library, this doesn't really answer the question... except that you are suggesting he re-write deep portions of requests to do what he wants.
    – Dan H
    Aug 28, 2018 at 20:14
  • @DanH, you are more than welcome to provide a better answer, at time of writing the answer was "no, unless you mess with the internals", which isn't helpful to OP.
    – dnozay
    Sep 2, 2018 at 17:27
3

No built-in support but, it is possible to use stream api.

>>> import requests
>>> import time
>>> req = requests.request('GET', 'https://httpbin.org/get', stream=True)
>>> for data in req.iter_content(chunk_size=1024):
...     time.sleep(0.001)
...

In advanced usage there is written that its's allow you to retrieve smaller quantities of the response at a time.

In my network the example above (leading to a several GB large file) without sleep had bandwidth 17.4 MB/s and with sleep 1 ms 2.5 MB/s.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.