11

What libraries out there let you control the download speed of network requests (http in particular). I don't see anything built-in in urllib2 (nor in (Py)Qt which I intend on using).

Can Twisted control bandwidth? If not, how can I control the read buffer size of urllib2 or Twisted? sleeping to suspend network operations isn't an option.

3 Answers 3

9

urllib2 doesn't offer a way to do this, so you'd have to extend some of the classes it uses and implement rate limiting yourself. You might want to look at this question. If you decide to write a limiter, read up on the token bucket and leaky bucket algorithms.

Some attempted hack solutions available on github are Phredward/throttle and minkustree/socket-throttle.

Alternatively, you could use pycurl along with the CURLOPTMAXRECVSPEEDLARGE option.

EDIT: The urlgrabber package appears to support throttling as well, and is probably easier to understand than pycurl.

If you prefer to program using an event loop model, there's the Twisted approach, which has already been mentioned in another answer.

8

Of course twisted can. You want twisted.protocols.policies.ThrottlingFactory. Just wrap your existing factory in it before you pass it to whatever wants a factory.

1
  • 4
    Can you post a complete example? I'm a bit lost by all the classes.
    – Idan K
    Aug 15, 2010 at 18:47
0

I have tested previously that you can apply network restrictions to Python with this method

With this method, this typebook will be closed 😎

New contributor
ava is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.