10

I'm trying to set speed limits on downloading/uploading files and found that twisted provides twisted.protocols.policies.ThrottlingFactory to handle this job, but I can't get it right. I set readLimit and writeLimit, but file is still downloading on a maximum speed. What am I doing wrong?

from twisted.protocols.basic import FileSender
from twisted.protocols.policies import ThrottlingFactory
from twisted.web import server, resource
from twisted.internet import reactor
import os

class DownloadPage(resource.Resource):
    isLeaf = True

    def __init__(self, producer):
        self.producer = producer

    def render(self, request):
        size = os.stat(somefile).st_size
        request.setHeader('Content-Type', 'application/octet-stream')
        request.setHeader('Content-Length', size)
        request.setHeader('Content-Disposition', 'attachment; filename="' + somefile + '"')
        request.setHeader('Accept-Ranges', 'bytes')

        fp = open(somefile, 'rb')
        d = self.producer.beginFileTransfer(fp, request)

        def err(error):
            print "error %s", error

        def cbFinished(ignored):
            fp.close()
            request.finish()
        d.addErrback(err).addCallback(cbFinished)

        return server.NOT_DONE_YET


producer = FileSender()
root_resource = resource.Resource()
root_resource.putChild('download', DownloadPage(producer))
site = server.Site(root_resource)
tsite = ThrottlingFactory(site, readLimit=10000, writeLimit=10000)
tsite.protocol.producer = producer
reactor.listenTCP(8080, tsite)
reactor.run()

UPDATE

So sometime after I run it:

2012-10-25 09:17:03+0600 [-] Unhandled Error
Traceback (most recent call last):
      File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/application/app.py", line 402, in startReactor
        self.config, oldstdout, oldstderr, self.profiler, reactor)
      File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/application/app.py", line 323, in runReactorWithLogging
        reactor.run()
      File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/internet/base.py", line 1169, in run
        self.mainLoop()
      File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/internet/base.py", line 1178, in mainLoop
        self.runUntilCurrent()
    --- <exception caught here> ---
      File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/internet/base.py", line 800, in runUntilCurrent
        call.func(*call.args, **call.kw)
      File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/protocols/policies.py", line 334, in unthrottleWrites
        p.unthrottleWrites()
      File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/protocols/policies.py", line 225, in unthrottleWrites
        self.producer.resumeProducing()
      File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/protocols/basic.py", line 919, in resumeProducing
        self.consumer.unregisterProducer()
      File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/web/http.py", line 811, in unregisterProducer
        self.transport.unregisterProducer()
      File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/protocols/policies.py", line 209, in unregisterProducer
        del self.producer
    exceptions.AttributeError: ThrottlingProtocol instance has no attribute 'producer'

I see that I'm not supposed to assign producer like I do know tsite.protocol.producer = producer, I'm new to Twisted and I don't know how to do that another way.

5
  • 1
    Looking at the source there is a line, log.msg("Throttling reads on %s" % self) can you verify that this is being logged?
    – John
    Oct 24, 2012 at 12:03
  • it doesn't log *throttleReads methods, but it does *throttleWrites: Throttling writes on <twisted.protocols.policies.ThrottlingFactory instance at 0x308e3b0>
    – dan.ch
    Oct 25, 2012 at 3:19
  • I reckon you have to insert Throttling object either between file read and and render(), or perhaps better between reactor and your render/DownloadPage instance. Right now you seem to pass producer to both DownloadPage and Throttling objects, this doesn't seem right. Oct 27, 2012 at 17:32
  • ThrottlingFactory takes factory as an argument and since the only factory instance here is site I passed it. If there is a working example of bandwidth throttling example exists it would be nice to see. However I solved problem by overriding FileSender's resumeProducing
    – dan.ch
    Oct 29, 2012 at 5:21
  • Regardless of what else is going on, it sounds like you've discovered a bug. You should not be getting an AttributeError out of ThrottlingProtocol regardless of what you do. Worst case, the exception message should be informative, but I suspect the error just shouldn't be happening.
    – Glyph
    Jan 15, 2013 at 20:54

2 Answers 2

1

Every producer needs (eventually) to be registered with whatever you want to consume the data. I don't see registration happening anywhere here. Maybe that is the issue you are having?

Twisted has been used on some big-time projects like Friendster, but all the callbacks do not sit well with the usual way I write in python (and I have some experience with functional programming). I switched to gevent.

If you are working with gevent libraries, many of the details (callbacks/generators that provide the asynchronous functionality) are abstracted out, so that you can typically get away with just monkey patching your code and writing it in the usual object-oriented style you are used to. If you are working on a project with anyone unfamiliar with a callback-heavy language like js/lisp, I bet they will appreciate gevent over twisted.

6
  • This answer would be better (worthy of an upvote) without the pointless sniping. You don't even mention how gevent deals with flow control.
    – Glyph
    Dec 31, 2012 at 8:28
  • @Glyph really sorry about that -- hopefully I have removed the snarkiness. Tons of respect for the (brilliant) people who developed Twisted -- I meant no offense. Perhaps just some residual scars from my bad experience. :)
    – egbutter
    Dec 31, 2012 at 16:31
  • Thanks for altering it - but you still don't explain how gevent actually deals with the specific issue of flow control :).
    – Glyph
    Dec 31, 2012 at 21:45
  • Haha well I do not quite understand what you want ... It is hard to explain flow control with gevent works when the principle behind gevent is to be agnostic of those details -- as opposed to a callback-centered tool like Twisted.
    – egbutter
    Jan 2, 2013 at 20:35
  • It's not "agnostic" to flow control. It suspends execution of your microthread in the send() call; it's not any more "abstracted" than Twisted's version of flow control. If anything, it's less abstracted, since in Twisted at least there's an explicit abstraction object.
    – Glyph
    Jan 14, 2013 at 19:37
1

As egbutter said, you have to register a producer. So instead of this:

tsite.protocol.producer = producer

you have to call a registerProducer method explicitly:

tsite.protocol.registerProducer( ... )

or, if you're using FileSender as a producer, call its beginFileTransfer method, in our case:

file_to_send = open( ... )
producer.beginFileTransfer(file_to_send, tsite.protocol)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.