I'm using epoll reactor and sample HTTP Proxy realization.
From another process a execute ~30 HTTP requests per second and twisted process is consuming 10% cpu
The profiler output is:
937706 function calls (934675 primitive calls) in 59.988 CPU seconds
Ordered by: cumulative time
List reduced from 312 to 15 due to restriction <15>
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 59.988 59.988 base.py:1167(run)
1 0.076 0.076 59.988 59.988 base.py:1172(mainLoop)
12697 55.456 0.004 59.253 0.005 epollreactor.py:169(doPoll)
13553 0.086 0.000 3.797 0.000 log.py:71(callWithLogger)
13553 0.096 0.000 3.691 0.000 log.py:66(callWithContext)
13553 0.047 0.000 3.481 0.000 context.py:117(callWithContext)
13553 0.067 0.000 3.426 0.000 context.py:61(callWithContext)
13552 0.095 0.000 3.359 0.000 posixbase.py:544(_doReadOrWrite)
5454 0.097 0.000 1.736 0.000 tcp.py:182(doRead)
5453 0.167 0.000 1.639 0.000 basic.py:543(dataReceived)
12697 0.095 0.000 0.587 0.000 base.py:762(runUntilCurrent)
4453 0.025 0.000 0.574 0.000 http.py:495(rawDataReceived)
6000 0.036 0.000 0.547 0.000 http.py:1515(lineReceived)
2000 0.029 0.000 0.454 0.000 tcp.py:371(doConnect)
5098 0.104 0.000 0.428 0.000 abstract.py:212(doWrite)
937706 function calls (934675 primitive calls) in 59.988 CPU seconds
Ordered by: internal time
List reduced from 312 to 15 due to restriction <15>
ncalls tottime percall cumtime percall filename:lineno(function)
12697 55.456 0.004 59.253 0.005 epollreactor.py:169(doPoll)
59101 0.277 0.000 0.277 0.000 <string>:1(fileno)
5098 0.252 0.000 0.252 0.000 tcp.py:212(writeSomeData)
24453 0.198 0.000 0.198 0.000 __init__.py:1230(getEffectiveLevel)
5453 0.167 0.000 1.639 0.000 basic.py:543(dataReceived)
5098 0.104 0.000 0.428 0.000 abstract.py:212(doWrite)
5454 0.097 0.000 1.736 0.000 tcp.py:182(doRead)
13553 0.096 0.000 3.691 0.000 log.py:66(callWithContext)
12697 0.095 0.000 0.587 0.000 base.py:762(runUntilCurrent)
13552 0.095 0.000 3.359 0.000 posixbase.py:544(_doReadOrWrite)
13553 0.086 0.000 3.797 0.000 log.py:71(callWithLogger)
1 0.076 0.076 59.988 59.988 base.py:1172(mainLoop)
16556 0.074 0.000 0.074 0.000 context.py:86(getContext)
17451 0.073 0.000 0.094 0.000 epollreactor.py:71(_add)
17098 0.068 0.000 0.085 0.000 epollreactor.py:112(_remove)
What is wrong? Why twisted perfomance is so poor here?
Source code being tested applied
import hotshot, hotshot.stats
from twisted.internet import epollreactor
epollreactor.install()
from twisted.internet import reactor
from twisted.web import http
from twisted.web.proxy import Proxy
factory = http.HTTPFactory()
factory.protocol = Proxy
reactor.listenTCP(18001, factory)
reactor.callLater(60, reactor.stop)
prof = hotshot.Profile("stones.prof")
prof.start()
reactor.run()
prof.close()