4

I am developing a program which allows for simulations of networks on a single machine. For this I am using Twisted for asynchronous I/O, as having a thread for each 'connection' might be a bit much. (I have also implemented a similar program in Java using their NIO). However as I scale the emulated network size up the throughput on Twisted decreases. When comparing this to the Java implementation, for the same network size, the Java throughput continues to grow. (The growth rate slows down, but it's still an increase). E.g. (Python 100 nodes = 58MB Total throughput, 300 nodes = 45MB, Java 100 nodes = 24 MB, 300 nodes = 56MB).

I am wondering if anyone has any suggestion on why this might be happening?

The only reason that I can think of is that the Java one has each 'peer' running in its own thread (which contains its own selector that monitors that peers connections). In the python version everything is registered with the reactor (and subsequently the one selector). As the python one scales up the one selector is not able to respond as fast. However this is just a guess, if anyone has any more concreate information it would be appriciated.

EDIT: I ran some testing as suggested by Jean-Paul Calderone, the results are posted at imgur. For those who might be wondering the following Avg throughput was reported for the tests. (The profiling was done with cProfile, tests were run for 60 seconds)

Epoll Reactor: 100 Peers: 20.34 MB, 200 Peers: 18.84 MB, 300 Peers: 17.4 MB

Select Reactor: 100 Peers: 18.86 MB, 200 Peers: 19.08 MB, 300 Peers: 16.732 MB

A couple things that seemed to go up and down with the reported throughput was the calls made to main.py:48(send), but this corrolation is not really a surprise as this is where the data is being sent.

For both of the reactor the time spent in the send function on the socket(s) increased as throughput decreased, as well as the number of calls to the send function decreased as throughput decreased. (That is: more time was spent sending on the socket, with less calls to send on a socket.) E.g. 2.5 sec for epoll {method 'send' of '_socket.socket' objects} on 100 peers for 413600 calls, to 5.5 sec for epoll on 300 peers, for 354300 calls.

So as to try to answer the original question, does this data seem to point to the selector being a limiting factor? The time spent in the Selector seems to decrease as the number of peers increases ( if the selector was slowing everything down wouldn't one expect the time spent inside to rise?) Is there anything else that might be slowing the amount of data being sent? (The sending of the data is just one function for each peer that is registered with reactor.calllater again and again. That is the main.py:49 (send))

4
  • If you are using deferToThread whenever possible (even on callbacks and errbacks explicitly) then you may be running into a limitation of Twisted or even of CPython. Have you tried raising the Twisted thread pool size, or running under PyPy or Stackless Python?
    – wberry
    Aug 19, 2011 at 21:25
  • 1
    Another thing that could matter at the high end is the reactor you are using. By default Twisted uses SelectReactor on Unix. Other reactor classes are provided. I have never used any of the other ones. It's not a common use case to use a different reactor, but there may be one that is designed for high-degree threading.
    – wberry
    Aug 19, 2011 at 21:27
  • @wberry: You're right, as Jean-Paul suggests using the epoll reactor. Check his link. Aug 20, 2011 at 15:39
  • @wberry I'm not using deferToThread anywhere, as there isn't much heavy lifting being done, (just the sending and a few small calculations.) I did try the epoll, one thing that I found was that it scaled a little better, but more interestingly is that it seemed to be a little more stable in the ThroughPut (less variation from measurement to measurement). See the link provided in the edit above.
    – K_D
    Aug 22, 2011 at 20:38

1 Answer 1

1

Try profiling the application at different levels of concurrency and see which things get slower as you add more connections.

select is a likely candidate; if you find that it is using noticably more and more time as you add connections, try using the poll or epoll reactor.

2
  • Hey, sorry for taking so long to get back to you but I didn't have access over the weekend to run the tests. I've now run them and posted some images (see link ) The things I noticed was the following:
    – K_D
    Aug 22, 2011 at 19:21
  • The epoll reactor seemed to have less variation in the throughput. While the calls to the functions doPoll and doSelect decrease as the number of peers increase, it does not seem to be a great indicator. (for doPoll the amount of calls to it decreased by half, but the throughput decreased by only 7%, and in *one case for select the calls decreased but the throughput went up.) The only thing that I found that corrolated between throughput and calls was the main.py:49 (send), but this isn't surprising as this is where I send the messages. * going to re-run that test shortly to dbl check it
    – K_D
    Aug 22, 2011 at 19:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.