I have two integers in my program; let's call them "a
" and "b
". I would like to add them together and get another integer as a result. These are regular Python int
objects. I'm wondering; how do I add them together with Twisted? Is there a special performAsynchronousAddition
function somewhere? Do I need a Deferred
? What about the reactor? Is the reactor involved?
3 Answers
OK, to be clear.
Twisted doesn't do anything about cpu bound tasks and for good reason. there's no way to make a compute bound job go any quicker by reordering subtasks; the only thing you could possibly do is add more compute resources; and even that wouldn't work out in python because of a subtlety of its implementation.
Twisted offers special semantics and event loop handling in case the program would become "stuck" waiting for something outside if its control; most normally a process running on another machine and communicating with your twisted process over a network connection. Since you would be waiting anyways, twisted gives you a mechanism to get more things done in the meantime. That is to say, twisted provides concurrency for I/O Bound tasks
tl;dr: twisted is for network code. Everything else is just normal python.
-
5(Worth noting though is that with
spawnProcess
, Twisted does provide a little support for CPU-bound tasks, by converting them into I/O bound tasks and then communicating with them the same way. But this is a good explanation nonetheless.)– GlyphCommented Sep 6, 2011 at 4:16 -
5As a friend and neighbor, I'm begging you: Please do not spawn a process for the purpose of adding two integers. Commented Sep 6, 2011 at 4:49
-
26Tempting as it is to troll you further, I should note at this point that I'm the original author of Twisted and that this question was written as a sort of experiment. We get a lot of questions of the flavor: "How do I call function
X
, with Twisted?", where the answer ("X()
") is so obvious that we have trouble communicating it. Many programmers are too used to using idioms like "time.sleep()
" to mean "... and then do a bunch of I/O and/or computation ... " that they can't or won't flesh out their examples further, when the distinction between I/O and computation is really critical.– GlyphCommented Sep 6, 2011 at 7:45 -
2I actually quite liked your answer and will probably use elements from it in the future. I was surprised to get such useful feedback so quickly ;).– GlyphCommented Sep 6, 2011 at 7:48
How about this:
c = a + b
That should work, and it doesn't need to be done asynchronously (it's pretty fast).
-
1You are forgetting that Python integers are unbound. It is very easy to build numbers for which any operation takes minutes. Commented Aug 26, 2015 at 8:48
-
Good question, and Twisted (or Python) should have a way to at least spawn "a + b" of to several cores (on my 8 core i7).
Unfortunately the Python GIL prevents this from happening, meaning that you will have to wait, not only for the CPU bound task, but for one core doing the job, while the seven others core are doing nothing.
Note: Maybe a better example would be "a() + b()", or even "fact(sqrt(a()**b())" etc. but the important fact is that above operation will lock one core and the GIL pretty much prevents Python for doing anything else during that operation, which could be several ms...
-
-1: whether or not
a + b
runs on a single or multiple CPUs depends on whata
andb
are e.g., it is possible to release GIL ifa
andb
are numpy arrays. The task may be embarrassingly parallel i.e., it is trivial to distribute it among several CPUs. The result can be slower than sequential variant but it can be done easily.– jfsCommented Dec 21, 2013 at 7:39 -
It is not possible to distribute processing across several cores in the same CPU and process, because of the GIL. So even if I have 8 cores in my brand new MBP and I started 8 threads to distribute a calculation, the performance will degrade a factor 1000 or more.– renejsumCommented Dec 28, 2013 at 22:38
-
it is possible to distribute processing across several CPU cores in the same process because GIL can be released in C extensions such as
numpy
,lxml
,regex
, etc.– jfsCommented Dec 29, 2013 at 5:26 -
I understand that there are workarounds, but if you just wanted to start 8 threads in "pure" Python and have them crunch ahead one on each core, possible coordinating progress through some locks you are out of luck...– renejsumCommented Dec 30, 2013 at 1:11
-
It's probably a bit late, but have a look at Pony language, it has an
actor
keyword, which means async class/object. Pony will start a thread for each core on CPU, potentially making the same program running on a 16 core CPU (almost) twice as fast as running on a 8-core. I wish I could do the same in Python, but the GIL prevent's it. (It does work in GIL'less Jython and IronPython though)– renejsumCommented Jan 11, 2018 at 6:12
a + b
?a + b
" block?a + b
is not I/O, it will not block. It will take some time to execute, but so will any other operation that you do.a+b
as a normal arithmetic operation will never block - it will just use CPU until it's completed, so the process/thread that runs it will remain runnable throughout.