I am trying to optimize my code using Python's multiprocessing.Pool
module, but I am not getting the speed-up results that I would logically expect.
The main method I am doing involves calculating matrix-vector products for a large number of vectors and a fixed large sparse matrix. Below is a toy example which performs what I need, but with random matrices.
import time
import numpy as np
import scipy.sparse as sp
def calculate(vector, matrix = None):
for i in range(50):
v = matrix.dot(vector)
return v
if __name__ == '__main__':
N = 1e6
matrix = sp.rand(N, N, density = 1e-5, format = 'csr')
t = time.time()
res = []
for i in range(10):
res.append(calculate(np.random.rand(N), matrix = matrix))
print time.time() - t
The method terminates in about 30
seconds.
Now, since the calculation of each element of results
does not depend on the results of any other calculation, it is natural to think that paralel calculation will speed up the process. The idea is to create 4 processes and if each does some of the calculations, then the time it takes for all the processes to complete should decrease by some factor around 4
. To do this, I wrote the following code:
import time
import numpy as np
import scipy.sparse as sp
from multiprocessing import Pool
from functools import partial
def calculate(vector, matrix = None):
for i in range(50):
v = matrix.dot(vector)
return v
if __name__ == '__main__':
N = 1e6
matrix = sp.rand(N, N, density = 1e-5, format = 'csr')
t = time.time()
input = []
for i in range(10):
input.append(np.random.rand(N))
mp = partial(calculate, matrix = matrix)
p = Pool(4)
res = p.map(mp, input)
print time.time() - t
My problem is that this code takes slightly above 20
seconds to run, so I did not even improve performance by a factor of 2
! Even worse, the performance does not improve even if the pool contains 8
processes! Any idea why the speed-up is not happening?
Note: My actual method takes much longer, and the input vectors are stored in a file. If I split the file in 4
pieces and then run my script in a separate process for each file manually, each process terminates four times as quickly as it would for the whole file (as expected). I am confuzed why this speed-up (which is obviously possible) is not happening with multiprocessing.Pool
Edi: I have just found Multiprocessing.Pool makes Numpy matrix multiplication slower this question which may be related. I have to check, though.
4
cores. That is why I manually split the file into4
, not8
pieces. – 5xum Oct 16 '14 at 8:21time.time()
benchmarks inside yourcalculate
method, you'll see the 50dot
calls take nearly 4 times longer than they do in the non-parallel case. It's not clear to me why because tools liketop
make it the non-parallel case is only using one CPU fully, whereas the parallel case makes it look like 4 CPUs are being fully used. – Amit Kumar Gupta Oct 16 '14 at 9:17numpy
when I use multiple processes. – 5xum Oct 16 '14 at 9:19numpy
out of the equation and benchmark. Here's a pastebin with parallel and serial implementations that just do a bunch of arithmetic: pastebin.com/B3M6GZb8. To make them more similar, the series version uses the built-inmap
to contrast with thep.map
in the parallel case. On my machine (with 4 CPU), the series case takes about1.8s
per calculate for a total of about22.8s
. The parallel cases for 1-4 workers take on average1.75s
,2.6s
,2.6s
and3.3s
per calculate, respectively, for totals of about21s
,16s
,11.0s
, and11.3s
respectively. – Amit Kumar Gupta Oct 16 '14 at 9:41