0

I'm trying to multiprocess a function being applied to a pandas dataframe using the map function. I can return the result, turn that into a list and create a new column in the data frame using the list, and that happens quickly. If, however, I try to directly change a cell in the dataframe, the function slows down significantly.

Is there a faster way to change a cell directly? Or should I just return a map and change the dataframe column with that map?

from multiprocessing import Pool

def function1(a):
    return a**2

def function2(a):
    train['result'] = a**2 

pool            = Pool(processes=4)
result_list     = pool.map(function1, train['id'])
results         = list(result_list)
train['result'] = results


%timeit pool.map(function1, train['id'])
%timeit pool.map(function2, train['id'])

1 loop, best of 3: 689 ms per loop
1 loop, best of 3: 34.4 s per loop
8
  • How do you think you benefits from have 4 processes adjusting the SAME DataFrame column?
    – Merlin
    Commented Jul 4, 2016 at 4:11
  • Each row of the dataframe contains an integer sequence which I'll by applying a few different non trivial functions to, eg linear regression. Their output will sometimes be added to another column of the dataframe. They're all independent, so I assume that running multiple parallel processes will be faster than calculating them sequentially in one process.
    – mpotma
    Commented Jul 4, 2016 at 4:39
  • Are you on Windows.mac or linux?
    – Merlin
    Commented Jul 4, 2016 at 4:41
  • Linux Mint 17, although I could run in Windows
    – mpotma
    Commented Jul 4, 2016 at 4:49
  • No, dont ever run mutliprocess on Windows. The OS does not have .fork() which make you effort a waste.
    – Merlin
    Commented Jul 4, 2016 at 4:51

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.