I'm trying to multiprocess a function being applied to a pandas dataframe using the map function. I can return the result, turn that into a list and create a new column in the data frame using the list, and that happens quickly. If, however, I try to directly change a cell in the dataframe, the function slows down significantly.
Is there a faster way to change a cell directly? Or should I just return a map and change the dataframe column with that map?
from multiprocessing import Pool
def function1(a):
return a**2
def function2(a):
train['result'] = a**2
pool = Pool(processes=4)
result_list = pool.map(function1, train['id'])
results = list(result_list)
train['result'] = results
%timeit pool.map(function1, train['id'])
%timeit pool.map(function2, train['id'])
1 loop, best of 3: 689 ms per loop
1 loop, best of 3: 34.4 s per loop