I'm trying to avoid having to pass variables redundantly into dataList
(e.g. [(1, globalDict), (2, globalDict), (3, globalDict)]
) and use them globally instead. global globalDict
is not a solution to do so in the following code, however.
Is there a straightforward way to access data in a multiprocessing function globally?
I read the following here:
"Communication is expensive. In contrast to communication between threads, exchanging data between processes is much more expensive. In Python, the data is pickled in to binary format before transferring on pipes. Hence, the overhead of communication can be very significant when the task is small. To reduce the extraneous cost, better assign tasks in chunk."
I'm not sure if that would apply here, but I would like to simplify data access in any case.
def MPfunction(data):
global globalDict
data += 1
# use globalDict
return data
if __name__ == '__main__':
pool = mp.Pool(mp.cpu_count())
try:
globalDict = {'data':1}
dataList = [0, 1, 2, 3]
data = pool.map(MPfunction, dataList, chunksize=10)
finally:
pool.close()
pool.join()
pool.terminate()