2

I am parallelizing my simulation with Multiprocessing.pool, however i cannot pass a type 'module' in the pool.map which gives pickling error, i need to use that argument in the target function of pool.map to parallelize my code. In the function sample(), the fourth argument 'sim' is of type 'Module', so i cannot pass it with p.map(), since it cannot be iterated, but i need that argument in the function parallel(), which should be used as

       model=sim.simulate(modelname, packname, config)

But currently i am importing that module statically and calling in the function parallel() as

       model=OpenModelica.simulate(modelname, packname, config)

Currently my code looks like this, is there way to declare the argument 'sim' in function sample() as global and access it in the target function parallel().

       def sample(file,model,config,sim,resultDir,deleteDir):
           from multiprocessing import Pool
           p=Pool()
           p.map(parallel,zip(file,model,dirs,resultpath,config))

       def parallel(modellists):
         packname=[]
         packname.append(modellists[0])     
         modelname=modellists[1]
         dirname=modellists[2]
         path=modellists[3]
         config=modellists[4]
         os.chdir(dirname)   
         model=OpenModelica.Model(modelname, packname, config)

1 Answer 1

2

This is because pickle is unable to serialize a module, and thus multiprocessing can't pass the module through the map. However, if you use a fork of multiprocessing called pathos.multiprocessing, it works. This is because pathos uses the dill serializer, which can pickle modules.

>>> import dill 
>>> import numpy
>>> 
>>> from pathos.multiprocessing import ProcessingPool as Pool
>>> p = Pool()
>>> 
>>> def getname(x):
...   return getattr(x, '__package__', None)
... 
>>> p.map(getname, [dill, numpy])
['dill', 'numpy']

It also works for multiple arguments, so it's a bit more natural than having to zip all the arguments -- and has asynchronous and iterative maps as well.

>>> packname = list('abcde')
>>> modelname = list('ABCDE')
>>> dirname = list('12345')
>>> config = [1,2,3,4,5]
>>> import math
>>> f = [math.sin, math.cos, math.sqrt, math.log, math.tan]
>>> 
>>> def parallel(s1, s2, si, i, f): 
...     s = (s1 + s2).lower().count('b')
...     return f(int(si) + i - s) 
... 
>>> res = p.amap(parallel, packname, modelname, dirname, config, f)
>>> print "asynchronous!"
'asynchronous!'
>>> res.get()
[0.9092974268256817, -0.4161468365471424, 2.449489742783178, 2.0794415416798357, 0.6483608274590867]

Get pathos here: https://github.com/uqfoundation

4
  • hi I installed pathos, but i get this error when importing >>> from pathos.multiprocessing import ProcessingPool as Pool "ImportError: No module named multiprocessing", does pathos works in windows
    – arun
    Jan 23, 2015 at 9:23
  • @arun: it should -- however there isn't a windows installer for it... so you have to build it from source, and there are several dependencies. Did you install all of the dependencies? If you find you get errors, you can check against known issues or fill out a bug report here: github.com/uqfoundation/pathos/issues Jan 23, 2015 at 11:51
  • Hi mike, i have installed the pathos dependencies, Now i figured two problem , the pathos works fine in the terminal (cmd prompt) but whenever i tried put the as a python script and run, it does not work, Does it works only on terminal.
    – arun
    Jan 23, 2015 at 14:34
  • @arun: that shouldn't be the case. Can you post a ticket with any example code and your traceback on the link I gave above? Jan 24, 2015 at 15:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.