In my project, I have a long list of trades to be priced so I break it into chunks and delegate them to multiprocessing#Pool to process. Let's say the wrapping class is called Valuation, the method is pool_calculate(), which looks like:
multiprocessing.freeze_support()
p = multiprocessing.Pool() # default is a number of processes equal to the number of CPU cores
results = p.map(self.pooled_valuation_tasks, chunks)
p.close()
p.join()
The problem is that we have to import an in-house module, say, magic, for all pricing.
import magic
class Valuation(object):
....
It seemed that the module is not propagated into the pool workers, the error messages look like:
Process SpawnPoolWorker-3:
Traceback (most recent call last):
File "c:\opt\magic0.0.46\PythonVenv\lib\multiprocessing\process.py", line 258, in _bootstrap
self.run()
File "c:\opt\magic0.0.46\PythonVenv\lib\multiprocessing\process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "c:\opt\magic0.0.46\PythonVenv\lib\multiprocessing\pool.py", line 108, in worker
task = get()
File "c:\opt\magic0.0.46\PythonVenv\lib\multiprocessing\queues.py", line 337, in get
return _ForkingPickler.loads(res)
ModuleNotFoundError: No module named 'magic.rec'
anyone can help?
Update:
To provide more details in response to some questions from the comments:
this is how I instantiate the class and call the func:
if __name__ == '__main__':
valu = Valuation(...)
valu.pool_calculate(...)
magic
at the top-level of the module? Are you protecting the creation of themultiprocessing.Pool()
with aif __name__ == "__main__":
guard?