7

In my project, I have a long list of trades to be priced so I break it into chunks and delegate them to multiprocessing#Pool to process. Let's say the wrapping class is called Valuation, the method is pool_calculate(), which looks like:

    multiprocessing.freeze_support() 
    p = multiprocessing.Pool()  # default is a number of processes equal to the number of CPU cores
    results = p.map(self.pooled_valuation_tasks, chunks)
    p.close()
    p.join()

The problem is that we have to import an in-house module, say, magic, for all pricing.

import magic
class Valuation(object):
    ....

It seemed that the module is not propagated into the pool workers, the error messages look like:

Process SpawnPoolWorker-3:
Traceback (most recent call last):
  File "c:\opt\magic0.0.46\PythonVenv\lib\multiprocessing\process.py", line 258, in _bootstrap
    self.run()
  File "c:\opt\magic0.0.46\PythonVenv\lib\multiprocessing\process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "c:\opt\magic0.0.46\PythonVenv\lib\multiprocessing\pool.py", line 108, in worker
    task = get()
  File "c:\opt\magic0.0.46\PythonVenv\lib\multiprocessing\queues.py", line 337, in get
    return _ForkingPickler.loads(res)
ModuleNotFoundError: No module named 'magic.rec' 

anyone can help?

Update:

To provide more details in response to some questions from the comments:

this is how I instantiate the class and call the func:

if __name__ == '__main__':
   valu = Valuation(...)
   valu.pool_calculate(...)
1
  • It's tough to answer this without a complete example to reproduce the problem. A couple of thoughts: Are you importing magic at the top-level of the module? Are you protecting the creation of the multiprocessing.Pool() with a if __name__ == "__main__": guard?
    – dano
    Commented Aug 27, 2019 at 15:24

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.