1

I'm using map_async with processes that return a ton of data. The normal map_async results in the data being stored in memory, then returned when everything is processed. To get around this, I've used a generator approach from:

Combining itertools and multiprocessing?

However, this doesn't make full use of multi-threading (as in, if you have 29 threads finished and 1 thread hanging, it won't start the next batch of jobs until everyone is done). Is there a way to have the map_async or does there exist a similar function which will send its returns to a callback function as each thread finishes?

2
  • not clear if you're using threads or sub processes?
    – shx2
    Apr 11, 2013 at 17:52
  • I'm using subprocesses. I think using apply_async though is what I was looking for.
    – Chrismit
    Apr 12, 2013 at 3:57

1 Answer 1

0

What you want is to use a producer-consumer-based solution. The producer puts tasks in a multiprocessing.Queue, and the consumers (subprocesses) get and processes them, in a loop.

This is a good SO question with a (detailed) possible solution.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.