I need to fetch really big data chunks by iterating over the data. In total I need a few million iterations. So I'd thought subpressing would speed up my process and it almost did. I use subprocess.Queue
to invoke different Threads, which actually works fine, but when I call *subprocess.Queue.get()`the programm takes forever to get the results. Maybe I did something wrong. Here is my minimal example:
def get_losses(self, tags=None):
return_dict = {}
output_list = multiprocessing.Queue()
process_list = []
# Create quese definition
for experiment, path in self.tf_board_dicts.items():
t = multiprocessing.Process(target=self._load_vec_from_tfboard, args=(path, tags, experiment))
process_list.append(t)
print("Starting subprocesse with a total of {} workers. \n These are {}".format(len(process_list),
process_list))
# Run processes
for p in process_list:
p.start()
# Exit the finished threads
for p in process_list:
p.join()
print("All subprocesses are termianted")
# Get results
results = [output_list.get() for p in process_list]
print("All losses are gathered: {}".format([tup[0] for tup in results]))
# Create dict
for experiment_losses in results:
return_dict[experiment_losses[0]] = experiment_losses[1]
return return_dict