I need to run the same script with many different sets of hyperparameters. I have far more sets than CPUs, so I want to do it in parallel, but I want it to schedule one run per CPU. Is there a simple way to do this in Python?
1 Answer
Yes. You can use Python's multiprocessing
module; in particular you can use multiprocessing.pool
.
Without more details, it's difficult to give a concrete answer. However, assuming the work you want to do is encapsulated by a function called my_function
and the parameters are in my_iterable
, you can do something like:
from multiprocessing import Pool
with Pool(40) as processing_pool:
processing_pool.map(my_function, my_iterable)
-
But I need to run it 1000 times, not 40. How would I do that without overburdening the CPUs? Jan 13, 2021 at 1:23
-
Also my file isn't a function, it's a python file with parsed arguments. Can I pass a file into multiproc? Jan 13, 2021 at 1:27