-1

I need to run the same script with many different sets of hyperparameters. I have far more sets than CPUs, so I want to do it in parallel, but I want it to schedule one run per CPU. Is there a simple way to do this in Python?

2

1 Answer 1

2

Yes. You can use Python's multiprocessing module; in particular you can use multiprocessing.pool.

Without more details, it's difficult to give a concrete answer. However, assuming the work you want to do is encapsulated by a function called my_function and the parameters are in my_iterable, you can do something like:

from multiprocessing import Pool
with Pool(40) as processing_pool:
        processing_pool.map(my_function, my_iterable)
2
  • But I need to run it 1000 times, not 40. How would I do that without overburdening the CPUs?
    – Sam Lerman
    Jan 13, 2021 at 1:23
  • Also my file isn't a function, it's a python file with parsed arguments. Can I pass a file into multiproc?
    – Sam Lerman
    Jan 13, 2021 at 1:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.