I am running test cases for a matlab based program. I have several hundred test cases to run and since each test case uses a single core I have been using multiprocessing, Pool, and map to help do this work in parallel
The program takes command line arguments where I execute a bash script. I have written code which creates a csv file of the bash commands that need to be called for each test case. I read each test case from the csv file into variable testcase_to_run
which creates a set of individual lists (needed in this format to be fed into the map function I believe
I have a 12 core machine so I run (12-1) instances at a time in parallel. I have noticed that with certain test-cases and their arguments not every test case gets run. I am seeing up to 20% of test cases just not being run (bash script first command is to create a new file to store results)
from multiprocessing import Pool
import subprocess
number_to_run_in_parallel = 11
testcase_to_run = ([testcase_1 arguments], [testcase_2 arguments], ....[testcase_250 arguments])
def execute_test_case(work_data):
subprocess.call(work_data, shell=True)
def pool_handler(number_to_run_in_parallel):
p = Pool(number_to_run_in_parallel)
p.map(execute_test_case, test_cases_to_run)
if __name__ == "__main__":
pool_handler(number_to_run_in_parallel)