6

I have a python script main_script.py that looks like this:

import os

Files = os.listdir(os.path.join(path, "."))
FilesNumber = len(Files)

for fileID in range (0,FilesNumber):
    filename = Files[fileID]

    # load file specified in filename and do stuff

basically it does the same kind of operations for each file in the variable Files

I would like to use qsub to parallelize the for loop.

Assuming that I have a txt file files.txt containing all the files names:

//mypath//pathfile1
//mypath//pathfile2
...
//mypath//pathfile100

how can I write the shell script that calls qsub and runs main_script.py I think that I would also need to adapt main_script.py but I do not know how...

The scheduler is Torque/Maui

3
  • Which scheduler are you using? The command qsub is used by several of them.
    – eduffy
    Commented Mar 7, 2017 at 13:15
  • I should use Torque/Maui
    – gabboshow
    Commented Mar 7, 2017 at 13:16
  • Personal plug on this one: you can use fyrd (fyrd.science) to submit functions to torque or slurm from python, it is built for exactly this kind of thing.
    – Mike D
    Commented Oct 31, 2017 at 3:40

1 Answer 1

9

One way to call any executable from a job script is to simply wrap it inside a bash script:

#/bin/bash

<full path to call executable>

If you name that script script.sh, and script.sh is executable, then you can execute:

qsub script.sh

and it will be submitted to the batch system. The gotchas - which you may well already know - are things like: if your executable isn't accessible from the compute node, then it won't be found when the job executes. The same is true for files that your script is using, so you'll want to make sure they're all located appropriately, usually a network-accessible filesystem.

If you wanted to directly submit the python script, you can add:

#!/usr/bin/python 

to the top (double-check that python is in /usr/bin on your system) and then you can directly qsub your python script. In your case,

qsub main_script.py

When submitted this way, the script no longer has to be in a network-accessible location, but the input files still do.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.