Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
  1. 31621240Introduction
  2. Example of a Job Script
  3. How to Convert a Task ID to a More Useful Set of Parameters

  4. How to Consolidate Small Jobs into Fewer Larger Jobs Using Job Arrays
  5. Rules for Submitting Job Arrays that use Parallel Environments (like MPI) 

...

one job script with embedded directivesis replaced by two files, a qsub_XXX.sou and a XXX.sh


Code Block
languagebash
titledemo.job
#--------
#$ -q mThC.q
#$ -pe orte 20
#$ -l mres=4G,h_data=4G,h_vmem=4G
#$ -cwd -y j -N demo -o demo.$TASK_ID.log
#$ -t 1-100
#-------
module load some/thing
#-------
echo + `date` job $JOB_NAME started in $QUEUE with jobID=$JOB_ID on $HOSTNAME
echo + taskID=$SGE_TASK_ID
echo + NSLOTS=$NSLOTS distributed over:
cat $PE_HOSTFILE
#
mpirun -np $NSLOTS crunch -i input.$SGE_TASK_ID -o output.$SGE_TASK_ID
#
echo = `date` job $JOB_NAME done



Code Block
languagebash
titleqsub_demo.sou
qsub \
 -q mThC.q \
 -pe orte 20 \
 -l mres=4G,h_data=4G,h_vmem=4G \
 -cwd -y j -N demo -o 'demo.$TASK_ID.log' \
 -t 1-100 \
 -b y $PWD/demo.sh

(warning) no spaces after the '\'


Code Block
languagebash
titledemo.sh
#!/bin/sh
# any embedded directives here will be ignored
#-------
source /etc/profile.d/modules.sh
module load some/thing
#-------
echo + `date` job $JOB_NAME started in $QUEUE with jobID=$JOB_ID on $HOSTNAME
echo + taskID=$SGE_TASK_ID
echo + NSLOTS=$NSLOTS distributed over:
cat $PE_HOSTFILE
#
mpirun -np $NSLOTS crunch -i input.$SGE_TASK_ID -o output.$SGE_TASK_ID
#
echo = `date` job $JOB_NAME done

(lightbulb) this This can be any type of executable script, the line "but if you use:

  • #!/bin/sh or #!/bin/bash, you'll need source /etc/profile.d/modules.sh"
is bash/sh specific tho
  • to access modules,
  • #!/bin/sh -l or #!/bin/bash -l, module will be defined, but the script will read ~/.profile,
  • #!/bin/csh or #!/bin/tcsh, module will be defined,
  • #!/bin/csh -f or #!/bin/tcsh -f, you'll need source /etc/profile.d/modules.csh to access modules.
    (wink) Don't you love Linux? (It's all in the man pages, tho).

Before submitting the job array, make sure the script is executable:

...