Slurm job arrays provide a simple way to submit a large number of independent processing jobs. For example, Slurm job arrays can be useful for applying the same or similar computation to a collection of data sets.
The following examples demonstrate the use of slurm job arrays.
EXAMPLE1 #!/bin/sh #SBATCH --job-name="array_job1" #SBATCH --array=1-3 #SBATCH --ntasks=1 #SBATCH --mem=2000 echo "This is job #"$SLURM_ARRAY_TASK_ID" of the array "$SLURM_ARRAY_JOB_ID >> results.txt exit 0
The file results.txt might be as follows
This is job job #2 of the array 1250 This is job job #1 of the array 1250 This is job job #3 of the array 1250
In the following example 5 single core are spawned and submitted. At each job the same single core command is executed but with a different input file. Input files are named file1, file2 etc.
EXAMPLE2 #!/bin/sh #SBATCH --job-name="array_job2" #SBATCH --array=1-5 #SBATCH --ntasks=1 #SBATCH --mem=2000 single_core_command "file"$SLURM_ARRAY_TASK_ID exit 0
The third example demonstrates how output files can be named and how the environment variable $SLURM_ARRAY_TASK_ID can be manipulated so as different values of job id’s to be available (related to the example1).
EXAMPLE3 #!/bin/sh #SBATCH --job-name="array_job3" #SBATCH --array=1-7:2 #SBATCH --output=outfile%A_%a #SBATCH --ntasks=1 #SBATCH --mem=2000 echo $SLURM_ARRAY_TASK_ID exit 0
For an array job id 500 the output files will be outfile500_1, outfile500_3,outfile500_5, and outfile500_7. Their content will be 1,3,5 and 7 correspondingly.
Users are prompted to be carefull with the use of job arrays. The limitation of maximum 160 array job tasks is enforced.
For more details on the use of job arrays, users can see here.