code-notes/slurm_commands at master · novigit/code-notes · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
SLURM COMMANDS

DIFFERENT COMPUTER CLUSTERS
Be aware that different clusters can run different versions of SLURM.
Beskow runs 19.05.7, while Graham runs 20.02.4

MODULE
module list				Show all currently loaded modules
module avail				Lists all the software modules available on Beskow
module load <module>			Load module
module unload <module>			Unload module
module purge  				Unload all loaded modules
module keyword <word>			Search all module descriptions for keyword <word>

SBATCH
sbatch -e <out>				Captures the STDERR output
sbatch -o <out>				Captures the STDOUT output
sbatch -n <number>			Set number of cores to use
sbatch -t <days-hours:min:seconds>	Set time limit
sbatch -VALUES				Check sbatch version

SCANCEL
scancel <jobid>				Cancels job with <jobid>

SQUEUE
squeue -u jmartijn			Check the job queue and estimated start time of your jobs or how long it has been running
squeue -u jmartijn -O "reason:40"	Check the reason why your job has not started running yet

SACCT
sacct					Alternative to squeue, more clearly states status of submitted jobs (CANCELLED and PENDING)
sacct --format "JobID,JobName%30"	Prints up to 30 characters of your submitted jobnames

VIEW COMPLETED JOBS HISTORY
sacct -S YYYY-MM-DD -u jmartijn --format=User,JobID,Jobname,partition,state,time,start,end,elapsed,MaxRss,MaxVMSize,nnodes,ncpus,nodelist
-S / --starttime
-E / --endtime

sacct -e				Printout of all the different fields you can use in --format=FIELD

User				The user name of the user who ran the job
JobID				Identification number of the job
JobName				The name of the job
State				Job status.
						P or PENDING
						R or RUNNING
						F or FAILED
						CD or COMPLETED
						CA or CANCELLED
						OOM or OUT_OF_MEMORY
Timelimit			The requested time limit of the job
Start				The timepoint that the job started
End					The timepoint that the job ended
Elapsed				The wall-clock time / real-life time of the job
MaxRSS				Maximum resident set size of the job. Equal to maximum occupied memory used not in swap or filesystem
AveRSS				Average resident set size of the job. Equal to average occupied memory used not in swap or filesystem


SEFF
seff <jobid>			Shows amongst others runtime, CPU efficiency, memory utilized etc

SSHARE
sshare -a					Display usage and shares for everyone in the project
sshare -A					Project ID
sshare -l					Display additional fields

Top line: Usage/Shares of project relative to all other projects
Other lines: Usage/Shares of project members relative to other members

COMMON ERRORS
sbatch: error: Batch job submission failed: Requested node configuration is not available
Some users experience this error when using "#SBATCH -n <number>".
Use this instead:
#SBATCH --nodes=<number/32>
#SBATCH --ntasks-per-node=32

REASONS
AssocGrpCPURunMinsLimit			Running this job would put the user’s group beyond its maximum number of CPU minutes allocated to currently running jobs
Resources				The job is waiting for resources to become available