Surrey Research Computing
Job Management#
Slurm offers many different tools for checking on the Queues and interacting with your jobs on the cluster.
Checking the job queue (squeue)#
You can view your jobs in the queue using the command: squeue or
you could use the command squeue -u <username> to see only jobs you have submitted.
[abc123@login1(eureka2) ~]$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
40143 shared castepjo abc123 R 8:01 1 node11
40072 shared raccoon abc123 R 1-20:07:59 1 node40
40145 shared hello abc123 R 0:01 2 node[102-103] <------ Here is my Job!
40116 shared es254.sh abc126 R 20:49:02 1 node108
40125 shared bash abc127 R 2:06:20 1 node07
34114 shared halo_332 abc131 R 4-11:08:00 4 node[30-33]
40090 gpu rbd_EPR_ abc140 R 1-03:18:31 1 node124
[abc123@login1(eureka2) ~]$ squeue -u abc123
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
40145 shared hello abc123 R 0:11 2 node[102-103]
If you have followed this example you should, have the following outputs in your directory.
[abc123@login1(eureka2) ~]$ ls -ltr
-rw-r--r-- 1 abc123 itsstaff 18084 Apr 17 12:48 helloworld.out
-rw-r--r-- 1 abc123 itsstaff 14 Apr 17 12:48 nodes
You can also query jobs to get the full information about the job using its job id number,
this can be done using the command scontrol show job <Job Id Number> as shown below:
abc123@login1(eureka2) ~]$ scontrol show job 40148
JobId= 40145 JobName=hello
UserId=abc123(282122) GroupId=itsstaff(40000) MCS_label=N/A
Priority=12940 Nice=0 Account=it QOS=normal
JobState=COMPLETED Reason=None Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
RunTime=00:00:01 TimeLimit=00:05:00 TimeMin=N/A
SubmitTime=2019-04-17T13:06:21 EligibleTime=2019-04-17T13:06:21
StartTime=2019-04-17T13:06:21 EndTime=2019-04-17T13:06:22 Deadline=N/A
PreemptTime=None SuspendTime=None SecsPreSuspend=0
Partition=shared AllocNode:Sid=login7:28627
ReqNodeList=(null) ExcNodeList=(null)
NodeList=node[102-103]
BatchHost=node102
NumNodes=2 NumCPUs=20 NumTasks=20 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
TRES=cpu=20,mem=4G,node=2
Socks/Node=* NtasksPerN:B:S:C=10:0:*:* CoreSpec=*
MinCPUsNode=10 MinMemoryNode=2G MinTmpDiskNode=0
Features=ib DelayBoot=00:00:00
Gres=(null) Reservation=(null)
OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
Command=/users/abc123/slurm_test.sh
WorkDir=/users/abc123
StdErr=/users/abc123/hello.out
StdIn=/dev/null
StdOut=/users/abc123/hello.out
Power=
Cancel a job (scancel)#
If you have submitted a job, and want to delete or cancel it you can use the command,
scancel <Job ID Number>.
- scancel <job-id>:
cancel a specific job
- scancel -u <username>:
cancel all your jobs
- scancel -t PENDING -u <username>:
cancel all your jobs in the PENDING status:
- scancel -t RUNNING -u <username>:
cancel all your jobs in the RUNNING status:
- scancel <jobid>_<taskid>:
cancel a specific task in an array job:
- scancel <jobid>_[<taskid>_<taskid>]:
cancel a range of specific tasks in an array job
View cluster information (sinfo)#
sinfo allows you to view information about status of cluster, partitions and node usage.
[abc123@login1(eureka2) ~]$ sinfo
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
shared* up 7-00:00:00 2 down* node[101,136]
shared* up 7-00:00:00 3 resv node[01,43-44]
shared* up 7-00:00:00 32 mix node[02,04,06-08,15-21,32,34,36-40,45,50,109-118,140]
shared* up 7-00:00:00 29 alloc galaxy[1-13],node[09-14,22-25,30-31,35,42,46,49]
shared* up 7-00:00:00 22 idle node[03,05,26-29,33,102-108,120-121,132-135,139,141]
debug_latest up 1:00:00 1 idle node41
debug_all up 1:00:00 1 idle node100
high_mem up 7-00:00:00 8 idle node[119,125-131]
gpu up 7-00:00:00 3 idle node[122-124]
View job stats after job completion (sacct)#
sacct retrieves useful job stats from slurm after the job has finished.
Retrieve information based on job number:
sacct -j <Job Number> --format=User,JobID,Jobname,partition,state,time,start,end,elapsed,MaxRss,MaxVMSize,nnodes,ncpus,nodelist
[abc123@login1(eureka2) ~]$ sacct -j 56814 --format=User,JobID,Jobname,partition,state,time,start,end,elapsed,MaxRss,MaxVMSize,nnodes,ncpus,nodelist
User JobID JobName Partition State Timelimit Start End Elapsed MaxRSS MaxVMSize NNodes NCPUS NodeList
--------- ------------ ---------- ---------- ---------- ---------- ------------------- ------------------- ---------- ---------- ---------- -------- ---------- ---------------
abc123 56814 Hello shared COMPLETED 00:05:00 2019-08-13T14:45:59 2019-08-13T14:46:00 00:00:01 4 112 node[45-46,49-+
56814.batch batch COMPLETED 2019-08-13T14:45:59 2019-08-13T14:46:00 00:00:01 1160K 147756K 1 28 node45
56814.0 pmi_proxy COMPLETED 2019-08-13T14:45:59 2019-08-13T14:46:00 00:00:01 828K 213820K 4 4 node[45-46,49-+
Get information by user and time:
sacct --starttime YYYY-MM-DD -u <username> --format=JobID,state,time,JobName,MaxRSS,Elapsed,nodelist
[abc123@login1(eureka2) ~]$ sacct --starttime 2020-01-17 -u abc123 --format=JobID,state,time,JobName,MaxRSS,Elapsed,nodelist
JobID State Timelimit JobName MaxRSS Elapsed NodeList
------------ ---------- ---------- ---------- ---------- ---------- ---------------
106178 COMPLETED 00:10:00 hello 00:02:00 node45
106178.batch COMPLETED batch 1956K 00:02:00 node45
106179 COMPLETED 00:10:00 hello 00:02:00 node45
106179.batch COMPLETED batch 1908K 00:02:00 node45
Check current cluster utilisation (sview & slurmtop)#
slurmtop is like the linux command ‘top’ but displays the layout of jobs running on compute nodes.
sview Displays a graphical overview of the live (current) cluster utilisation.
