1 / 34

ARC Overview & Queuing Tutorial

ARC Overview & Queuing Tutorial. An overview of Torque/Moab queuing. Topics. ARC topology Authentication Architecture of the queuing system Workflow Job Scripts Some queuing strategies. Network Topology. ARC Authentication. Accounts. Your account is your VT PID

jagger
Download Presentation

ARC Overview & Queuing Tutorial

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ARC Overview & Queuing Tutorial An overview of Torque/Moab queuing

  2. Topics • ARC topology • Authentication • Architecture of the queuing system • Workflow • Job Scripts • Some queuing strategies

  3. Network Topology

  4. ARC Authentication

  5. Accounts Your account is your VT PID Your password is your VT PID password Contact 4help to change your password

  6. Architecture • Resource Manager - Torque • Scheduler - Moab • Allocation Manager - Gold

  7. Account Requests • To request an account: http://www.arc.vt.edu/arc/UserAccounts.php • System X accounts https://portal.arc.vt.edu/allocation/alloc_request.html • To add users to a Hat/Project for System X PI Email arc@vt.edu to ask to have that person added

  8. Queue Architecture

  9. Resource Manager • Torque (Tera-scale Open-source Research and QUEue manager) • Branch of OpenPBS • 2 Parts • Pbs_mom • Daemon on each compute node • Handles job start up and keeps track of the node’s state • Pbs_server • Server that jobs are submitted to. • Keeps track of all nodes and jobs

  10. Moab Scheduler • Takes state information from the resource manager and then schedules jobs to run • “The Brains” • Implements and manages: • Scheduling policies • Dynamic priorities • Reservations • Fairshare

  11. Allocation Manager • Gold • Keeps track of cpu-hours

  12. Workflow From the queuing system point of view When a scheduling interval starts Moab asks pbs_server the state of the nodes and of any jobs Moab attempts to schedule any eligble jobs if there are enough resources free Moab tells pbs_server to schedule start any jobs that can be started Pbs_server contacts the pbs_mom on the first node assigned to the job (That pbs_mom is called the mother superior) The mother superior executes the jobs scripts submitted by the user When a pbs heartbeat happens The pbs_server will contact the pbs_mom and ask the status of its node

  13. Workflow From a user’s point of view • Submit a job script to the queuing system • Wait for the job to be scheduled and run • Get the results

  14. The Queue Queue divided into 3 subqueues: • Active – running • Eligible – idle, but waiting to run • Blocked – idle, held, deferred

  15. Blocked jobs A job can be “blocked” for several reasons: • Requested resources not available • Reserved nodes offline • User already has the maximum number of eligible jobs in the queue • User places intentional hold Moab supports four distinct types of holds: user, system, batch, and deferred

  16. Job Scripts • The job script has a few definitions to inform the queuing system of your job requirements and who you are • Includes environment variables and commands to run your application

  17. Script Definitions • Walltime request • #PBS -lwalltime=hh:mm:ss • CPU request • For System X • #PBS -lnodes=X:ppn=2 • X number of nodes with 2 processors per node • For Cauldron • #PBS -lncpus=X • X number of cores

  18. Script Definitions • Which queue you want to use • #PBS -q <queue name> • queues available now • System X OS X partition: production_q • System X Linux partition: linux_q • Cauldron: cauldron_q • Inferno2: inferno2_q • Ithaca: ithaca_q • Ithaca parallel matlab: pmatlab_q

  19. Script Definitions • Some information about who you are • Your submission group • #PBS -W group_list=<group> • For System X it is tcf_user • For Cauldron it is sgiusers • Type `groups` when logged into a head node to check that you belong to group of the machine you wish to submit to • Your cpu-hour hat • #PBS -A <hat> • On Cauldron it is sgim0000 • System X users were told their hat in their welcome letters.

  20. Job Script Template #!/bin/bash #PBS -lwalltime=01:00:00 #PBS -lncpus=8 #PBS -q cauldron_q #PBS -W group_list=sgiusers #PBS -A sgim0000

  21. Job Script • After the PBS definitions, put in the commands to start your job • There are example job scripts found in /apps/doc(s)

  22. Running Your Job • Use qsub to submit your job to the queue • qsub ./jobscript • To check on your job’s status • qstat -a <queue name> • showq -p <partition name> • OSX, LINUX, or CAULDRON • checkjob <job id number> • cstat (on Cauldron) • To delete a job, use qdel • qdel <job id number>

  23. Check Status To display jobs currently in the queue: -bash-3.1$ showq -p LINUX active jobs------------------------ JOBID USERNAME STATE PROCS REMAINING STARTTIME 176882 jalemkul Running 24 23:46:56 Mon Aug 2 07:11:24 176885 jalemkul Running 24 1:01:37:59 Mon Aug 2 09:02:27 176889 jalemkul Running 24 1:02:21:27 Mon Aug 2 09:45:55 176918 kmsong Running 44 6:14:25:16 Mon Aug 2 16:49:44 176897 kmsong Running 88 15:17:01:30 Tue Aug 3 11:25:58 5 active jobs 118 of 118 processors in use by local jobs (100.00%) 50 of 59 nodes active (84.75%) eligible jobs---------------------- JOBID USERNAME STATE PROCS WCLIMIT QUEUETIME 0 eligible jobs blocked jobs----------------------- JOBID USERNAME STATE PROCS WCLIMIT QUEUETIME 176956 kmsong Idle 112 33:08:00:00 Tue Aug 3 15:15:13 1 blocked job Total jobs: 6

  24. Check Status With qstat: -bash-3.1$ qstat linux_q Job id Name User Time Use S Queue ------------------- ---------------- --------------- -------- - ----- 176882.queue yt42_md1 jalemkul 1229:52: R linux_q 176885.queue yt42_md3 jalemkul 1185:20: R linux_q 176889.queue yt42_md2 jalemkul 1168:07: R linux_q 176897.queue DNS kmsong 00:00:00 R linux_q 176918.queue Re1200_2sec kmsong 1828:24: R linux_q 176956.queue LDNS kmsong 0 Q linux_q Note: status give by R – running and Q – queued

  25. Qstat -f -bash-3.1$ checkjob -v 176956 job 176956 (RM job '176956.queue.arc-int.vt.edu’) AName: LDNS State: Idle Creds: user:kmsong group:tcf_user account:engr1003 class:linux_q qos:sysx_qos WallTime: 00:00:00 of 33:08:00:00 SubmitTime: Tue Aug 3 15:15:13 (Time Queued Total: 1:23:40:11 Eligible: 00:00:19) NodeMatchPolicy: EXACTNODE Total Requested Tasks: 112 Total Requested Nodes: 56 Req[0] TaskCount: 112 Partition: ALL NodeAccess: SINGLEJOB TasksPerNode: 2 UMask: 0000 OutputFile: sysx2.arc-int.vt.edu:/home/kmsong/Turb_channel/Simulation/Re600/176103.queue.arc-int.vt.edu/LDNS.o176956 ErrorFile: sysx2.arc-int.vt.edu:/home/kmsong/Turb_channel/Simulation/Re600/176103.queue.arc-int.vt.edu/LDNS.e176956 BypassCount: 305 Partition List: LINUX,SHARED SrcRM: SystemX DstRM: SystemX DstRMJID: 176956.queue.arc-int.vt.edu Submit Args: -l walltime=800:00:00 -l nodes=56:ppn=2 -Wgroup_list -Aengr1003 -NLDNS -q linux_q -I Flags: INTERACTIVE Attr: INTERACTIVE,checkpoint StartPriority: 200 PE: 112.00

  26. NOTE: job violates constraints for partition OSX (partition OSX not in job partition mask) Node Availability for Partition LINUX -------- available for 2 tasks - n[925,951-958] rejected for State - n[833-1024] NOTE: job req cannot run in dynamic partition LINUX now (insufficient procs available: 18 < 112) NOTE: job violates constraints for partition CAULDRON (partition CAULDRON not in job partition mask) NOTE: job violates constraints for partition INFERNO2 (partition INFERNO2 not in job partition mask) NOTE: job violates constraints for partition TT (partition TT not in job partition mask) NOTE: job violates constraints for partition PECOS (partition PECOS not in job partition mask) NOTE: job violates constraints for partition ITHACA (partition ITHACA not in job partition mask) BLOCK MSG: job 176956 violates active SOFT MAXJOB limit of 2 for class linux_q user (Req: 1 InUse: 2) (recorded at last scheduling iteration)

  27. Queuing Strategies • Queue early, queue often • Queue your jobs up! • Can’t run jobs if they aren’t in the queue • Don’t wait for the queue to get smaller because the job will wait, its waiting anyways! • Possibility for backfill for smaller jobs • Have an accurate walltime • Accurate walltimes will help the queue try to backfill in smaller jobs in between runs of larger jobs, but only if it won’t effect the start time of the next job • Try to queue large jobs before downtimes • If you have a large job that can never seem to have enough cpus available, queue it up before a downtime.

  28. Queue Strategies • The command `showbf` • That command shows cpus available right now, and for how long • Showstart • estimated start time of a job • checkjob -v • Checkpointing • If your code does checkpointing you can exploit backfill, by queuing jobs to fill the small places but maybe not running to completion • Good idea in general, in case of hardware failure

  29. showbf -bash-3.1$ showbf Partition Tasks Nodes StartOffset Duration StartDate --------- ----- ----- ------------ ------------ -------------- ALL 146 43 00:00:00 INFINITY 09:29:01_08/10 OSX 4 2 00:00:00 INFINITY 09:29:01_08/10 LINUX 62 31 00:00:00 INFINITY 09:29:01_08/10 PECOS 8 1 00:00:00 INFINITY 09:29:01_08/10 ITHACA 72 9 00:00:00 INFINITY 09:29:01_08/10

  30. showstart -bash-3.1$ showstart 177165 job 177165 requires 64 procs for 12:00:00 Estimated Rsv based start in 8:47:53 on Tue Aug 10 18:11:40 Estimated Rsv based completion in 20:47:53 on Wed Aug 11 06:11:40 Best Partition: OSX

  31. showstart -bash-3.1$ showstart 64@12:00:00 job 64@12:00:00 requires 64 procs for 12:00:00 Estimated Rsv based start in 8:44:19 on Tue Aug 10 18:11:40 Estimated Rsv based completion in 20:44:19 on Wed Aug 11 06:11:40 Best Partition: OSX

  32. Documentation Torque/PBS and Moab scheduler and job submission documentation: http://www.clusterresources.com/pages/resources/documentation.php

More Related