Resource Management and Balancing

1. Resource Managementand Balancing ECI, July 2005

2. RMS – Overview • Resource management • Job management • Monitoring • Resource balancing • Information dissemination

3. Job Management • The need • Operating system offers job and resource management service for a single computer • The batch job control on multi-user mainframes was performed outside the operating system • Main advantages are: • Structured resource utilization planning and control • Abstraction, easy-to-{understand,use} for user • Provide a vendor independent user interface

4. Manager vs. Scheduler • Resource manager • Locating and allocate resources • Authentication • Process creation and migration • Resource scheduler • Queuing applications • Drive manager (enforce policy)

5. Job Management - Requirements • A typical job management system offers • Heterogeneous support • Batch support • Parallel support • Interactive support • Checkpointing and process migration • Load balancing • Job run-time limits • GUI

6. RMS Architecture • Prerequisites • Multi-user & multitasking capabilities • Homogeneous OS are not a restriction • In practice • “Similar” operating systems run on all machines • UNIX (in all variants) is very customary in the context of using RMS

7. Resource Description • Requirements • Easy to generate simple description • Powerful to generate complex description • Portable representation • Attributed components • RDL: Language to specify resources • Administrator: describe what’s available • User: describe what’s required • Hierarchical

8. RDL Example • A 1024 nodes transputer with unix front-end BEGIN PROC Frontend DECLARATION { PROC; OS=Unix; Repeat = 4; } END PROC CONNECTION FOR I = 0 to 3 DO Backend LINK i  Frontend LINK i OD END PROC DECLARATION BEGIN PROC Transputer DYNAMIC; EXCLUSIVE; DECLARATION BEGIN PROC Backend DECLARATION { PROC; CPU=T8; MEMORY=4; SPEED=30; REPEAT=1024; } { PORT; REPEAT=4; } END PROC

9. RMS Components • User interface • At the minimum - command line user interface • GUI becoming indispensable • Typical commands • Job submission to register for execution • status display to monitor progress or failure of a job • Job deletion to cancel jobs no longer needed

10. RMS Components (contd) • Administrative environment • Specify nodes characteristics • Define feasible job classes and map to hosts • Define user access permissions • Specify resource limitations for users and jobs • Specify policies for the assignment of jobs • Control and ensure proper operation of the RMS • Analyze accounting data to tune the system

11. RMS Entities • Queues • Queues bound to hosts, jobs assigned to queues • Hosts • Compute hosts, control hosts • Users • Capabilities, permissions, priorities • Jobs • Resources • Policies

12. RMS Entities – Jobs • Job: collection of computational tasks • A single program, or several interacting programs • In the context of RMS • Batch Jobs: require no manual interaction as soon as started • Interactive Jobs: require input during runtime • Parallel Jobs: subtasks spread across several hosts in a cluster • Check-pointing Jobs: periodically save status to the file system and can be aborted anytime

13. RMS Entities – Jobs • Batch jobs • Dispatch jobs according to policy and availability • Suspend/Resume & checkpoint/restart • Interactive jobs • Need to maintain a terminal connection • “Watchdog” monitor withdraw from pool • Parallel jobs • Need to integrate with parallel environment • Scheduling policy is more complex

14. RMS Entities - Resources • Available memory, CPU time, network bandwidth, and peripheral devices, licenses • Jobs declare resource requirements • RMS enforces resource consumption • ensures quality of service • prevents over-subscription • detects over-usage

15. RMS Entities - Policies • Abstract mechanisms to automate control • imbalanced load is common in clusters • important/urgent work starved • unauthorized users may take advantage • users may exceed desired resource usage over time • Resource Utilization Policies • Monitor resource consumption • Dispatch of new jobs • Scheduling Policies • Dispatch of new jobs • Relocation of jobs

16. Resource Utilization Policies • Share based • Resource “credit” is assigned to users, depts… • Hierarchical share tree defines sharing • Establish entitlements within time frame • Fair distribution of resources • Functional • Assignment by functional importance (priority) • Past usage is not taken into account • Deadline • Time-critical applications • Manual override • Administrators like power…

17. Scheduling Policies • Dispatch time – who, where • First-Come-First-Served • Select-Least-Loaded • Select-Fixed-Sequence • Combinations above • Relocation – who, when, where • Dynamic resource balancing

18. Scheduling of Parallel Processes • Gang scheduling • Requires tight-coupling (MPP’s) • Co-scheduling • Demand-based • False priority • Concurrent applications • Implicit • Busy wait to not relinquish cpu

19. RMS Challenges • Open Interfaces • Export load balancing/distributed capabilities • Export status info (load, job status, queues) • Control/assistance from application • Integration with other environments (MPI) • Extend functionality for special cases • API must be: simple, usable, abstract, robust

20. RMS Example: CODINE • CODINE/GRD • cod_qmaster: master daemon • cod_schedd: scheduler daemon • cod_execd: execution daemon • Continuously match utilization with policies • GRD monitors and adjusts resource usage correlated to all processes of a job • Feedback to adjust shares towards changing requirements

21. Static Scheduling Scheme

22. Dynamic Scheduling Scheme

23. RMS Example: PBS • Portable Batch Sysetm • Scheduler – job to node mapping, queues • Server – communications, logs • Control daemon (per node) – executive agent • Scope – single node • Job arrays • Task Management interface

24. RMS Example: Condor • Condor: a distributed job scheduler • Harvest idle workstations • Job scheduling and migration • Advertising mechanism • Both job and W/S advertise presence • Jobs advertise requirements (job description file) • W/S advertise their capabilities

25. Condor: Example JDF universe = vanilla # select runtime environment executable = some_job requirements = (Arch=="INTEL" && OpSys=="LINUX") rank = (Memory * 10000) + KFlops #target arguments = -verbose input = in.dat # redirect to stdin output = out.dat # redirect to stdout log = log.txt Queue # add job to queue

26. RMS: Condor (contd) • Universe • Vanilla: sequential apps (shared FS) • MPI, PVM: integrated with parallel environment • Globus: grid computing environment • Standard: enables process migration • Process migration • Reschedule higher priority job • User reclaims her W/S • Must be linked with a special library

27. RMS: Condor (contd) • Access to data • Shared file system • Condor file transfer mechanism • Automatically prefetch, postfetch • Remote I/O calls (in standard universe) • Architecture • Central manager • Server on each node

28. Known Condor Pools

29. Monitoring • Design choices • Centralized  decentralized • Periodic  request driven • Flat  hierarchical • Resolution of information • Focused view

30. Monitoring Example: Parmon • Features: • Online creation of Node and Group database • Component, Node, Group, or entire Cluster level • Monitoring of CPU, memory, disk and network, processes, log files etc • Facility to define events & automatic notification • Misc: message broadcast, remote admin, GUI

31. Load Balancing • Application  system • Static  dynamic  adaptive • Centralized  decentralized • Receiver initiated  sender initiated • Parallel applications • On-line nature

32. LB: Application Level • Application level • Round robin • Randomized • Recursive bisection • Other optimization • Hard to estimate execution times • Indeterminate no. of steps • Unpredictable load • Communication delays

33. LB: System Level • System level • Round robin • Randomized • Estimate run time • Specified by job description • Estimate from past experience

34. LB Example: MOSIX • Decentralized • Symmetric • Deterministic • Responsive • Stable • Competitive • Resources: CPU, memory, I/O

35. Load Balancing Over Network • Distribute workload or network traffic load across the cluster • Nodes may be interconnected among themselves • Must be connected to the balancing device • Processing nodes provide status information • current processor load • the application system load • number of active users • the availability of network protocol buffers • other specific resources

36. Load Balancing Over Network • Balancing device • monitors the status of all nodes • dictates where to direct the next job • a single unit or a group in tree hierarchy • use one or more algorithms or methods • static or dynamic setting • Decide which node gets the next incoming connection request

37. Factors in Network Balancing • Wire-speed processing • Node operating system limitation • Packet processing, no. of connections, interrupts • Balancing device limitations • Tables, memory • Session based traffic, non-session UDP • Application dependencies (affinity)

38. Simple Balancing Methods • Weighting • Assign weights to the nodes of different capacities • Randomization • Works good in identical node environment • Round-Robin • Commonly used by itself in DNS (address caching) • Effective where all the nodes in the cluster are identical in capacity and performance • Hashing • Packets from the same source address will always get assigned to the same server

39. Simple Balancing Methods • Least Connections • Assigns to the node which currently has the least connections ( ≠ least load ) • Minimum Misses • Assign to the nodes which has processed the least number of incoming request in its history • Fastest Response • Assigns to the node with the fastest response • Requires active monitoring of the individual nodes • Sending ICMP packets with the ‘ping’ command • Proprietary mechanism based upon UDP packets

40. Advanced Balancing Methods • Primary optimization vectors • Node traffic – predict volume of traffic • Network traffic – monitor node state • Node-load based balancing – (which load ?) • DNS load balancing - simple • Topology-based – reduce latency • Application-specific performance • Policy based optimization • Application ,bandwidth, admin, security

41. Common Errors • There are four common errors • Overflow • Underflow • Routing errors • Induced network errors • May destabilize efficient network clustering

42. Information Dissemination • Central  Decentralized • Load incurred on system • Processing load • Network load • Partial knowledge – gossip algorithms • Example: finding average load