1 / 42

Resource Management and Balancing

Resource Management and Balancing. ECI, July 2005. RMS – Overview. Resource management Job management Monitoring Resource balancing Information dissemination. Job Management. The need Operating system offers job and resource management service for a single computer

jodie
Download Presentation

Resource Management and Balancing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Resource Managementand Balancing ECI, July 2005

  2. RMS – Overview • Resource management • Job management • Monitoring • Resource balancing • Information dissemination

  3. Job Management • The need • Operating system offers job and resource management service for a single computer • The batch job control on multi-user mainframes was performed outside the operating system • Main advantages are: • Structured resource utilization planning and control • Abstraction, easy-to-{understand,use} for user • Provide a vendor independent user interface

  4. Manager vs. Scheduler • Resource manager • Locating and allocate resources • Authentication • Process creation and migration • Resource scheduler • Queuing applications • Drive manager (enforce policy)

  5. Job Management - Requirements • A typical job management system offers • Heterogeneous support • Batch support • Parallel support • Interactive support • Checkpointing and process migration • Load balancing • Job run-time limits • GUI

  6. RMS Architecture • Prerequisites • Multi-user & multitasking capabilities • Homogeneous OS are not a restriction • In practice • “Similar” operating systems run on all machines • UNIX (in all variants) is very customary in the context of using RMS

  7. Resource Description • Requirements • Easy to generate simple description • Powerful to generate complex description • Portable representation • Attributed components • RDL: Language to specify resources • Administrator: describe what’s available • User: describe what’s required • Hierarchical

  8. RDL Example • A 1024 nodes transputer with unix front-end BEGIN PROC Frontend DECLARATION { PROC; OS=Unix; Repeat = 4; } END PROC CONNECTION FOR I = 0 to 3 DO Backend LINK i  Frontend LINK i OD END PROC DECLARATION BEGIN PROC Transputer DYNAMIC; EXCLUSIVE; DECLARATION BEGIN PROC Backend DECLARATION { PROC; CPU=T8; MEMORY=4; SPEED=30; REPEAT=1024; } { PORT; REPEAT=4; } END PROC

  9. RMS Components • User interface • At the minimum - command line user interface • GUI becoming indispensable • Typical commands • Job submission to register for execution • status display to monitor progress or failure of a job • Job deletion to cancel jobs no longer needed

  10. RMS Components (contd) • Administrative environment • Specify nodes characteristics • Define feasible job classes and map to hosts • Define user access permissions • Specify resource limitations for users and jobs • Specify policies for the assignment of jobs • Control and ensure proper operation of the RMS • Analyze accounting data to tune the system

  11. RMS Entities • Queues • Queues bound to hosts, jobs assigned to queues • Hosts • Compute hosts, control hosts • Users • Capabilities, permissions, priorities • Jobs • Resources • Policies

  12. RMS Entities – Jobs • Job: collection of computational tasks • A single program, or several interacting programs • In the context of RMS • Batch Jobs: require no manual interaction as soon as started • Interactive Jobs: require input during runtime • Parallel Jobs: subtasks spread across several hosts in a cluster • Check-pointing Jobs: periodically save status to the file system and can be aborted anytime

  13. RMS Entities – Jobs • Batch jobs • Dispatch jobs according to policy and availability • Suspend/Resume & checkpoint/restart • Interactive jobs • Need to maintain a terminal connection • “Watchdog” monitor withdraw from pool • Parallel jobs • Need to integrate with parallel environment • Scheduling policy is more complex

  14. RMS Entities - Resources • Available memory, CPU time, network bandwidth, and peripheral devices, licenses • Jobs declare resource requirements • RMS enforces resource consumption • ensures quality of service • prevents over-subscription • detects over-usage

  15. RMS Entities - Policies • Abstract mechanisms to automate control • imbalanced load is common in clusters • important/urgent work starved • unauthorized users may take advantage • users may exceed desired resource usage over time • Resource Utilization Policies • Monitor resource consumption • Dispatch of new jobs • Scheduling Policies • Dispatch of new jobs • Relocation of jobs

  16. Resource Utilization Policies • Share based • Resource “credit” is assigned to users, depts… • Hierarchical share tree defines sharing • Establish entitlements within time frame • Fair distribution of resources • Functional • Assignment by functional importance (priority) • Past usage is not taken into account • Deadline • Time-critical applications • Manual override • Administrators like power…

  17. Scheduling Policies • Dispatch time – who, where • First-Come-First-Served • Select-Least-Loaded • Select-Fixed-Sequence • Combinations above • Relocation – who, when, where • Dynamic resource balancing

  18. Scheduling of Parallel Processes • Gang scheduling • Requires tight-coupling (MPP’s) • Co-scheduling • Demand-based • False priority • Concurrent applications • Implicit • Busy wait to not relinquish cpu

  19. RMS Challenges • Open Interfaces • Export load balancing/distributed capabilities • Export status info (load, job status, queues) • Control/assistance from application • Integration with other environments (MPI) • Extend functionality for special cases • API must be: simple, usable, abstract, robust

  20. RMS Example: CODINE • CODINE/GRD • cod_qmaster: master daemon • cod_schedd: scheduler daemon • cod_execd: execution daemon • Continuously match utilization with policies • GRD monitors and adjusts resource usage correlated to all processes of a job • Feedback to adjust shares towards changing requirements

  21. Static Scheduling Scheme

  22. Dynamic Scheduling Scheme

  23. RMS Example: PBS • Portable Batch Sysetm • Scheduler – job to node mapping, queues • Server – communications, logs • Control daemon (per node) – executive agent • Scope – single node • Job arrays • Task Management interface

  24. RMS Example: Condor • Condor: a distributed job scheduler • Harvest idle workstations • Job scheduling and migration • Advertising mechanism • Both job and W/S advertise presence • Jobs advertise requirements (job description file) • W/S advertise their capabilities

  25. Condor: Example JDF universe = vanilla # select runtime environment executable = some_job requirements = (Arch=="INTEL" && OpSys=="LINUX") rank = (Memory * 10000) + KFlops #target arguments = -verbose input = in.dat # redirect to stdin output = out.dat # redirect to stdout log = log.txt Queue # add job to queue

  26. RMS: Condor (contd) • Universe • Vanilla: sequential apps (shared FS) • MPI, PVM: integrated with parallel environment • Globus: grid computing environment • Standard: enables process migration • Process migration • Reschedule higher priority job • User reclaims her W/S • Must be linked with a special library

  27. RMS: Condor (contd) • Access to data • Shared file system • Condor file transfer mechanism • Automatically prefetch, postfetch • Remote I/O calls (in standard universe) • Architecture • Central manager • Server on each node

  28. Known Condor Pools

  29. Monitoring • Design choices • Centralized  decentralized • Periodic  request driven • Flat  hierarchical • Resolution of information • Focused view

  30. Monitoring Example: Parmon • Features: • Online creation of Node and Group database • Component, Node, Group, or entire Cluster level • Monitoring of CPU, memory, disk and network, processes, log files etc • Facility to define events & automatic notification • Misc: message broadcast, remote admin, GUI

  31. Load Balancing • Application  system • Static  dynamic  adaptive • Centralized  decentralized • Receiver initiated  sender initiated • Parallel applications • On-line nature

  32. LB: Application Level • Application level • Round robin • Randomized • Recursive bisection • Other optimization • Hard to estimate execution times • Indeterminate no. of steps • Unpredictable load • Communication delays

  33. LB: System Level • System level • Round robin • Randomized • Estimate run time • Specified by job description • Estimate from past experience

  34. LB Example: MOSIX • Decentralized • Symmetric • Deterministic • Responsive • Stable • Competitive • Resources: CPU, memory, I/O

  35. Load Balancing Over Network • Distribute workload or network traffic load across the cluster • Nodes may be interconnected among themselves • Must be connected to the balancing device • Processing nodes provide status information • current processor load • the application system load • number of active users • the availability of network protocol buffers • other specific resources

  36. Load Balancing Over Network • Balancing device • monitors the status of all nodes • dictates where to direct the next job • a single unit or a group in tree hierarchy • use one or more algorithms or methods • static or dynamic setting • Decide which node gets the next incoming connection request

  37. Factors in Network Balancing • Wire-speed processing • Node operating system limitation • Packet processing, no. of connections, interrupts • Balancing device limitations • Tables, memory • Session based traffic, non-session UDP • Application dependencies (affinity)

  38. Simple Balancing Methods • Weighting • Assign weights to the nodes of different capacities • Randomization • Works good in identical node environment • Round-Robin • Commonly used by itself in DNS (address caching) • Effective where all the nodes in the cluster are identical in capacity and performance • Hashing • Packets from the same source address will always get assigned to the same server

  39. Simple Balancing Methods • Least Connections • Assigns to the node which currently has the least connections ( ≠ least load ) • Minimum Misses • Assign to the nodes which has processed the least number of incoming request in its history • Fastest Response • Assigns to the node with the fastest response • Requires active monitoring of the individual nodes • Sending ICMP packets with the ‘ping’ command • Proprietary mechanism based upon UDP packets

  40. Advanced Balancing Methods • Primary optimization vectors • Node traffic – predict volume of traffic • Network traffic – monitor node state • Node-load based balancing – (which load ?) • DNS load balancing - simple • Topology-based – reduce latency • Application-specific performance • Policy based optimization • Application ,bandwidth, admin, security

  41. Common Errors • There are four common errors • Overflow • Underflow • Routing errors • Induced network errors • May destabilize efficient network clustering

  42. Information Dissemination • Central  Decentralized • Load incurred on system • Processing load • Network load • Partial knowledge – gossip algorithms • Example: finding average load

More Related