350 likes | 438 Views
System Management Planners: Transforming (High-level Specifications) (Configuration Actions). Sandeep Uttamchandani IBM Almaden Research Center. Jim Gray's Turing award speech “What next? - A dozen IT research goals”, 1999. Build a system used by millions of people each day
E N D
System Management Planners: Transforming (High-level Specifications) (Configuration Actions) Sandeep Uttamchandani IBM Almaden Research Center
Jim Gray's Turing award speech “What next? - A dozen IT research goals”, 1999 Build a system • used by millions of people each day • administered and managed by a ½ time person. • On hardware fault, order replacement part • On overload, adjust automatically
Automated Management: A Growing Necessity! • Demand for IT Management : • Growing number of applications and data footprint (988 exabytes in 2010 compared to 161 in 2006 -- IDC) • Government regulatory compliance (e.g., HIPAA, Sarbanes-Oxley), Disaster Recovery Planning, Application Performance requirements, Provisioning Planning • Growing number of heterogeneous devices, management protocols, application requirements and policies • Supply of Administrators • 1 Storage Admin manages approximately 300GB- 1000GB of storage -- enterprises moving towards petabyte scale systems • Lack of end-to-end knowledge: Application + Servers + Networks + Storage • Skilled administrators are scarce and costly
Talk Outline • Problem Drill-down: Understanding the System Administrative tasks • Taxonomy of Approaches for Automation • Management Planners • Building Blocks • Putting it together
Deploying a new Application within an Enterprise Data-Center • Find a server – create a new Virtual Machine • Install and configure application • Select a Storage controller • Find a storage pool -- create a new volume with required capacity • Select a FC switch with available ports -- connect to server and storage controller • Zone the switch followed by LUN Masking and mapping
- read/write ratio - rand/seq ratio - request-size - … Heterogeneous Component Models Application Performance Management (I) DAS NAS Workload access variations iSCSI Failures - Hardware failures - Software bugs - Operator errors Observe Analyze Request size Act SPC: OLTP Load Surges Time IOPS Time
Application Performance Management (II) • Problem Determination • Impact Analysis • Root-cause diagnosis • Event mining and configuration changes • Load balancing • Which application to move? Where? When? • Adding Hardware • Servers, Network, Storage? Where?
Post deployment Tasks • Performance Management • Disaster Recovery (Availability Management) • Regulatory Compliance • Security • Hardware changes • Changing applications and IT goals • …
Administrator’s Dream ? System Configuration Details/Corrective Actions Current State Task Requirements Objective Functions
Expert Systems: Capturing Human Problem Solving Mycin Expert System Pin-point Bacteria/ Medication Series of disease symptoms Rule-based Inference IF the infection is pimary-bacteremia AND the site of the culture is one of the sterile sites AND the suspected portal of entry is the gastrointestinal tract THEN there is suggestive evidence (0.7) that infection is bacteroid.
Management Planners:Model-based + Declarative Specifications
Self-evolving predictors using machine learning Declarative Specification Minimize high-priority workloads violating SLOs Knowledge-base Predictors of System Behavior Reasoning Engine Objective Constrained Optimizer Configuration/ Action Selection - Component capabilities - Workload dependencies on individual components - Effects of action invocation Capability Current Status Corrective Action Trigger Managed [Storage] System
Building Blocks • Requirements (Declarative Specifications) • Collecting data from devices • Creating device performance models • Formalizing the optimization problem
Knowledge-base: Intuition for generating models Models are mathematical functions e.g. r = ax + by Input variables = {x, y}; Output variables = {r}; Constants {a, b} Generating a function (curve-fitting approach): • Step 1: Designer Specification: Enumerates related parameters (r is a function of x, y, and z) • Step 2: Creating a Baseline model: Off-line data collection; for values of r, x, y, z, determine a best-fit curve (i.e. values of coefficients, and the form of function) • Step 3: Continuous on-line refinement of the functions with additional monitored data
Quadratic Fit: S= 3.284, r = 0.838 Component Models • Representation: Response time = c( req_size, r/w_ratio, rand/seq_ratio, req_rate, cache_hit_rate) • Bootstrapping: • Offline calibration tests OR • Performance specifications from vendor • Challenges: • Interleaving of workload streams • Caching effects due to sharing • Related Work: • CART model [CMU] • Table-based approach [HP] Linear fit (Non-saturated case): S = 0.2509, r = 0.989
Capturing Mean Workload Models • Representation: Component Load = wn(application request_rate) e.g. load at the storage controller and switch for 1000 database (OLTP) transactions • Bootstrapping: • Initial monitoring phase OR • Libraries for application workloads (e.g. OLTP, Decision support, E-mail) • Challenges: • Mean-value not sufficient for real-world workloads**; Using Cummulative Distribution Functions (CDF) • Related Work: • ClockWork (trend prediction) [IBM] • Using ARIMA for Predictive IO prefetching [UIUC] Capturing Variance SPC OLTP Harvard Campus
Optimization Formalism: Linear Programming • Objective function: While solving the SLA violation, minimize the throttling of high priority workloads • Variables: Throttle value for each workload • Constraints: • The response time of the components for a given component load • The request-rate at the component arriving from the workload streams • Change in the application request-rate with throttling • Latency SLA of the workloads
Management Planners: A reality! • Throttling Planner [Usenix’05] • SMART: Performance Management Planner [Usenix’06] • SAN Planner in IBM TotalStorage Productivity Center • Disaster Recovery Planner • End-to-end Provisioning Planner • …
Summary… • Data-centers are becoming growing to petabyte scale and beyond • Need for Automation • Administrative Tasks range from simple firmware upgrades to complex provisioning and disaster recovery planning • Back-of-the-envelop calculations are no longer feasible • Management Planners • Map high-level declarative specification to configuration commands • Hide the underlying device configuration, performance, event details • Automatically create and continuously refine device models
Food for thought… “How accurate can the models be?” “How accurate the models need to be?” Research Spectrum • Representation & creation of domain knowledge - Understanding of system details - Feature-set selection - Machine learning techniques • Formalisms for selection & execution of actions - Constrained optimization techniques - Handling uncertainty and inaccuracies - Variably aggressive action execution • Pragmatic rules-of-thumb • - Models don’t need to perfectly accurate • - Not critical to select the most “optimal” action invocation, but rather to avoid the worst ones - Creating domain knowledge is not a one-time activity – incremental addition and evolution - Automate the common-case
Thankyou! Sandeep Uttamchandani (sandeepu@us.ibm.com) http://www.almaden.ibm.com/StorageSystems/Storage_Management_and_Solutions/
Policy-Based Interface • Term Policy means different things to different people • Service Class • Goals • Constraints • Best Practices • Rules of thumb • If-Then-Scope-Priority (IBM’s PMAC model) • Some users want a lot of control over the policy specification • Other users want pre-packaged service classes (like Gold/Silver/Bronze) and they subsequently want to fine-tune the parameters and create customized service classes
Aperi: Open standard initiatives • Aperi’s goal • Delivers an open-source common management platform through the contribution and development of actual code. • The common platform will implement SNIA’s SMI-S specification for management of heterogeneous devices. • Targeted Benefits • Improve speed to market of new advanced tools designed for ease of use • Reduce the need for customers to replace storage management platforms when purchasing new hardware or software • Encourage vendors to support industry standards for management in their hardware implementations Aperi An open-source storage management community Startups Academia Fabric “Innovators” Aperi CommonOpen SourcePlatform VC’s Storage System Initial members
Latency(%) EXCEED FAILED 1 MEET LUCKY SLAi – a(current_throughputi, ti) if SLAi > a(current_throughputi, ti) 1 IOps(%) 0 Ai = 0 otherwise Optimization Formalism (cont.) Objective function: Minimize ∑paipbiAi /SLAi where pai= Workload priority pbi = Quadrant priority Minimize ∑paipbi[ SLAi – a(current_throughputi, ti)] SLAi Constraints: cachehiti*hittimei+(1-cachehiti) c(∑a(current_throughputi, ti) SLAi 0 ti 1
Non-Procedural Specifications Research • “Procedural-is-best” controversy • Separation of facts and formalisms • Strategies such as “backtracking” to search the knowledge-base • Logic-based • Network-based • Relational model Machine learning Research • Correlating observed behavior with system parameters • Statistical Learning techniques: Neural Networks, SVMs • Gray-box approaches such as the Snowball project • Supervised/ Re-enforcement • Boosting Related Fields: A Lot to Learn! Expert Systems Research • Low Road: Dendral • Middle Road: Mycin, R1 • High Road: Sophie • Architecture: Knowledge-base & Reasoning engine • Knowledge-base encodes (generic) domain knowledge • Reasoning engine can interpret knowledge in multiple ways
A Typical Data-center Application (SAP Application Server) Executables NTFS File System DB Server WINDOWS IBM DB2 (Database Managed Storage) DB Server AIX IBM DB2 (System Managed Storage) DB Server WINDOWS Oracle (Database Managed Storage) DB DB DB Logical Volume Manager Logical Volume Logical Volume JFS JFS Data Log Data Log Data Log Temp Volume Volume Volume Volume Volume Volume Volume
Application Downtime = $$$ Losses Applications require 24 x 7 availability of business critical applications Application Availability = Ensuring availability of multiple tiers • Storage Controllers • SAN Appliances • Servers • Virtual Machines • Databases/File-systems Failures come in several flavors • Virus failures • Mis-configuration errors • Subsystem failures • Site failures (hurricanes, planes)
Preparing for IT Disasters: Administrator’s Task-list • Planning • Understand DR requirements • Evaluate replication services available at storage and other levels • Analyze existing copy services configuration (if any); Generate a DR plan • Deployment • Configure various replication technologies from IBM and non-IBM vendors • Replication at different levels namely the database, server, operating system and storage level (e.g., RM, HACMP, SRDF, MSCS, VCS) • Validation • Validate DR plans for changes in configuration changes (e.g., changes in zoning) and application characteristics (e.g., write rate) • Continuous Optimization • Optimize DR plans for unused copy relationships • Recommend updates to existing configuration based on hardware and software changes
Preparing for IT Disasters: A Consultant’s Gold-mine • Planning: Complex Search Space, Manual & Error-prone • DR Requirements (RMAF questionnaire); Storage and Server Characteristics • Replication Technology Characteristics • Constraints: Interoperability, # sites, dollar cost • Best Practices • Deployment: Requires expertise in multiple replication technologies • Vendor-specific CLI commands/API for creating/updating/deleting copy pairs, sessions, consistency groups; done manually today by administrators • Validation and Continuous Optimization: 24X7 impact analysis • Analyze impact of configuration changes and application properties • Periodic sampling at primary and secondary storage for Recovery Point Objective (RPO)
y = f(x5,x11) f monitored data MonitorMining Related Work: Creating models Spectrum y = ax5 + bx11 y = a1x1 + a2x2 +… a100x100 Analytical Approaches Black-box Approaches Representation of models? Evolution of models? Incomplete designer specifications? Brittle, error-prone Convergence, Accuracy • - John Wilkes Ecosystem: Minerva, Ergastrulum, Hippodrome - Modeling disk behavior, formulas for data prefetching , modelling migration - [Bre94], [Bre95], [Mat97], [Agr00], [Nob97], [Men01], [Laz84], [Vap95], [Vud01] - Case-base reasoning - Multi-relational mining - Table-based models - CART [CMU], CMiner [UIUC]
Related Work: Policy-based Management (Pattern-based Procedure Invocation [Hewitt67]) Example: Rules for the Prefetch knob [Event]: Latency_violation [Condition]: If ((Memory_available > 70) && (access_pattern < 0.4 sequential) && (read/write > 0.4)) [Action]: Prefetch = 1.2*Prefetch Event: Latency_violation If {(15 < Memory_available > 70 && FC_interconnect_available > 60 ) && ( access_pattern > 0.7 sequential && read/write > 0.4)} Prefetch = 1.4*Prefetch Event: Latency_violation If {(Memory_available > 70 && FC_interconnect_available > 60 ) && ( 0.4 < access_pattern < 0.7 sequential && read/write > 0.4)} Prefetch = 1.3*Prefetch Event: Latency_violation If {(Memory_available < 15) && ( access_pattern > 0.8 sequential && read/write > 0.4)} Prefetch = 1.2*Prefetch Event: Latency_not_met If {(Memory_available < 15 ) && ( access_pattern < 0.8 sequential && read/write > 0.4)} Prefetch = 1.05*Prefetch Event: Latency_not_met If {(FC_interconnect_available < 20) && ( access_pattern > 0.8 sequential && read/write > 0.4)} Prefetch = 1.3*Prefetch ………AND MORE………………….. • Complexity • Level of details in terms of thresholds and invocation values • Deciding among the action set • Number of rules and conflicts analysis: O( Resource-state x Workload x Action-sets x Current-behavior) • Brittleness • Closely tied with system configurations, workloads andaction-sets • No systematic model/approach for refining specifications