1 / 59

Some Systems, Applications and Models I Have Known

Some Systems, Applications and Models I Have Known. … a retrospective look at many performance evaluation studies. Ken Sevcik University of Toronto. Overview. In the past 35 years, … Systems Have Changed Applications Have Grown Models Have Matured and Adapted

corin
Download Presentation

Some Systems, Applications and Models I Have Known

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Some Systems, Applicationsand Models I Have Known … a retrospective look at many performance evaluation studies Ken Sevcik University of Toronto DCS Colloquium

  2. Overview • In the past 35 years, … • Systems Have Changed • Applications Have Grown • Models Have Matured and Adapted • … and some interesting problems have been encountered Sigmetrics and Performance 2004

  3. [One-slide Tutorial:]Approaches To Performance Evaluation How to answer “What if …” questions (about hardware, software, and workload) • Three alternatives: • Analysis using queueing theory • Abstract model, but fast and cheap • Stochastic Simulation • Detailed model, and takes some time and work • Experimentation • Actual system, but lots of time and work Sigmetrics and Performance 2004

  4. Problems with Voting Systems • Defn: “Majority Winner” • A candidate who wins every pairwise election • Problems: • Voting for a single candidate … • Primaries and Drop Last can eliminate a majority winner • Expressing a full preference ordering … • There may be no majority winner! • Question: • How likely is a “cyclical majority” (or “voters’ paradox) where there is no majority winner? Sigmetrics and Performance 2004

  5. Assume voter preferences are: 30% : L > M > R 10% : M > L > R 20% : M > R > L 40% : R > M > L Single Vote: R wins with 40% Yet pairwise M beats both R and L Preference order and 40% : R > L > M R M L Elections and the “will of the people” 40% 60% 30% 60% 70% 30% 60% R M 60% “Cyclical Majority” 70% L Sigmetrics and Performance 2004

  6. First Research Application: Exact Probability of a Voters’ Paradox • C candidates for election • V voters with strict preference orderings • Can one candidate beat each other pairwise? Example: V = 3 & C = 3 V1 : X > Y > Z V2 : Y > Z > X V3 : Z > X > Y Then, in pair-wise elections, X beats Y ; and Y beats Z ; yet Z beats X ! Paradox occurs in 12 of the (3!)3 = 216 possible configurations. In general, there are (C!)V voting configurations. Sigmetrics and Performance 2004

  7. My first “personal” computer: IBM System 360 Model 30 with BOS Sigmetrics and Performance 2004

  8. Exact Probabilities of Voters’ Paradox • V = 3 & C = 3  12 cycles in 216 configs. • V = 7 & C = 7  26,295,386,028,643,902,475,468,800 cycles in 82,606,411,253,903,523,840,000,000 configs. (Computed in approximately 40 hours of CPU time.) C = 3 5 7 ~ 40 V = 3 .0555… .1600… .238798185941 ~ .61 V = 5 .06944… .19999525 .295755170299 ~ .71 V = 7 .075017 .215334 .318321370333~ .74 V ~ 40 ~ .09 ~ .24 ~ .36 ~ .80 Sigmetrics and Performance 2004

  9. C = 3 5 7 9 ~ 40 V = 3 .0555… .1600… .238798 .298917~ .61 V = 5 .06944… .199995 .295755 .367573~ .71 V = 7 .075017 .21533 .318321.??????~ .74 V = 9.070549 .223717 .330239.?????? ~ .76 V ~ 40 ~ .09 ~ .24 ~ .36 ~ .45 ~ .80 Exact Probabilities of Voters’ Paradox Recent results: V = 9 & C = 7  692,953,571,964,418,337,059,197,419,520,000 cycles 2,098,335,016,107,155,751,174,144,000,000,000 configs. V = 5 & C = 9  2,312,910,445,872,026,769,020,928,000 cycles 6,292,383,221,978,976,013,516,800,000 configs. Sigmetrics and Performance 2004

  10. j1 j1 j1 j2 j2 j2 j2 j2 j2 Job Sequencing on a Single Processor (using service time distribution knowledge) Given N jobs and their service time distributions, Specify a schedule that minimizes average completion time. Example with two jobs: job 1 t1 = k job 2 t2 = s w. prob. 1 - p = t w. prob. p j1 1st: j2 1st: [j2, j1, j2]: [j2, j1, j2] BEST IFF: s (1 – p) < min [k, s +p (t – s)] Sigmetrics and Performance 2004

  11. Job Sequencing on a Single Processor (using service time distribution knowledge) “Smallest Rank” (SR) Scheduling: Minimize Investment (quantum length) Payoff (Pr [Completion]) = Service Time Knowledge exact average distribution No SPT SEPT SEPT Preemption Allowed? Yes SRPT SERPT SR Sigmetrics and Performance 2004

  12. Job Sequencing with Two Processors & Two Customers Extending “Shortest First” to Multiple Resources SBT-RSBT -- Based on average service time per visit of each customer at each resource SBT:  A gets priority at k RSBT:  A gets priority at 1 Sigmetrics and Performance 2004

  13. In the Beginning … • Single Server Queue • Many variations • arrival process, service process • multiple servers, finite buffer size • scheduling discipline • FCFS, RR, FBn, PS, SRPT, … N , Z S RR, FBn, and PS increased relevance of models Sigmetrics and Performance 2004

  14. Z avg. think time K centers Dj demand at j Queuing Network Models “Central Server” Model “Separable” (or “product form”) models N customers and efficient computational algorithms Variants: Open, Closed, Mixed scheduling disciplines Sigmetrics and Performance 2004

  15. The “Great Debate”:Operational Analysis vs. Stochastic Modeling • SM • Ergodic stationary Markov process in equilibrium • Coxian distributions of service times • independence in service times and routing • OA • finite time interval • measurable quantities • testable assumptions OAmade analytic modelling accessible to capacity planners in large computing environments Sigmetrics and Performance 2004

  16. Uses and Analysis of Queuing Network Models • Applications • System Sizing; Capacity Planning; Tuning • Analysis Techniques • Global Balance Solution • Massive sets of Simultaneous Linear Equations • Bounds Analysis • Asymptotic Bounds (ABA), Balanced System Bounds (BSB) • Solutions of “Separable” Models • Exact (Convolution, eMVA) • Approximate (aMVA) • Generalizations beyond “Separable” Models • aMVA with extended equations Sigmetrics and Performance 2004

  17. Bounding Analysis Case Study: Insurance Company with 20 sites • Upgrade alternatives: Upgrade Dcpu Dio Dtot Improvement Current 4.6 4.0 10.6 ----- # 1 5.1 1.9 7.0 1.5 to 2.0 # 2 3.1 1.9 5.0 2.0 to 3.5 ABA Inputs:N, Z, Dtot,Dmax Throughput Bound: Response Time Bound: Sigmetrics and Performance 2004

  18. Bounding Analysis Case Study: Insurance Company with 20 sites • Upgrade alternatives: Upgrade Dcpu Dio Dtot Improvement Current 4.6 4.0 10.6 # 1 5.1 1.9 7.0 1.5 to 2.0 # 2 3.1 1.9 5.0 2.0 to 3.5 .4 #2 .3 X Cur .2 #1 .1 2 4 6 8 10 N Sigmetrics and Performance 2004

  19. Bounding Analysis Case Study: Insurance Company with 20 sites • Upgrade alternatives: Upgrade Dcpu Dio Dtot Improvement Current 4.6 4.0 10.6 # 1 5.1 1.9 7.0 1.5 to 2.0 # 2 3.1 1.9 5.0 2.0 to 3.5 # 1 Cur 20 # 2 15 R 10 5 N 2 4 6 8 10 Sigmetrics and Performance 2004

  20. Exact Mean Value Analysis Algorithm Initialize (for zero customers): Iterate up to N customers: for n = 1, … , N Set Arrival Instant Queue Lengths: Set Residence Time: Understandable and Easy to Implement Sigmetrics and Performance 2004

  21. Approximate Mean Value Analysis Initialize to Equal Queue Lengths: Iterate until convergence: loop until Qk ( N ) are stable Revise Arrival Instant Queue Lengths: Revise Residence Times: Substantial time savings; Little loss of accuracy Sigmetrics and Performance 2004

  22. “Details” of Real Systems • Going beyond “Separable” models • Priority Scheduling • Alter Residence Time equation • FCFS with high variance service times • Reflect coefficient of variation in service times • Memory Constraints • Alter MPL limit N , or Dpaging • I/O Subsystems (simultaneous resource possession) • Reflect contention by inflating Ddisk • Enhanced Utility of QNM’s for Real Systems Sigmetrics and Performance 2004

  23. QNM’s for Capacity Planning & Tuning • Existing system with measurable workload • “What if …” • … the workload volume increases? • … the workload mix changes? • … the processor is upgraded? • … memory is added? • … the I/O configuration is enhanced? • … class priorities are adjusted? • … file placements are changed? • … changing usage of memory? • Answer by changing model parameters CAPACITY PLANNING TUNING Sigmetrics and Performance 2004

  24. System Sizing Case Study:NASA Numerical Aerodynamic Simulator GOAL: to attain a sustainable Gigaflop Work Stations Data Mgmt Cray 1 Cray 2 Cray 3 Graphics QNM’s proved more useful than a simulation model Sigmetrics and Performance 2004

  25. Capacity Planning Case Study: FAA Air Traffic Control System • ~ 40 distributed air traffic control centers • Each with the SAME: • software • hardware family • 35 transaction types • But DIFFERENT: • transaction volumes and mixes • Single QNM (one class per transaction type) supports capacity planning for all sites Sigmetrics and Performance 2004

  26. QNM’s for System and Architecture Analysis • Architectures • caching structures • Communication networks • Local Area Networks • Rings, buses • Store and Forward • flow control • end to end response time • Interconnection networks • omega, shuffle-exchange, … Sigmetrics and Performance 2004

  27. Space Station Orbital Platform Tethered Platform Shuttle Extra-Vehicular Activity Network for NASA’s Space Station (circa 1984) • Distributed LAN for many components Results: Some properties of the FDDI Protocol Ground Station Sigmetrics and Performance 2004

  28. Continuing vs. Upward Exiting vs. Entering Architectural Analysis Case Study: NUMAchine • 4 x 4 x 4 Hierarchical Ring Architecture Setting Routing Priorities: Message Handling: Contiguous vs. Interleaved Shortest First ? Sigmetrics and Performance 2004

  29. SE&EU Interconnection Network Source 000 001 010 011 100 101 110 111 Destination 000 001 010 011 100 101 110 111 Exchange Unshuffle Shuffle Exchange Sigmetrics and Performance 2004

  30. Bn B1 Sn Sn-1 Sn-2 S4 S3 S2 S1 Bn-1 B2 Bn-2 B3 B4 Bn-3 (Longest Matching Bit String) EU: Right 2 SE: Left 5 EU: Right 1 Dn Dn-1 Dn-2 D4 D3 D2 D1 SE&EU operation Combination Lock Algorithm: Up to 40% increase in throughput Sigmetrics and Performance 2004

  31. Job Scheduling for Parallel Processing Variants:Rigid Moldable Evolving Malleable Job j = ( tj , pj ) 1 2 3 processors P time Sigmetrics and Performance 2004

  32. j1 j2 j1 j2 Parallelism: Early or Late ? • Problem • Schedule N jobs of two tasks each on two processors to minimize average residence time • Each pair of jobs can be executed as … PARALLEL: SEQUENTIAL: overhead of parallel execution Sigmetrics and Performance 2004

  33. Results of two similar studies: [RN et al.] Start parallel; Finish sequential Parallelism: Early or Late ? S S S P P P P P P S S S Sigmetrics and Performance 2004

  34. Results of two similar studies: [RN et al.] Start parallel; Finish sequential [KCS] Start sequential; Finish parallel Parallelism: Early or Late ? S S S P P P P P P S S S S S S P P P P P P S S S Sigmetrics and Performance 2004

  35. Results of two similar studies: [RN et al.] Start parallel; Finish sequential [KCS] Start sequential; Finish parallel Differences in assumptions: Some variability in task service times ( or ) [RN] Some overhead of parallelism ( ) [KCS] Parallelism: Early or Late ? S S S P P P P P P S S S S S S P P P P P P S S S Sigmetrics and Performance 2004

  36. P P P S P P S S S P S S S S S P P P P P P P P P P P P P P P P P P P P P S S S S S S S S S P P P P P S P P S S P P S S S P P P S P P S S P P S S S P P P P P P P S P S S S S S S S S S S S S S S S S S S S S S S S Parallelism: Early or Late ? • Resolution increasing increasing Sigmetrics and Performance 2004

  37. The Case for Popt = 1 : • (Assume p > 1 Ej (p) < 1 ) • Argument: • Demand is insatiable (unbounded backlog) • Economies of scale (100’s of users) • “Good” systems will be heavily used • Parallelism overhead decreases throughput and increases queuing times Sigmetrics and Performance 2004

  38. Distributed Processing Models • Processor selection strategies • local vs. global execution • Load Sharing • sender-initiated vs. receiver-initiated Sigmetrics and Performance 2004

  39. Small example: Individual Versus Social Optimum • Arriving customers must pick one of two processors, one fast and one slow: pF F pS S Individual Optimum: Pick server with lower response time (  response times are equalized) Social Optimum: Control pF to minimize average response time Sigmetrics and Performance 2004

  40. Satisfying Social and Individual Goals Individual Goal: Equalize Response Times Individual Optimum: Social Goal: Minimize Average Response Time min Social Optimum: Sigmetrics and Performance 2004

  41. Resolution of Social and Individual Goals 1. Charge a Toll on the Fast processor; 2. Give a Rebate to users of the Slow processor; 3. Set total of Rebates to equal the total of Tolls. Toll on users of F: Rebate to users of S: RESULT: Individual Choice Yields Social Optimum So Everybody Wins !!! Sigmetrics and Performance 2004

  42. Resolution of Social and Individual Goals Example: pF RF RS R IND: .87 16.7 16.7 16.7 SOC: .85 12.1 27.0 14.3 With Toll = 2.2 (and Rebate = 12.7): pF RF RS R CF CS C Toll: .85 12.1 27.0 14.3 14.3 14.3 14.3 Sigmetrics and Performance 2004

  43. +2 0 -2 -2 0 +2 Anomaly of High Dimensional Spaces 2k Spheres (radius = 1) in Cube (vol. 4k & 2 k sides) and an Inner sphere 1. Pointy-ness Property 2. Radius of Inner Sphere R2 = .414 R10 = 2.16 !!! 3. Volume Ratio Sigmetrics and Performance 2004

  44. Diagonal of a k-dimensional Cube (Example: k = 25 ) Corners = Red = Blues = Sigmetrics and Performance 2004

  45. Blue width = Red width = Corner width = Diagonals of Cube K = 1 K = 2 K = 3 K = 4 Sigmetrics and Performance 2004

  46. Diagonals of Cube K = 9 K = 121 (There are 2121 blue spheres) Sigmetrics and Performance 2004

  47. A1 A2 A3 A4 … Ak-1 Ak A1 A3 A2 Multidimensional Databases Relational View: (Records of k Attributes) Multidimensional View: (Points in k-dimensional space) Indexing Support for: -- point search -- range search -- similarity search -- clustering Sigmetrics and Performance 2004

  48. Bounding Spheres and Rectangles circumscribed inscribed ratio of Dim k sphere cube sphere volumes -------- ---------------- ---------- --------------- ------------- 2 1.57 1.00 .785 2 4 4.93 1.00 .308 16 8 64.94 1.00 .0159 4096 16 15422.64 1.00 .000004 4294967296 Sigmetrics and Performance 2004

  49. Edge Density in High-Dimensions • Proportion of points near some side: Fraction near some edge: 1 k eps = .002 .020 .200 ---- ------ ------ ----- 1 .004 .040 .400 2 .007 .078 .640 4 .015 .150 .870 8 .031 .278 .983 16 .062 .479 .999 Sigmetrics and Performance 2004

  50. Lessons and Conclusions • Exact answers are overrated • accurate approximate answers often suffice • (e.g., Voters’ Paradox and Exact QNM solutions ) • Analytic models have an important role • quick, inexpensive answers in many situations • (e.g., Insurance Co., NAS System, and FAA System ) • Assumptions matter • subtle differences can have big effects • (e.g., in Early or Late Parallelism, NUMAchine analysis and PRI vs. FCFS or PS) Sigmetrics and Performance 2004

More Related