Selection of Measurement Techniques and Metrics

1. Selection of Measurement Techniques and Metrics FaizaIshaq

2. Overview One or more systems, real or hypothetical You want to evaluate their performance What technique do you choose? -Analytic Modeling? - Simulation? - Measurement?

3. Outline • Selecting an Evaluation Technique • Selecting Performance Metrics • Case Study • Commonly Used Performance Metrics • Setting Performance Requirements • Case Study

4. Outline • Selecting an Evaluation Technique • Selecting Performance Metrics • Commonly Used Performance Metrics • Setting Performance Requirements • Case Study

5. Selecting Performance Metrics response timerepresenting the elapses between the time of sending a message and the time when the error diagnostic is received. – S. Kelly-Bootle, The Devil’s DP Dictionary Time Speed Rate Request Resource Correct Done Probability System Not Correct Errori Reliability Time between Not Done Eventk Duration Availability Time between

6. Selecting Performance Metrics • May be more than one set of metrics • Resources: Queue size, CPU Utilization, Memory Use … • Criteria for selecting subset, choose: • Low variability – need fewer repetitions • Non redundancy – don’t use 2 if 1 will do • ex: queue size and delay may provide identical information • Completeness – should capture tradeoffs • ex: one disk may be faster but may return more errors so add reliability measure

7. Outline • Selecting an Evaluation Technique • Selecting Performance Metrics • Commonly Used Performance Metrics • Setting Performance Requirements • Case Study

8. Commonly Used Performance Metrics • Response Time • Turn around time • Reaction time • Stretch factor • Throughput • Operations/second • Capacity • Efficiency • Utilization • Reliability • Uptime • MTTF

9. User’s Request System’s Response Time Response Time (1 of 2) • Interval between user’s request and system response • But simplistic since requests and responses are not instantaneous • Users type and system formats

10. Response Time (2 of 2) System Starts Execution System Starts Response User Starts Request User Finishes Request System Finishes Response Time Reaction Time Think Time Response Time 1 Response Time 2 • Can have two measures of response time • Both ok, but 2 preferred if execution long • Think time can determine system load

11. Response Time+ • Turnaround time – time between submission of a job and completion of output • For batch job systems • Reaction time - Time between submission of a request and beginning of execution • Usually need to measure inside system since nothing externally visible • Stretch factor – ratio of response time at load to response time at minimal load • Most systems have higher response time as load increases

12. Throughput (1 of 2) • Rate at which requests can be serviced by system (requests per unit time) • Batch: jobs per second • Interactive: requests per second • CPUs • Millions of Instructions Per Second (MIPS) • Millions of Floating-Point Ops per Sec (MFLOPS) • Networks: pkts per second or bits per second • Transactions processing: Transactions Per Second (TPS)

13. Knee Nominal Capacity Thrput Usable Capacity Knee Capacity Load Response Time Load Throughput (2 of 2) • Throughput increases as load increases, to a point • Kneeis where response time goes up rapidly for small increase in throughput

14. Efficiency Number of Processors Efficiency • Ratio of maximum achievable throughput • For multiprocessor, ration of n-processor to that of one-processor (in MIPS or MFLOPS)