ICOM 6115: COMPUTER SYSTEMS PERFORMANCE MEASUREMENT AND EVALUATION

1. ICOM 6115: COMPUTER SYSTEMS PERFORMANCE MEASUREMENT AND EVALUATION Nayda G. Santiago August 18, 2006

2. Administrative Stuff • Quiz 1 • August 23, 2006, Wednesday • WebCT – if activated • Article: Designing computer architecture research workloads • Students who do not have cards to access lab • Cesar and Idalys

3. Back to performance measurements • How to measure software performance on a system?

4. Experiment • In order to analyze the performance of an application, one has to conduct one or more experiments to find out about its possible performance problems.

5. General Structure of Measurement • Typical execution of an application program Source repository Output Data Repository Control system Input Data Application Process Source Code Compilation System Executable Executor/ Launcher

6. Typical execution of an application program • Typical execution of an application program • The source code of the application is either available directly or through a repository control system like RCS or CVS which also allows to store and access multiple versions versions of the application's code at the same time in a so-called source repository. • A compilation system is used to produce an executable. • The compilation system typically consists of several phases including preprocessors, parsers, optimizers, assemblers, linkers, and runtime libraries. Compilation systems for high-level parallel languages like HPF often generate output in another language like Fortran or C in a first step and then use compiler systems for these languages to produce an executable.

7. Typical execution of an application program • A launcher / executor takes the executable and suitable input data and executes it as one ore more application processes on a computer system. • Launching an application can be done interactively (e.g. through a UNIX shell program or a program manager) or through a batch system under the control of a command script. Typically, the program generates some output data containing the results of the computation.

8. Instrumentation • In order to determine the performance of an application, the application has to be modified in order to insert code which collects information about the performance of the program. This process is called instrumentation. This can be done in several ways: • A source code instrumentor modifies the original source code to add the necessary measurement instructions. It is typically implemented as a source-to-source translation tool which runs as a preprocessor before the actual compilation or it is handled by the compilation system itself. • An object code instrumentor modifies object code or executables to accomplish the same task. • An dynamic instrumentor also modifies executables but does this during the execution of the program. It has the great advantage that instrumentation can be changed / adapted during the execution.

9. Measurement Experiment Setup

10. Selecting Performance Metrics response timen. An unbounded, random variable … representing the elapses between the time of sending a message and the time when the error diagnostic is received. – S. Kelly-Bootle, The Devil’s DP Dictionary Time Speed Rate Request Resource Correct Done Probability System Not Correct Errori Reliability Time between Not Done Eventk Duration Availability Time between

11. Selecting Performance Metrics • Mean is what usually matters • But variance for some (ex: response time) • Individual vs. Global • May be at odds • Increase individual may decrease global • Ex: response time at the cost of throughput • Increase global may not be most fair • Ex: throughput of cross traffic • Performance optimizations of bottleneck have most impact • Example: Response time of Web request • Client processing 1s, Latency 500ms, Server processing 10s  Total is 11.5 s • Improve client 50%?  11 s • Improve server 50%?  6.5 s

12. Selecting Performance Metrics • May be more than one set of metrics • Resources: Queue size, CPU Utilization, Memory Use … • Criteria for selecting subset, choose: • Low variability – need fewer repetitions • Non redundancy – don’t use 2 if 1 will do • ex: queue size and delay may provide identical information • Completeness – should capture tradeoffs • ex: one disk may be faster but may return more errors so add reliability measure

13. Commonly Used Performance Metrics • Response Time • Turn around time • Reaction time • Stretch factor • Throughput • Operations/second • Capacity • Efficiency • Utilization • Reliability • Uptime • MTTF

14. User’s Request System’s Response Time Response Time • Interval between user’s request and system response • But simplistic since requests and responses are not instantaneous • Users type and system formats

15. Response Time • Can have two measures of response time • Both ok, but 2 preferred if execution long • Think time can determine system load System Starts Execution System Starts Response User Starts Request User Finishes Request System Finishes Response Time Reaction Time Think Time Response Time 1 user starts next request Response Time 2

16. Response Time • Turnaround time – time between submission of a job and completion of output • For batch job systems • Reaction time - Time between submission of a request and beginning of execution • Usually need to measure inside system since nothing externally visible • Stretch factor – ratio of response time at load to response time at minimal load • Most systems have higher response time as load increases

17. Throughput • Rate at which requests can be serviced by system (requests per unit time) • Batch: jobs per second • Interactive: requests per second • CPUs • Millions of Instructions Per Second (MIPS) • Millions of Floating-Point Ops per Sec (MFLOPS) • Networks: pkts per second or bits per second • Transactions processing: Transactions Per Second (TPS)

18. Knee Nominal Capacity Thrput Usable Capacity Knee Capacity Load Response Time Load Throughput • Throughput increases as load increases, to a point • Nominal capacity is ideal (ex: 10 Mbps) • Usable capacity is achievable (ex: 9.8 Mbps) • Knee is where response time goes up rapidly for small increase in throughput

19. Efficiency Number of Processors Efficiency • Ratio of maximum achievable throughput (ex: 9.8 Mbps) to nominal capacity (ex: 10 Mbps)  98% • For multiprocessor, ration of n-processor to that of one-processor (in MIPS or MFLOPS)

20. Utilization • Typically, fraction of time resource is busy serving requests • Time not being used is idle time • System managers often want to balance resources to have same utilization • Ex: equal load on CPUs • But may not be possible. Ex: CPU when I/O is bottleneck • May not be time • Processors – busy / total makes sense • Memory – fraction used / total makes sense

21. Miscellaneous Metrics • Reliability • Probability of errors or mean time between errors (error-free seconds) • Availability • Fraction of time system is available to service requests (fraction not available is downtime) • Mean Time To Failure (MTTF) is mean uptime • Useful, since availability high (downtime small) may still be frequent and no good for long request • Cost/Performance ratio • Total cost / Throughput, for comparing 2 systems • Ex: For Transaction Processing system may want Dollars / TPS

22. HB Better Utility LB NB Metric Better Utility Utility Best Metric Metric Utility Classification • HB – Higher is better (ex: throughput) • LB - Lower is better (ex: response time) • NB – Nominal is best (ex: utilization)

23. Setting Performance Requirements • The system should be both processing and memory efficient. It should not create excessive overhead • There should be an extremely low probability that the network will duplicate a packet, deliver it to a wrong destination, or change the data • What’s wrong?

24. Setting Performance Requirements • General Problems • Nonspecific – no numbers. Only qualitative words (rare, low, high, extremely small) • Nonmeasureable – no way to measure and verify system meets requirements • Nonacceptable – numbers based on what sounds good, but once setup system not good enough • Nonrealizable – numbers based on what sounds good, but once started are too high • Nonthorough – no attempt made to specify all outcomes