Lecture 2b: Performance Metrics

1. Lecture 2b:Performance Metrics

2. Performance Metrics Measurable characteristics of a computer system: Count of an event Duration of a time interval Size of a parameter Rate: Operations executed per second

3. Performance Mertrics Clock Speed Clock speed/frequency (f): the rate of clock pulses (ex: 1GHz) Cycle time (Tc): time between two clock pulses (Tc = 1/f) Tc

4. Performance Mertrics Instruction Execution Rate Cycles per Instruction (CPI): is an average depends on the design of micro-architecture (hardwired/microprogrammed, pipelined) Number of instructions: is the number of instructions executed at runtime Depends on instruction set architecture (ISA) compiler CPIi: number of cycles required for instruction i Ii: number of executed instructions of type i CPI =

5. Performance Metrics CPU Performance CPU time of a program (T) = instructions x cycles x time program instruction cycle CPI (cycles per instruction) T = instruction count x CPI x 1 f

6. Performance Metrics CPU Performance Drawbacks: In modern computers, no program runs without some operating system running on the hardware Comparing performance between machines with different operating systems will be unfair

7. Performance Metrics Execution time Elapsed time / wall-clock time / response time Latency to complete a task, including disk access, memory access, I/O, operating system overhead, and everything (includes time consumed by other programs in a time-sharing system) CPU time The time CPU is computing, not including I/O time or waiting time User time / user CPU time CPU time spent in the program System time / system CPU time CPU time spent in the operating system performing tasks requested by the program

8. Performance Metrics Performance Comparison Relative performance Performancex = 1 . Execution timeX Performance Ratio = PerformanceX = Execution timeY PerformanceY Execution timeX

9. Performance Metrics Relative Performance If workload consists of more than one program, total execution time may be used. If there are more than one machine to be compared, one of them must be selected as a reference.

10. Performance Metrics Throughput Total amount of work done in a given time Measured in tasks per time unit Can be used for Operating system performance Pipeline performance Multiprocessor performance

11. Performance Metrics MIPS (Million instructions per second) Includes both integer and floating point performance Number of instructions in a program varies between different computers Number of instructions varies between different programs on the same computer MIPS = Instruction count = Clock rate Execution time x 106 CPI x 106

12. Performance Metrics MFLOPS (Million floating point operations per second) Give performance of only floating-point operations Different mixes of integer and floating-point operations may have different execution times: Integer and floating-point units work independently Instruction and data caches provide instruction and data concurrently

13. Performance Metrics Utilization Speciality ratio 1  generalpurpose Utilization = Busy time . Total time Speciality ratio = Maximum performance . Minimum performance

14. Performance Metrics Asymptotic and Half performance r – asymptotic performance n1/2 – half performance T = r (n + n1/2) r = 1/t n1/2 = t0/t Slope = r-1 2t0 t0 n1/2 -n1/2

15. Performance Metrics Speedup Express how much faster is system 2 than system 1 Calculated directly from execution time Performancex = 1 = 1 Execution timeX TX Speedup2,1 = Performance2 = T1 Performance1 T2

16. Performance Metrics Relative Change It expresses the performance of system 2 relative to system 1 Performancex = 1 = 1 Execution timeX TX Relative change2,1 = Performance2 - Performance1 = T1 - T2 = Speedup2,1 - 1 Performance1 T2

17. Performance Metrics Statistical Analysis Used to compare performance Workload consists of many programs Depends on the nature of the data as well as distribution of the test results

18. Performance Metrics Indices of Central Tendency Used to summarize multiple measurements Mean Median Mode

19. Performance Metrics Mean (average) Gives equal weight to all measurements Arithmetic mean = S xi , 1 ≤ i ≤ n n

20. Performance Metrics Median Order all n measurements The middle value is the median. If n is even, median is the mean of the middle 2 values Using Median instead of Mean reduces the skewing effect of the outliers. Median = = 17

21. Performance Metrics Mode Mode is the value that occurs most frequently If all values occur once, there is no mode If there are several samples that all have the same value, there would be several modes Mode = 20

22. Mean, Median, Mode Mean • Incorporates information from the entire measured values • Sensitive to outliers Median and Mode • Less sensitive to outliers • Do not effectively use all information ex

23. Performance Metrics Arithmetic mean (average) May be misleading if the data are skewed or scattered Arithmetic mean = S xi , 1 ≤ i ≤ n n

24. Performance Metrics Weighted average weight is the frequency of each program in daily processing Results may change with a different set of execution frequencies Weighted average = ∑ wi . xi 1 ≤ i ≤ n

25. Performance Metrics Geometric mean Results are stated in relation to the performance of a reference machine Geometric mean = (  xi )1/n , 1 ≤ i ≤ n • Results are consistent no matter which system is chosen as reference

26. Performance Metrics Harmonic mean Used to compare performance results that are expressed as a rate (e.g. operations per second, throughput, etc.) Slowest rates have the greatest influence on the result It identifies areas where performance can be improved Harmonic mean = n , 1 ≤ i ≤ n ∑ 1/xi

27. Performance Metrics Characteristics of a good performance metric If the time values are averaged, then the resulting mean value must be directly proportional to the total time. If the rate values are averaged, then the resulting mean value must be inversely proportional to the total time.

28. Performance Metrics Ex n benchmark programs Ti is the execution time of program i F floating-point operations in each program Mi = F / Ti is the execution rate of program i (MFLOP/s) Arithmetic mean Inappropriate for summarizing rates TA = TA is directly proportional to the total execution time MA is inversely proportional to the total execution time MA = =

29. Performance Metrics Harmonic mean Inappropriate for summarizing execution times Appropriate for summarizing rates TH = TH is not directly proportional to the total execution time MH is inversely proportional to the total execution time MH = =

30. Performance Metrics Ex

31. Performance Metrics Geometric mean Inappropriate for summarizing execution times Inappropriate for summarizing rates TG = TG is not directly proportional to the total execution time MH is not inversely proportional to the total execution time MH = =

32. Performance Metrics Geometric mean Produces a consistent ordering of the systems but it is the wrong ordering

33. Performance Metrics Histogram Used to display the distribution of a set of measured values (variability) First find the minimum and maximum values. Then divide the range into b subranges, called cells.

34. Histogram

35. Performance Metrics Index of Dispersion Index of dispersion is used to compare the spread of measurements around the mean value Range is the simplest metric for an index of dispersion Range is sensitive to a few extreme values

36. Performance Metrics Index of Dispersion Maximum of the absolute values of the difference of each measurement from the mean It is also sensitive to extreme values

37. Performance Metrics Index of Dispersion Sample variance is the simplest metric for an index of dispersion • Requires 2 passes through the data to calculate first x and then s2 • Requires 1 pass

38. Performance Metrics Index of Dispersion Standard deviation Coefficient of variance (COV): normalizes standard deviation wrt the mean