Chapter 4 Assessing and Understanding Performance

1. Chapter 4Assessing and Understanding Performance

2. 4.1 Performance • Measure, Report, and Summarize • Make intelligent choices • See through the marketing hype • Key to understanding underlying organizational motivationWhy is some hardware better than others for different programs?What factors of system performance are hardware related? (e.g., Do we need a new machine, or a new operating system?)How does the machine's instruction set affect performance?

3. Performance • For example, to improve the performance of a software system, we may need to understand what factors in the hardware contribute to the overall system performance and the relative importance of these factors. • The factors may include how well the program uses the instructions of the computer, how well the underlying hardware implements the instructions, and how well the memory and I/O system performance. • Understanding how to determine the performance impact of these factors is crucial to understanding the motivation behind the design of particular aspects of the computer.

4. Which of these airplanes has the best performance? • How much faster is the Concorde compared to the 747? • How much bigger is the 747 than the Douglas DC-8? • If we define performance in term of speed. There are two possibility: • The fastest one as the one with the highest cruising speed/one passenger • The one transporting 450 passengers from place to another place Airplane Passengers Range (mi) Speed (mph) Boeing 737-100 375 4630 610 Boeing 747 470 4150 610 BAC/Sud Concorde 132 4000 1350 Douglas DC-8-50 146 8720 544

5. Computer Performance: TIME, TIME, TIME • Response Time/execution Time (latency)(The total time required for the computer to complete a task.) — How long does it take for my job to run? — How long does it take to execute a job? — How long must I wait for the database query? — Individual computer user will be interested in this value (how fast the computer can get my job done) • Throughput — How many jobs can the machine run at once? — What is the average execution rate? — How much work is getting done in a given time? — Data center managers are often interested in increasing this value • If we upgrade a machine with a new processor what do we increase? • If we add a new machine to the lab what do we increase? • Example p.242

6. Book's Definition of Performance • For some program running on machine X, PerformanceX = 1 / Execution timeX • "X is n times faster than Y" PerformanceX / PerformanceY = nOR Excution timey / Excution timex = n • Example: • machine A runs a program in 20 seconds • machine B runs the same program in 25 seconds How much faster is A than B? • Do Example: Page 243

7. Execution Time • Elapsed Time (or response time, system performance)– The total time to complete a task • counts everything (disk and memory accesses, I/O , operating system overhead, etc.) • computer is often shared and a processor may work on several programs simultaneously  optimize throughput rather than minimize the elapsed time for one program • a useful number, but often not good for comparison purposes • CPU time (CPU performance) • doesn't count I/O or time spent running other programs • can be broken up into system CPU time (operating system performing tasks on behalf of the program), and user CPU time(CPU time spent in the program) • Our focus: user CPU time • time spent executing the lines of code that are "in" our program

8. Clock Cycles • A measure that relates to how fast the hardware can perform basic functions • A clock that runs at a constant rate and determines when events tale place in the hardware. • These discrete time intervals are called clock cycle. • The length of a clock period both as the time for a complete clock cycle and as the clock rate which is the inverse of the clock period. • Example: clock cycle: 0.25 nanoseconds (ns) clock rate: 4 gigahertz (GHz)

9. 4.2 Clock Cycles • Instead of reporting execution time in seconds, we often use cycles • Our goal  reduce either the length of the clock cycle or the number of clock cycles • BUT, a trade-ff between the number of clock cycles needed for a program and the length of each cycle.

10. time 4.2 Clock Cycles • Instead of reporting execution time in seconds, we often use cycles • Clock “ticks” indicate when to start activities (one abstraction): • cycle time = time between ticks = seconds per cycle • clock rate (frequency) = cycles per second (1 Hz. = 1 cycle/sec)A 4 Ghz. clock has a cycle time

11. How to Improve Performance So, to improve performance (everything else being equal) you can either (increase or decrease?)________ the # of required cycles for a program, or________ the clock cycle time or, said another way, ________ the clock rate.

12. 1st instruction 2nd instruction 3rd instruction ... 4th 5th 6th How many cycles are required for a program? • Could assume that number of cycles equals number of instructions time This assumption is incorrect, different instructions take different amounts of time on different machines.Why?hint: remember that these are machine instructions, not lines of C code

13. Different numbers of cycles for different instructions • Multiplication takes more time than addition • Floating point operations take longer than integer ones • Accessing memory takes more time than accessing registers • Important point: changing the cycle time often changes the number of cycles required for various instructions (more later) time

14. Example • Our favorite program runs in 10 seconds on computer A, which has a 4 GHz. clock. We are trying to help a computer designer build a new machine B, that will run this program in 6 seconds. The designer can use new (or perhaps more expensive) technology to substantially increase the clock rate, but has informed us that this increase will affect the rest of the CPU design, causing machine B to require 1.2 times as many clock cycles as machine A for the same program. What clock rate should we tell the designer to target?" • Don't Panic, can easily work this out from basic principles

15. 4.2 Clock Cycles CUP timeA = (CPU clock cyclesA ) / (clock rate A) 10 seconds = (CPU clock cyclesA ) /(4 * 109 cycles/seconds) CPU clock cyclesA = 10 * 4 * 109 cycles = 40 * 109 cycles CUP clock cyclesA = 40 * 109cycles CUP timeB = 6 seconds = (1.2 * CPU clock cyclesA ) / (clock rateB) clock rateB = (1.2* 40 * 109cycles)/ 6 seconds =8 GHz

16. Now that we understand cycles • A given program will require • some number of instructions (machine instructions) • some number of cycles • some number of seconds • We have a vocabulary that relates these quantities: • cycle time (seconds per cycle) • clock rate (cycles per second) • CPI (cycles per instruction) (average number of clock cycle each instruction takes to execution)a floating point intensive application might have a higher CPI • MIPS (millions of instructions per second)this would be higher for a program using simple instructions

17. Performance • Performance is determined by execution time • Do any of the other variables equal performance? • # of cycles to execute program? • # of instructions in program? • # of cycles per second? • average # of cycles per instruction? • average # of instructions per second? • Common pitfall: thinking one of the variables is indicative of performance when it really isn’t.

18. CPI Example • Suppose we have two implementations of the same instruction set architecture (ISA). For some program,Machine A has a clock cycle time of 250 ps and a CPI of 2.0 Machine B has a clock cycle time of 500 ps and a CPI of 1.2 What machine is faster for this program, and by how much?See page 248 • If two machines have the same ISA which of our quantities (e.g., clock rate, CPI, execution time, # of instructions, MIPS) will always be identical?

19. # of Instructions Example Example: A compiler designer is trying to decide between two code sequences for a particular machine. Based on the hardware implementation, there are three different classes of instructions: Class A, Class B, and Class C, and they require one, two, and three cycles (respectively). The first code sequence has 5 instructions: 2 of A, 1 of B, and 2 of CThe second sequence has 6 instructions: 4 of A, 1 of B, and 1 of C.Which sequence will be faster? How much? (CUP clock cycle1 = (2X1) + (1X2) +(2X3) = 2+2+6 = 10 cycles)What is the CPI for each sequence? (CPU clock cycle2 = (4X1) +(1X2) +(1X3) = 4+2+3 = 9 cycles)

20. 4.3 Benchmarks • Workload: A set of programs run on a computer that is either the actual collection of applications run by a user or is constructed from real programs to approximate such as mix. • Benchmark: Programs specially chosen to measure performance. • Small benchmarks • nice for architects and designers • easy to standardize • can be abused (specific benchmark for special purpose and is not for general case), for example engineering and developing machine • May be misleading • Different classes and applications of computers will require different types of benchmarks. • SPEC (System Performance Evaluation Cooperative) • companies have agreed on a set of real program and inputs • valuable indicator of performance (and compiler technology) • can still be abused

21. Benchmarks • Reproducibility- see page 256, Figure 4-3. • Input data will affect the memory system • Total execution Time • Directly proportional to execution time - our final measure of performance. • The average of the execution times that is directly proportional to total execution time is the arithmetic mean (AM): where Timei is the execution time for the ith program of a total of n in the workload. A weight wi can be assigned to each program to indicate the frequency of the program in that workload. For example 20% of program1 and 80% of program2. By summing the products of weighting factors and execution times, we can obtain clear picture of the performance of the workload. weighted arithmetic mean.

22. Check your self • See page 258. Check your self

23. Benchmark Games • An embarrassed Intel Corp. acknowledged Friday that a bug in a software program known as a compiler had led the company to overstate the speed of its microprocessor chips on an industry benchmark by 10 percent. However, industry analysts said the coding error…was a sad commentary on a common industry practice of “cheating” on standardized performance tests…The error was pointed out to Intel two days ago by a competitor, Motorola …came in a test known as SPECint92…Intel acknowledged that it had “optimized” its compiler to improve its test scores. The company had also said that it did not like the practice but felt to compelled to make the optimizations because its competitors were doing the same thing…At the heart of Intel’s problem is the practice of “tuning” compiler programs to recognize certain computing problems in the test and then substituting special handwritten pieces of code… Saturday, January 6, 1996 New York Times

24. SPEC ‘89 • Compiler “enhancements” and performance

25. SPEC CPU2000

26. SPEC CPU2000 • The execution time measurement are first normalized by dividing the execution time on Sum Ultra 5_10 with 300 MHz processor by the execution time on the measured computer. SPEC ratio. • The bigger the faster. • For a given instruction set architecture, increase in CPU performance can come from three sources: • Increase in clock rate • Improvements in processor organization that lower the CPI.\ • Compiler enhancements that lower the instruction count or generate instructions with a lower average CPI. (e.g. By using simpler instruction sets).

27. SPEC 2000 Does doubling the clock rate double the performance? Can a machine with a slower clock rate have better performance?