Ch1. Fundamentals of Computer Design 2. Performance

1. ECE Department University of Massachusetts Dartmouth285 Old Westport Rd.North Dartmouth, MA 02747-2300 Ch1. Fundamentals of Computer Design 2. Performance ECE562/468 Advanced Computer Architecture Prof. Honggang Wang Slides based on the PowerPoint Presentations created by David Patterson as part of the Instructor Resources for the textbook by Hennessy & Patterson Updated by Honggang Wang.

2. Administrative Issues (02/04/2014) • Project team set-up due Thursday, Feb. 27 • Each group has 4-5 group members • Two types of groups: ECE 468 and ECE 562 • If you registered ECE 468, you only can join ECE 468 group • If you registered ECE 562, you only can join ECE 562 group • Your group leader must send me an email about your group information by 02/27 • If you missed the first week class, go to the course website for syllabus and 1st lecture. • My office hours: • T./TH. 11 am-12:30 pm, Fri. 1:00-2:00 pm www.faculty.umassd.edu/honggang.wang/teaching.html

3. Review of Lecture #1 In the first lecture, we covered the • Technology Trends • Define and quantity power • Define and quantity relative cost • Define and quantity dependability

4. Outline • Careful, quantitative comparisons: Performance • Compute Mean • Benchmark

5. Definition: Performance • Performance is in events per sec: • bigger is better • If we are primarily concerned with response time: • the time between the start and the completion of an event (execution time). • " X is n times faster than Y" means:

6. Performance: What to measure • To increase predictability, collections of benchmark applications, called benchmark suites, are popular • reliable measure of the performance is the execution time of real program • the users would simply compare the execution time of their workloads • a popular measure of performance of processors with a variety of applications • SPECCPU: popular desktop benchmark suite • CPU only, split between integer and floating point programs • SPECint2000 has 12 integer, SPECfp2000 has 14 integer pgms • SPECCPU2006 to be announced Spring 2006 • SPECSFS (NFS file server) and SPECWeb (WebServer) added as server benchmarks • Transaction Processing Council measures server performance and cost-performance for databases (www.tpc.org) • TPC-C Complex query for Online Transaction Processing • TPC-H models ad hoc decision support • TPC-W a transactional web benchmark • TPC-App application server and web services benchmark

7. How Summarize Suite Performance (1/5) • Arithmetic average of execution time of all pgms? • But they vary by 4X in speed, so some would be more important than others in arithmetic average • Could add a weights per program, but how pick up weight? • Different companies want different weights for their products • SPECRatio: Normalize execution times to reference computer, yielding a ratio proportional to performance

8. How Summarize Suite Performance (2/5) • If program SPECRatio on Computer A is 1.25 times bigger than Computer B, then • Note that when comparing 2 computers as a ratio, execution times on the reference computer drop out, so choice of reference computer is irrelevant

9. How Summarize Suite Performance (3/5) • Since ratios, proper mean is geometric mean (SPECRatio unitless, so arithmetic mean meaningless) 1. The SPECRatio are multiplied and then the nth root (where n is the count of SPECRatio in the set) of the resulting product is taken 2. indicates the central tendency or typical value of a set of numbers

10. How Summarize Suite Performance (3/5) • Suppose an orange tree yields 100 oranges one year and then 180, 210 and 300 the following years, so the growth is 80%, 16.7% and 42.9% for each year respectively. • Using the arithmetic mean calculates a (linear) average growth of 46.5% (80% + 16.7% + 42.9% divided by 3). • However, if we start with 100 oranges and let it grow 46.5% each year, the result is 314 oranges, not 300, so the linear average over-states the year-on-year growth. • Instead, we can use the geometric mean. • Growing with 80% corresponds to multiplying with 1.80, so we take the geometric mean of 1.80, 1.167 and 1.429, i.e. , thus the "average" growth per year is 44.3%. • If we start with 100 oranges and let the number grow with 44.3% each year, the result is 300 oranges.

11. How Summarize Suite Performance (4/5) • Does a single mean well summarize performance of programs in benchmark suite? • Can decide if mean is a good predictor by characterizing variability of distribution using standard deviation • Like geometric mean, geometric standard deviation is multiplicative rather than arithmetic • Can simply take the logarithm of SPECRatios, compute the standard mean and standard deviation, and then take the exponent to convert back:

12. How Summarize Suite Performance (5/5) • Standard deviation is more informative if know distribution has a standard form • bell-shaped normal distribution, whose data are symmetric around mean • lognormal distribution, where logarithms of data--not data itself--are normally distributed (symmetric) on a logarithmic scale • For a lognormal distribution, we expect that 68% of samples fall in range 95% of samples fall in range • Note: Excel provides functions EXP(), LN(), and STDEV() that make calculating geometric mean and multiplicative standard deviation easy

13. Outside 1 StDev Example Standard Deviation (1/2) • GM and multiplicative StDev of SPECfp2000 for Itanium 2

14. Outside 1 StDev Example Standard Deviation (2/2) • GM and multiplicative StDev of SPECfp2000 for AMD Athlon

15. Comments on Itanium 2 and Athlon • Standard deviation of 1.98 for Itanium 2 is much higher-- vs. 1.40--so results will differ more widely from the mean, and therefore are likely less predictable • Falling within one standard deviation: • 10 of 14 benchmarks (71%) for Itanium 2 • 11 of 14 benchmarks (78%) for Athlon • Thus, the results are quite compatible with a lognormal distribution (expect 68%)

16. Fallacies and Pitfalls (1/2) • Fallacies - commonly held misconceptions • When discussing a fallacy, we try to give a counter example. • Pitfalls - easily made mistakes. • Often generalizations of principles true in limited context • Show Fallacies and Pitfalls to help you avoid these errors • Fallacy: Benchmarks remain valid indefinitely • Once a benchmark becomes popular, tremendous pressure to improve performance by targeted optimizations or by aggressive interpretation of the rules for running the benchmark: “benchmarksmanship.” • 70 benchmarks from the 5 SPEC releases. 70% were dropped from the next release since no longer useful • Pitfall: A single point of failure • Rule of thumb for fault tolerant systems: make sure that every component was redundant so that no single component failure could bring down the whole system (e.g, power supply)

17. Fallacies and Pitfalls (2/2) • Fallacy: Rated MTTF of disks is 1,200,000 hours or 140 years, so disks practically never fail • manufactures will put thousands of disks in a room, run them for a few months and count the number that fail • But disk lifetime is 5 years  replace a disk every 5 years; on average, 28 replacements wouldn't fail • A better unit: % that fail (1.2M MTTF = 833 FIT) • Fail over lifetime: if had 1000 disks for 5 years= 1000*(5*365*24)*833 /109 = 36,485,000 / 106 = 37 = 3.7% (37/1000) fail over 5 yr hr MTTF) • But this is under pristine conditions lifetime (1.2M • little vibration, narrow temperature range  no power failures • Real world: 3% to 6% of SCSI drives fail per year • 3400 - 6800 FIT or 150,000 - 300,000 hour MTTF [Gray & van Ingen 05]

18. Summary • Quantify and summarize performance • Ratios, Geometric Mean, Multiplicative Standard Deviation

19. Things To Do Next Topics Fundamentals of Computer Design 3. Principles • Find your partners for the class project • Feb. 27, Thursday (email me the team information) • Check out the class website about • lecture notes • reading assignments