CSE403 Software Engineering Autumn 2000 Performance Issues

1. CSE403 Software EngineeringAutumn 2000Performance Issues Gary Kimura Lecture #16 November 5, 2001

2. Today’s Dilbert

3. The coming week • We’re at the half way point • Today - Performance • What is it? • How do we measure it? • How do we improve it? • Wednesday - Group Project Work Day • Friday - Guest Speaker • Next week on Friday - Group Oral Status Reports

4. Group Oral Status Reports • This is almost like a written weekly status report • Except that you get questioned • Plan for 20 minutes • Do this one a little differently with one moderator and no more than four presenters (less is fine) • Your audience for this report is your group and superior • Have an agenda that you can hand out • Other relevant handouts would also add a nice touch

5. Report contents • Accomplishments • From each major subgroup • Any major changes by subgroup • Personnel changes • Functional changes • Etc. • Schedule • Be sure to highlight any changes • Be sure to justify any changes • Give some “Great Job” recognition

6. What do we mean by performance? • The requirements document might specify certain performance goals • Usually some qualitative or quantitative metric based on time or space • Does the system run in a given amount of time on a given amount of hardware? • Competing software companies use unique feature sets to differentiate themselves, but they also use various performance benchmarks to help sell their product

7. IM Performance Requirements • Spend a few minutes to list some AutoIM/BuddyIM performance requirements

8. How do we measure performance? • The most common performance metric is the time needed to complete a given task. But even then time can be measured different ways • Wall clock time. What was the overall time needed to run a particular program or solve a problem? • CPU usage. How much user and/or kernel CPU time was spend running the program? • Average response. For example in a database application what is the average response time for a query? • Longest response. Again for a database application what is the longest response time for a query?

9. Measuring beyond time • Time is not the only performance metric. Other metrics can be • Memory usage or utilization. What is the applications working set size? • Disk space utilization. How much disk space does the application need to use for its program and data? • I/O usage. How much I/O is the application doing and is it distributed over various devices? • Network throughput. How many packets are being sent between machines? • Client capacity. What is the maximum number of clients or users that an application can support?

10. Benchmarks • Commercial benchmark suites, often used by magazines, compare competing systems • Ziff-Davis Benchmark Operation (ZDBOp) is a large developer to benchmark software, and also runs test • Benchmarks try and simulate actual loads, not always successfully • With applications such as word processors, spreadsheets, and databases they typically run scripts and measure the performance • For network application they set up entire labs of servers and clients on a private network. • Software developers often focus on these benchmarks, to the determent sometimes of the real product. • Major software venders are even invited to help ZDBOp tune the system to run the benchmark properly and fix bugs • There is a funny story with Windows CopyFile performance concerning benchmarks

11. Academic versus Commercial • The academic and research community also have their own genre of benchmarks. For example, some old timers favored solving the towers of Hanoi problem, 8 Queens chess problem, or compiling and running a set of Unix sources as a measure of performance. • Side note: many benchmarks miss the fact that performance can degrade over an extended period of time. Some try and account for degradation but not always successfully. A good example is disk fragmentation where we can allocate sectors really fast if we ignore long term fragmentation issues. This is sort of like “pay me now or pay me later.”

12. Both hardware and software affect performance • In hardware there is CPU speed, memory size, network speed, cache size, disk size and speed, etc. • In software its in the basic algorithms, data layout, and various things that we can do in software to exploit the hardware. Think of the software goal as helping the hardware run more efficiently.

13. What can we do to improve performance? • Buying faster and bigger machines should improve performance but this is not always practical • From a software viewpoint we always need to examine the algorithmic complexity of the system. Some examples are: • When sorting data it is important to pick an algorithm appropriate for the data items and keys (Sometimes insertion sort is the right choice) • Avoid redoing complicated calculations multiple times. This is where you weigh the cost of storing an answer versus redoing the calculation • Avoid unnecessary overhead such as maintaining extra pointers linked data structures. Note that this item can go against making a product extensible, maintainable, and easily debug-able. • Are there other examples?

14. More improvements • Picking the right implementation language and constructs within the language also has an impact on performance. • But beyond algorithmic complexity the single most important thing you can do to increase performance is properly layout your data structures. Grouping commonly accessed data item close together typically increases both hardware and software cache efficiency (i.e., a smaller working set). Note however, that this might go against the logically design of the system. Also on a MP system we have to worry about cache line tearing.

15. Still more areas for improvement • Other software things to consider include: • Function call overhead • Process and thread switching overhead • Kernel call overhead • Disk and other I/O bottlenecks • Lock contention, especially on MP systems

16. A sideways look at performance • Percent of CPU usage and idle times are sometimes a measure of system efficiency. This metric is pretty much ignores speed and is more concerned with why the processor is idling. Resolving page faults and lock contention are two of the big bug-a-boos here.

17. Cross platform development • By nature, does cross platform development go against achieving performance goals?