1 / 74

Where's Waldo Uncovering Hard-to-Find Application Killers

Where's Waldo Uncovering Hard-to-Find Application Killers. Claire Cates SAS Institute, Inc Claire.Cates@sas.com. Always something different. Just like the Waldo pictures, performance is always different and not easy to find. Performance based on the test that was run

claire
Download Presentation

Where's Waldo Uncovering Hard-to-Find Application Killers

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Where's Waldo Uncovering Hard-to-Find Application Killers Claire Cates SAS Institute, Inc Claire.Cates@sas.com

  2. Always something different • Just like the Waldo pictures, performance is always different and not easy to find. • Performance based on the test that was run • Performance based on the data that was used • Performance based on the hardware used • Performance based on the number of users

  3. What is a Performance Problem • When jobs cannot execute within a required time period. • Something that makes the system run slower than it should • When a system cannot sustain expected or unexpected growth (scalability) • More users • More data • More processors

  4. Causes of Performance Problems • Hardware • Software • Integration of software systems • Network • Web

  5. How Do You Find Performance Problems • Vendor profilers • Looking at system variables (cache hits, IO timings,…) • Profilers that analyze the code • Code reviews • Network sniffers • Profiles that profile the entire machine • Code instrumentation

  6. Common Software Performance Problems Antipatterns • Excessive Memory Allocation • The Ramp • Unnecessary Processing • One Lane Bridge • How Many Times Do I have to tell you. • Unbalanced Processing • More is Less

  7. Excessive Memory Allocation • Memory Allocations/Frees are expensive • OS calls • If your system is multi-threaded, locking may need to occur • Often times memory is allocated temporarily within a function • Often times structures are allocated and freed and not reused • Memory allocations in loops. • Possible Fixes • Use a stack variable instead of temporary memory allocations • Maintain a free chain for structures that are used often • Make one large allocation outside loop – split it within the loop

  8. The Ramp • When processing slows down over time • Can be caused by long lists that grow and need to be searched • Can be caused by too many resources being used • Another kind of Ramp happens when the system doesn’t scale well. • Possible fixes • Use a different algorithm when you know large amounts of data will be required • Make sure all resources (especially memory) are returned.

  9. Unnecessary Processing • Loops that contain statements that do not need to execute on each iteration of the loop • Macros that are inefficient because they are too general • Zeroing of allocated memory that does not need to be zeroed. • Inefficient algorithms • Possible fixes • Rework algorithm • Don’t use macros

  10. One Lane Bridge • Resource contention – every thread waiting on the same resource • Eventually causes a traffic jam • Possible fixes • Change the algorithm to use less resources that are shared

  11. How Many Times Do I Have To Tell You • Calling a routine too often with the same parameters • Sometimes a routine (foo) is the hot spot yet, (foo) is fine, it is just that foo is called to often. • Most common occurrence of this that I see occurs when (afoo) (bfoo) (cfoo) (foo) So cfoo is called twice and foo is called 4 times all because afoo does not know what bfoo and cfoo actually call

  12. Unbalanced Processing • Occurs in a multi-threaded environment running on a multi-processor machine. • When one or two threads do all the processing while other threads are idle • Possible fixes • Rebalance the work in the threads so that all threads process a similar amount of work

  13. More is Less or Thundering Herd • Adding threads does not make a system run faster • The system spends its time thrashing • Thrashing can occur because of • Excessive thread switching • Destroying the system memory or I/O caches when each thread runs • Possible fixes • Don’t create more threads than there is work to be done

  14. Common Profilers in Use at SAS • InTune – z/OS • VTune – Intel – PCs • Caliper – HP • Rational Quantify – Windows, Solaris, HP, Linux • Jprobe • OS tools – not really profilers but gives information on • System calls • IO bottlenecks • Caching problems

  15. SAS’s use of Quantify • Other Profilers we use • Event / timer based • Based more on the function level, not the source level • Network traffic & other processes on the machine distorted the results • Great results from all profilers • Tips I’m using with Quantify do apply to these other tools

  16. Quantify differences… • Quantify • Counts instruction cycles for the user code • Times the system calls (can be turned off) • Does not have to run on a clean machine (a machine with no other activity) – developer machine • Provides performance information down to the source line level • Performance can be broken down by threads

  17. A Little about Quantify • Quantify integrates itself with your system. As your system executes, Quantify will accumulate a mass of data. • Function counts for every function used • The amount of time spent in every function • The amount of time spent in every function plus the amount of time spent in all its descendent routines. • Full Call tree information

  18. Quantify Quick Overview

  19. Excessive Memory Allocation

  20. How Many Times Do I Have to Tell You?

  21. More Is Less

More Related