1 / 42

Profiling and Detecting Bottlenecks in Software

Profiling and Detecting Bottlenecks in Software. Bryan Call OSCON 2011 Yahoo! Engineer and Apache Commiter. Overview. Why profile your code? Rules of thumb Profiling pitfalls Types of bottlenecks Basic command line tools What is a profiler? Types of profilers Profiling Examples

courtneya
Download Presentation

Profiling and Detecting Bottlenecks in Software

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Profiling and Detecting Bottlenecks in Software Bryan Call OSCON 2011 Yahoo! Engineer and Apache Commiter

  2. Overview • Why profile your code? • Rules of thumb • Profiling pitfalls • Types of bottlenecks • Basic command line tools • What is a profiler? • Types of profilers • Profiling Examples • Ways to improve performance

  3. Why profile your code? • Better understanding of your application and architecture • Reduced hardware and maintenance costs • Less hardware to setup and maintain • Learn how to be a better coder • Look smart

  4. Rule of thumb • 80/20 rule • 80% of the runtime using only 20% of the code • Some people say 90/10

  5. Profiling pitfalls • Pre-optimization, waist of time • Optimizing the 80% of the code that only runs 20% of the time • Don’t fully understand the architecture or workload • Over optimize code • Can overcomplicate code

  6. Types of Bottlenecks • CPU • Disk • Network • Memory • Lock contention • External resources • Databases, web service, etc..

  7. Basic Command-line Tools • top, htop (great for threaded apps) • vmstat, dstat • strace • time

  8. htop Example • 4 core server

  9. htop Example • 24 “core” – 12 core with hyper-threading

  10. dstat Example – CPU bottleneck • Apache Traffic Server – 470B objects in cache

  11. Understand Your Workload • Changing the workload can change the bottleneck

  12. dstat Example – Network bottleneck • Apache Traffic Server – 200KB object in cache

  13. dstat Example – Disk bottleneck • dd - /dev/zero to raid0 (two drives)

  14. dstat Example - syscall issue • Writes are too small and can’t max out the disk

  15. strace Example • Effects performance ~100MB/sec to 1.1MB/sec

  16. What is a Profiler? • Dynamic program analysis • Shows • Frequency of functions called • Usage of lines in code • Duration of function calls

  17. Types of Profilers • Statistical • Examples: oprofile, google profiler • Good for interactive systems with lots of code • Doesn't slow down the application much (1% to 8%) • Fixed cost • Doesn't take up more CPU as the number of function calls per second increases

  18. Types of Profilers • Instrumenting • Examples: valgrind'scallgrind, gprof • More detail (time for each function call) • Can make programs much slower • Good for non-interactive systems

  19. Oprofile • Requires kernel driver, need root access • System wide profiling, profiles everything running • Application doesn’t know about the profiler • Scripts to convert output for kcachegrind

  20. Oprofile Example • Profiling ab (Apache Bench) • 30K rpswith profiler, 32K rps without

  21. Oprofile Example

  22. Oprofile Example

  23. Oprofile Example • Showing everything that was running

  24. Google profiler • All in userland • Profiles specific applications, not system wide • Command-line LD_PRELOAD support • Support to build it into your application • Has graphing built in

  25. Google Profiler Example • Profiling ab (Apache Bench) • 30K rps with profiler, 32K rps without

  26. Google Profiler Example

  27. Google Profiler Example • Making a diagram of the profile

  28. Google Profiler Example

  29. Google Profiler Example

  30. Vagrind’scallgrind • All in userland • Requires no code changes • Really slows down your application • Lots of detail since it is not sampling

  31. callgrind Example • Running callgrindon ab (Apache Bench) • 1.6K rps with profiler, 32K rps without - 95% slower

  32. callgrind Example

  33. callgrind Example - kcachegrind

  34. Recap • Understand your workload • Find your bottleneck • Profile

  35. Ways to Improve Performance • Caching • Don't do the same work twice • Choose the correct algorithms and data structures • dqueuevs list, hash vs trees, locks vs read/write locks, bloom filter • Memory allocation • Reuse memory, stack vs heap, tcmalloc • Make fewer system calls • Larger writes and reads • Faster hardware • Bonded NICs, SSDs or RAID, CPU more cores

  36. References • Email: bcall@apache.org • How to profile ATS • https://cwiki.apache.org/TS/profiling.html

  37. Links to Software • dstat • http://dag.wieers.com/home-made/dstat/ • htop • http://htop.sourceforge.net/ • oprofile • http://oprofile.sourceforge.net/news/ • google profiler (part of the prof tools) • http://code.google.com/p/google-perftools/ • callgrind • http://valgrind.org/docs/manual/cl-manual.html • kcachegrind • http://kcachegrind.sourceforge.net/html/Home.html

  38. Appendix setup httpd/ab: cd ~/tmp/ wget http://mirror.candidhosting.com/pub/apache//httpd/httpd-2.2.19.tar.bz2 tar xf httpd-2.2.19.tar.bz2 cd httpd-2.2.19 ./configure gmake -j 8 cd support

  39. Appendix oprofile commands: # at the start - only need to this once after reboot - because of watchdog timers sudoopcontrol --deinit sudo bash -c 'echo 0 > /proc/sys/kernel/nmi_watchdog' sudoopcontrol --no-vmlinux sudoopcontrol --start-daemon sudoopcontrol --reset sudoopcontrol --status # in another terminal run ab - needs to run for 60 seconds, increase -n if need be .libs/ab -k -n 2000000 -c 100 -X homer.bryancall.com:8080 http://l.yimg.com/a/i/ww/met/mod/ybang_22_111908.gif sudoopcontrol -s; sleep 60; sudoopcontrol -t sudoopcontrol --dump sudoopreport --symbols .libs/ab 2>/dev/null sudoopreport -cg 2>/dev/null | head -50

  40. Appendix google profiler commands: export CPUPROFILE=/tmp/mybin.prof LD_PRELOAD="/usr/lib64/libprofiler.so" .libs/ab -k -n 2000000 -c 100 -X homer.bryancall.com:8080 http://l.yimg.com/a/i/ww/met/mod/ybang_22_111908.gif pprof --text .libs/ab /tmp/mybin.prof | head pprof --pdf .libs/ab /tmp/mybin.prof > ~/Desktop/ab.pdf

  41. Appendix callgrind commands: rm -f callgrind.out.* # clean up anything there valgrind --tool=callgrind .libs/ab -k -n 100000 -c 100 -X homer.bryancall.com:8080 http://l.yimg.com/a/i/ww/met/mod/ybang_22_111908.gif callgrind_annotate --tree=caller callgrind.out.* kcachegrindcallgrind.out.*

  42. Notes • Had problems with --separate=lib or --separate=thread not changing output on Fedora Core 15

More Related