Advanced Performance Forensics

1. AdvancedPerformance Forensics Uncovering the Mysteries of Performance and Scalability Incidents through Forensic Engineering Stephen Feldman Senior Director Performance Engineering and Architecture stephen.feldman@blackboard.com

2. Sessions Goals The goals of today’s session are… • Introduce the practice of performance forensics. • Present an argument for session level analysis. • Discuss the difference between Resources and Interfaces. • Present tools that can be used for performance forensics at different layers of the architectural stack and the client layer.

3. Definition of Performance Forensics • The practice of collecting evidence, performing interviews and modeling for the purpose of root cause analysis of a performance or scalability problem. • In context of a performance (response time problem) • Discussing an individual event (session experience) • Performance problems can be classified in two main categories: • Response Time Latency • Queuing Latency

4. Performance Forensics Methodology

5. Putting Performance Forensics in Context • Emphasis on the user and the user’s actions and experiences. • How can this be measured? • Capture the response time experience and the response time expectations of the user. • Put into perspective user action in-line with the goals of Method-R (what’s most important to the business) • Identify the contributors of response latency • Everyone needs to be involved

6. Measuring the Session • When should this happen? • When a problem statement cannot be developed from the data you do have (evidence or interviews) and more data needs to be collected. • How should you go about this? • Want to minimize disruption to the production environment. • Adaptive collection: Less Intensive to More Intensive over time. Basic Sampling Continuous Collection Profiling

7. Resources vs. Interfaces • One of the most critical data points to collect • Interfaces are critical for understanding throughput and queuing models. • Queuing is another cause of latency • Also a cause of time-outs • Resources are critical for understanding the cost of performing a transaction. • Core Resources: CPU, Memory and I/O • Response Time = Service Time + Queue Time

8. The Importance of Wait Events • Rise of Session Level Forensics • Underlying theme with all of these tools that “Session” is more important then “System” • Wait event tuning used to account for latency • Exists in SQL Server (Waits and Queues) and Oracle (10046) • Other components not mature enough to represent • Waits are statistical explanations of latency • Each individual wait event might be deceiving, but looking at both aggregates and outliers can explain why a performance problem exists. • When sampling directly, usually only have about 1 hour to act on the data.

9. Performance Forensics Tools

10. Categories of Tools • HTTP and User Experience • JVM Instrumentation Tools • Database Instrumentation • Session and Wait Event • Cost Execution Plans • Profilers

11. Breaking Down Latency

12. Fiddler2 • Fiddler 2 measures end-to-end client responsiveness of a web request. • Little to no overhead (less intrusive forensics) • Captures requests in order to present http codes, size of objects, sequence of loading, time to process request, performance by bandwidth speed. • Rough estimation of User Experience based on locality. • Inspects every detail of the http request • Detailed session inspection • Breakdown of http transformation • Other Tools in Category: Y-slow/Firebug, Charlesproxy, liveHTTPheaders and IEInspector

14. Coradiant Truesight • Commercial tool used for passive user experience monitoring. • Captures page, object and session level data. • Capable of defining Service Level Thresholds and Automatic Incident Management. • Used to trace back session as if you were watching over the user’s shoulder. • Exceptional tool for trend analysis. (Less Intrusive) • Primarily used in forensics as evidence for analysis. • Other Tools in the Category: Quest User Experience and Citrix EdgeSight

15. Coradiant Truesight

16. Coradiant Truesight

17. Log Analyzers • Both commercial and open source tools are available to parse and analyze http access logs. • Provides trend data, client statistical data, http summary information. • Recommend using this data to study request and bandwidth trends for correlation purposes with resource utilization graphs. • Such a large volume of data. • Recommend working within small time slices • Post-processing tool (No Impact to Application) • Examples: Urchin, Summary, WebTrends, SawMill, Surfstats and AlterWind Log Analyzer

18. JSTAT • Low intrusive statistic collector that provides • Percentages of usage by each region • Frequency/Counts of collections • Time spent in pause state • Can be invoked any time without restarting the JVM by obtaining the Process ID • Exception is on Windows when the JVM is run as a background service • Critical for understanding windows of stall times between sampling • Assume you collect every 5 seconds and observe a 3 second pause time • Means the application could only work for 2 seconds

19. JSTAT

20. Process of Garbage Collection

21. Process of Garbage Collection

22. -VerboseGC and -Xloggc • JVM flags that invoke JVM logging • Verbose JVM logging is a low-overhead collector (less intrusive measurement) • Requires a restart of the instance to run • -XX:+PrintGCDetails is a recommended setting to be used with: • -XX:+PrintGCApplicationConcurrentTime • -XX:+PrintGCApplicationStoppedTime • Provides aggregate statistics about Pause Times versus Working Times.

23. -VerboseGC and -Xloggc

24. IBM Pattern Modeling Tool for Java GC • Post processing tool used for visualizing a –VerboseGC or –Xloggc file. • Can make the analysis efforts for analyzing a log file substantially easier. • Represents pauses/stalls at particular times • Has no affect on the application environment as it reads a log file that is dormant.

25. IBM Pattern Modeling Tool for Java GC

26. JHAT, JMAP and SAP Memory Analyzer • Jhat: Java Heap Analysis Tool takes a heap dump and parses the data into useful and human-digestible information about what's in the JVM's memory. • JMap: Java Memory Map is a JVM tool that provides information about what is in the heap at a given time. • Provides text and OQL views into JHat data • SAP Memory Analyzer will visualize the JHat output • Should be run when a problem is occurring right now • When the system is unresponsive • When the JVM runs into continuous collections

28. ASH • ASH: Active Session History • Samples session activity in the system every second. • 1 hour of history in memory for immediate access at your fingertips • ASH in Memory • Collects active session data only • History v$session_wait + v$session + extras • Circular Buffer - 1M to 128M (~2% of SGA) • Flushed every hour to disk or when buffer 2/3 full (it protects itself so you can relax) • Tools to Consider: SessPack and SessSnaper

29. SQL Server Performance Dashboard • Feature of SQL Server 2005 SP2 • Template report that take advantage of DMVs • Provides views into wait events • Doesn’t link events to SQL IDs in the report • Provides aggregate views of wait events • Session Level DMVs (sys.dm_os_wait_stats and sys.dm_exec_sessions) • Complimentary Tools: SQL Server Health and History Tool and Quest Spotlight for SQL Server

31. Importance of Cost Execution Plans • Can be run on databases with low overhead • Do not need the literal values to run • Both SQL Server and Oracle can run “Estimated Cost Plans” • Each database uses an “Optimizer” that determines the best path of execution of SQL • Calculates IO, CPU and Number of Executes (Loop Conditions) • Understanding cost operations on a particular object can help change your tuning strategy (ex: TABLE ACCESS BY INDEX ROWID) • Cost is time • Query cost refers to the estimated elapsed time, in seconds, required to complete a query on a specific hardware configuration.

32. RML and Profiler • The RML utilities process SQL Server trace files and view reports showing how SQL Server is performing. • Which application, database or login is using the most resources, and which queries are responsible for that. • Whether there were any plan changes for a batch during the time when the trace was captured and how each of those plans performed. • What queries are running slower in today's data compared to a previous set of data • Profiler captures statements, query counts/statistics, wait events • Can capture and correlate profile data to Perfmon data • Heavy overhead with both • Other Tools to Consider: Quest Performance Analysis for SQL Server

33. Oracle OEM and 10046 • Oracle finally delivered with OEM with a web-based interface. • Performance dashboard provides great historical and present overview • Access to ADDM and ASH simplifies job of DBA • SQL History • Problems • licensing somewhat cost prohibitive • Still doesn’t provide wait events • For 10046 still need to consider profiling on your own and using a profiler reader like Hotsos P4. • Difficult to trace and capture sessions

35. Want More? • Check-out my blog for postings of the presentation: http://sevenseconds.wordpress.com • To view my resources and references for this presentation, visit www.scholar.com • Simply click “Advanced Search” and search by sfeldman@blackboard.com and tag: ‘bbworld08’ or ‘forensics’