Advanced Troubleshooting with Debug Diagnostics on IIS 6

Advanced Troubleshooting with Debug Diagnostics on IIS 6 Chris Adams Program Manager IIS Product Unit Microsoft Corporation

Agenda • Debug Diagnostics 1.0 • Dig into crash scenarios • Understanding hangs • Investigating memory leaks • Session Summary

Debug Diagnostics 1.0 • “Core” pieces of DebugDiag • Understanding “Rules” • Using DebugDiag’s User Interface

Core pieces of Debug Diag • 3 pieces make up Debug Diag 1.0 • Service • Host (also called Controller) • User Interface DebugDiag.exe DbgHost.exe DbgSvc.exe

Core pieces of Debug Diag The Service – DbgSvc.exe • DbgSvc.exe • To register the service: • C:\DebugDiag> DbgSvc /service • To start the service: • C:\DebugDiag> net start DbgSvc • To stop the service • C:\DebugDiag> net stop DbgSvc • Unregister the service • C:\DebugDiag> DbgSvc /unregserver • Service is multi-purposed • Interacts with the Debugger Host • Allows: • Service required to attach invasively to process • To run from Terminal Services

Attach DbgSvc.exe DbgHost.exe Core pieces of Debug Diag The Host – DbgHost.exe • Host • Where the DbgEng.dll instance is loaded • Exposes 3 classes DbgControl: Attach/Detach from processes or open/analyze a memory dump DbgObj: Collect process or memory dump information Manager: Mainly to output analysis data to the report file

DebugDiag.exe Core pieces of Debug Diag The User Interface – DebugDiag.exe • User Interface designed to simplify: • Rule creation • View processess • Initiate analysis of memory dumps • Add analysis scripts • Change properties for Debug Diag Control Scripts DbgHost.exe

Crash Scenarios • Background of crashes • Using DebugDiag to quickly identify root cause of crashes

Background: Exceptions • What is an exception? • How do applications cause exceptions • What is a “handled” exception? • Unhandled exceptions need debugging and resolution Access Violation: C000 0005 Breakpoint Exception: 8000 0003 Stack Overflow: C000 00FD C++ exception: E06D7363 Managed exception: E0434F4D Invalid Handle: C000 0008

Background: Heap (2) • What is a heap? • Heap is a data structure • Heap Corruption: • Certain rules must be followed when using the heap • When rules are violated, heap corruption occurs • Heap corruption crashes • Code that corrupts heap is usually not the ones impacted

Debugging Crasheswith Debug Diagnostics:Crash Rule Chris Adams Program Manager IIS Product Unit

Hang Scenarios • Investigating the theory of “process hangs” • Using DebugDiag to debug “process hangs”

Theory of “Process Hangs” • What is a process? • Memory boundary for execution of code • Allocated by Operating System done by calling CreateProcess or CreateProcessAsUser API • Processes do not execute code • Contains a minimum of 1 thread • What is a thread? • An independent sequence of instructions executing within a process

What are common thread pools for IIS? • ATQ thread pool • ASP thread pool • ASP.NET thread pool Theory of “Process Hangs” • Thread implications on applications • The level of concurrency (virtual) is dictated by the number of threads • Real concurrency is dictated by the number of execution units • 1 processor = one or more execution units

Runaway Threads • A runaway thread is a thread stuck in a infinite (or near) loop • Performing a task that is consuming CPU • Either by design or because of error conditions • Runaway thread is taking CPU resources away from other threads • Often takes away from good threads • Causes performance degradation for server

SQL IIS Defining a Blocked Thread • A thread that isn’t using any CPU resources • Waiting on database calls to return • Waiting on web service calls to return • Waiting on a socket/network call to return • Waiting on a lock Thread 3 Thread 4 Thread 1

Locks and Deadlocks • What is a lock? • A synchronization mechanism to protect resources from being corrupted due to simultaneous access by multiple threads • Why do developers use locks? • Protecting data • Protecting other resources that are not memory based • Impact on applications if locks are poorly implemented • Causes threads to block • Can cause deadlock situations when using multiple locks in the application • Process hangs when no threads are available

Debugging Deadlocks and Process Hangs with DebugDiag: Hang Rule Chris Adams Program Manager IIS Product Unit

Looking into Leaks… • What is a leak? • Types of Leaks • Debugging a Leak with Debug Diagnostics

Leak Scenarios (2) • Why are leaks difficult to identify and resolve? • Server applications often cache memory allocations • The allocations are long term • These applications, to outsiders, look to have memory leaks • There are many other caches • Using Performance Monitor and seeing memory growth does not equal a memory leak

Leak Scenarios (3) • What are short-term allocations? • Memory expensive applications use “spike” allocations but are destroyed immediately after request (not long- sustaining) These are not Memory Leaks

Leak Scenarios (4) • Difference between cache and memory leaks? • Memory leaks are caused by unbounded allocations • Cache memory usage rises quickly, but tops out after application is stabilized

Nailing a Memory Leak in a Web Application using DebugDiag: Memory Rule Chris AdamsProgram Manager IIS Product Unit

Session Summary • Debugging Live Web Applications without downtime is challenging • Crashes cause processes to terminate due to unhandled exceptions • Hangs are caused due to race conditions, lock contention, network blocking, database blocking, infinite loops and deadlocks • Leaks are caused by the failure to release memory allocated beyond typical short term allocations and caching • Debug Diagnostics has an extensibility model that allows it to successfully capture and analyze common debugging scenarios to determine the root cause

Debug Diagnostics 1.0 web resources • http://www.iisdiagnostics.com

Advanced Troubleshooting with Debug Diagnostics on IIS 6