problem isolation with web- based applications aaron barth ...

1. Problem Isolation with Web- Based Applications

2. 2

3. 3 Objective To provide the necessary tools to effectively troubleshoot and isolate some of the most complex problems with Web-based applications.

4. 4 Architecture Microsoft� Internet Information Server (IIS) architecture CGI ISAPI COM components

5. 5 IIS Architecture IIS Admin Service (Inetinfo.exe) World Wide Web Publishing Service (W3svc.dll) FTP Publishing Service (Ftpsvc2.dll) SMTP Service (Smtpsvc.dll) NNTP Service (Nntpsvc.dll)

6. 6 CGI Applications CGI applications have an extension of .EXE Run in separate process Can use Script Map to launch Example: .pl ? PERL.EXE %s %s Disadvantages Performance may not be optimal Create a new process for each Web request

7. 7 ISAPI Applications ISAPI applications have an extension of .DLL Run in same process Two types of ISAPI applications ISAPI Extension Explicitly requested through URL Can use Script Map to launch Example: .asp ? ASP.DLL ISAPI Filter Can handle encryption, authentication, or URL remapping

8. 8 ISAPI Applications (cont.) Advantages Fast Loads In Process with the EXE only once and can access memory within the calling process. Disadvantages Server Stability If a poorly written ISAPI application becomes unstable, it may take down the whole Web server as a result.

9. 9 COM Components Either has an extension of .DLL or .EXE Usually runs in the same process Exceptions: OOP Web sites MTS packages Local servers Can be written in many languages Visual Basic�, Visual C++�, Visual J++�, and others Called from ASP pages Same disadvantage as ISAPI applications have when running In Process

10. 10 Common Symptoms Crash Hang Memory leak 100% CPU utilization

11. 11 Crash Unhandled Exceptions Dr. Watson �Process xxxx.exe referenced memory that could not be read.� ASP 0115: Trappable Error Occurred in External Object An ASP page called a COM component that encountered an exception.

12. 12 Hang Types of Hangs ASP pages stop responding Users get an hourglass in their browser when trying to access any .ASP page Commonly caused by ASP queuing requests HTM pages stop responding Neither ASP pages nor HTM pages serve TELNET to port 80 fails to connect TELNET to port 80 succeeds but �GET /� fails

13. 13 Memory Leak INETINFO.EXE appears to be leaking memory Performance Monitor Process ? Private Bytes ? inetinfo.exe Task Manager Mem Usage column Less precise than Performance Monitor More Information: Using Performance Monitor to detect memory leaks in NT Previous WebCast by Ben Christenbury

14. 14 100% CPU Utilization INETINFO.EXE takes approximately 100% of the CPU processor time. Performance Monitor Process ? % Processor Time ? inetinfo.exe Task Manager CPU column Less precise than Performance Monitor Usual Cause Sequence of instructions in a tight (endless) loop

15. 15 Information Gathering Ask the right questions Examine the event log Examine the IIS log Gather Performance Monitor data

16. 16 Ask the Right Questions How many sites on the server? What does the virtual directory structure look like? Are there any custom ISAPI filters? Are there any custom ISAPI extensions? Are there any custom COM components? What database(s) is this server connecting to?

17. 17 Ask the Right Questions (cont.) Tools to obtain some of this information: Mdutil enum_all >metabase1.txt Found on the Windows NT� Option Pack CD Tlist inetinfo.exe > tlist-inetinfo.txt Found in the Microsoft� Windows NT 4.0 Resource Kit MetaEdit Found in the IIS 4 Resource Guide IIS Exception Monitor http://msdn.microsoft.com/workshop/server/iis/ixcptmon.asp

18. 18 Examine the Event Log Check For W3SVC errors WAM errors ASP errors Note source ASP file and Line Number if present

19. 19 Examine the Event Log (cont.) MTS Exception Errors SP4 and later Look for the stack in the details An object call caused an exception. (Package: IIS-{WebCast//Root//OOP}) (IID: {51372AEF-CAE7-11CF-BE81-00AA00A2FA25}) (Method: 3) (Microsoft Transaction Server Internals Information: File: d:\viper\src\runtime\mtxex\activity.cpp, Line: 889) (Exception: C0000005) (Address: 0x77f6481a) ntdll!_RtlAllocateHeap@12 + 0x198 MSVCRT!malloc + 0xC2 vbscript!Scanner::ScanIdentifier(int) + 0x80 vbscript!Scanner::Scan(void) + 0xFE vbscript!Parser::Parse(unsigned short const *,unsigned long) + 0x32 vbscript!Parser::ParseSource(class ExecBody * *,class COleScript *,unsigned short const *,unsigned long,void *,long,class CompileScriptException *,unsigned short const *,unsigned short const *,unsigned short const *) + 0xCD vbscript!Parser::ParseSource(class ExecBody * *,class COleScript *,unsigned short const *,unsigned long,void *,long,class CompileScriptException *,unsigned short const *,unsigned short const *,unsigned short const *) + 0x5E

20. 20 Examine the IIS Log Look for 500 status codes 0115 errors from ASP pages Examine Last requests completed prior to crash

21. 21 Gather Performance Monitor Data Hang Create a Performance Monitor log Log Process, Memory, ASP, and WWW counters Look for Active Server Pages ? Requests Queued Are ASP requests are queuing? Memory ? Available Bytes Does the system have enough memory free?

22. 22 Gather Performance Monitor Data (cont.) Memory Leak Create a Performance Monitor log Log Process, Memory, Active Server Pages, and WWW counters Look for Process ? Private Bytes ? inetinfo.exe Are Private Bytes steadily increasing over time? Active Server Pages ? Memory Allocated Does Memory Allocated continue to grow even after 24 hours of uptime? RESOLUTION: Tune ASP Template cache

23. 23 Gather Performance Monitor Data (cont.) 100% CPU Utilization Create a Performance Monitor chart While CPU(s) are using >90% Processor Time Add the Process ? % Processor Time -> inetinfo.exe See if inetinfo.exe is taking up the CPU time Add the Thread ? % Processor Time ? inetinfo==> <Thread Number> for all inetinfo threads Try to identify 1 or 2 threads that are consistently >90% Run the IIS Exception Monitor In Process ? Automatic Manually trip log Analyze the log Open the log and examine the stack of the problematic thread(s)

24. 24 Application Isolation The goals of Isolation Methods of Isolation

25. 25 The Goal of Isolation The goal of Process Isolation is to move as many DLLs out of inetinfo.exe as possible so you can identify what subset of DLLs are causing the problem.

26. 26 Methods of Isolation Removal method �Separate Memory Space� method MTS Package method

27. 27 Removal Method Steps Try to isolate the application that is causing the problem by removing all suspect applications from the server. Move suspect applications from one problem server to a separate server, and see if the problem follows to the second server. Disadvantages Very intrusive for application operation Very time-consuming process

28. 28 �Separate Memory Space� Method Steps Select the application that you would like to isolate. Go to �Properties� and check the �Run in separate memory space� checkbox

29. 29 �Separate Memory Space� Method (cont.) Outcome In MTS Explorer there is a package created for the Out Of Process Web application The naming convention for this package is IIS-{WebSite//Root/Application} Example: IIS-{WebCast//Root/OOP}

30. 30 �Separate Memory Space� Method (cont.) If you select the properties of the package you will notice that the Package Identity is set to IWAM_computer name. All anonymous access for OOP Web applications uses the IWAM_computer name account instead of the IUSR_computer name account.

31. 31 �Separate Memory Space� Method (cont.) Advantages Less Intrusive than the Removal method Applications behave normally Takes a subset of DLLs out of INETINFO.EXE and moves them into MTX.EXE Disadvantages Uses more memory per OOP application Slight performance hit since we need to �talk� cross-process May need to grant the IWAM_computer name account the same rights as the IUSR_computer name account

32. 32 MTS Package Method Steps Right click on �Packages Installed� and select New ? Package

33. 33 Package Wizard In the �Package Wizard� dialog, select �Create an Empty Package�

34. 34 Create Empty Package In the �Create Empty Package� dialog, enter in a name for the package and press ? NEXT Example: WebCast Package

35. 35 Set Package Identity In the �Set Package Identity� dialog, select �This User� and type in the name of the user you wish to run this package. Example: MYDOMAIN\WebCastPkg

36. 36 Set Package Identity (cont.) Type in the password and confirm the password Then press ? FINISH

37. 37 Add the Component to the Package Steps Under the Package, right click on �Components� and select New ? Component

38. 38 Component Wizard In the �Component Wizard� dialog, select �Import component(s) that are already registered�

39. 39 Choose Components To Import In the �Choose Components To Import� dialog, select the component that you would like added to the package Example: ABVB.Test Select FINISH when done

40. 40 MTS Package Method (cont.) Outcome The ABVB.Test component is added to the WebCast Package package

41. 41 MTS Package Method (cont.) Any calls to create the ABVB.Test object will instantiate that component inside an MTX.EXE process and not INETINFO.EXE. If the ABVB.Test component becomes unstable then the process is torn down and a new instance is created when the next request is made.

42. 42 Putting It All Together Scenario: ASP returns ASP 0115 in the browser Steps to Troubleshoot Examine event log for ASP error.Look for ASP File and line number. Mark the application where the ASP file is OOP. Next time you encounter the error check the event log. You should see an MTS event log entry for the �object causes an exception.� Examine the log entry for the stack trace and identify the culprit DLL. Remove/fix culprit DLL.

43. 43 Putting it all together (cont.) Scenario: ASP pages hang Steps to Troubleshoot Try a test �Hello World� ASP page. Check to see if HTM pages are serving. Run performance monitor and examine the ASP ? Requests Queued counter (look for a high number). Start running suspect sites and applications OOP. Once you determine which OOP Web site or application is hanging, download the IIS Exception Monitor from http://msdn.microsoft.com/workshop/server/iis/ixcptmon.asp and run it against the OOP Web site or application. Examine the log yourself or send the log to Microsoft Technical Support for analysis.

44. 44 References Troubleshooting with the IIS Exception Monitor http://msdn.microsoft.com/workshop/server/iis/ixcptmon.asp Analyzing Logs from IIS Exception Monitor http://msdn.microsoft.com/workshop/server/iis/readlogs.asp Using Performance Monitor to Detect Memory Leaks in Microsoft Windows NT� Support WebCast by Ben Christenbury Microsoft Windows�NT and Windows�2000 Debugging Resources http://www.microsoft.com/hwdev/driver/ntdebugging.htm

45. 45