Leveraging WebSphere Application Server v6.0.2 Integrated Tools to Solve Production Application Problems

1. WebSphere Software Serviceability - 2005 Leveraging WebSphere Application Server v6.0.2 Integrated Tools to Solve Production Application Problems 

2. WebSphere Software Serviceability Blueprints - 2004 Introduction WebSphere Serviceability Development Context WebSphere Serviceability Yesterday WebSphere Serviceability Today WebSphere Serviceability Tomorrow

3. WebSphere Software Serviceability Blueprints - 2004 Introduction Problem Discovery/Resolution Trouble Shooting Basics – Getting Started Connection Timeout – Light-weight Trace Thread Hang – WebSphere Application Server Thread Monitors Class Loader – WebSphere Application Server Class Loader Viewer Memory Leak – Memory Management Diagnostics for Java Further Information Questions?

4. WebSphere Software Serviceability Blueprints - 2004 WebSphere Serviceability Development Context WebSphere development team as part of the support organization Uses their customer experience when designing new serviceability tools Works directly with development component teams or develops code changes directly Uses the results of problem trend analysis in their serviceability component design Work to publish and disseminate available solutions to customer and IBMers

5. WebSphere Software Serviceability Blueprints - 2004 WebSphere Serviceability Yesterday Always in firefight mode – little time spent building solutions or analyzing trends Adhoc problem determination – methodology of PD varies from individual to individual Knowledge bases were non-existent (RETAIN) Minimal serviceability integration with WebSphere Application Server

6. WebSphere Software Serviceability Blueprints - 2004 WebSphere Serviceability Today Focus: Reduce fire fighting and increase fire prevention Problem determination methodologies are maturing, but still not fully realized Tool set is expanding but not yet complete or fully integrated Knowledge bases are growing – there is a lot of information collected in the repositories Trend analysis is recognized as highly valuable - what are the classes of problems our customers see? Auto support – this is the goal

7. WebSphere Software Serviceability Blueprints - 2004 WebSphere Serviceability Tomorrow PMRs are trending downward since 2004 with fewer crit sits IBM Education Assistant/WebSphere Support Page An auto support solution Manual process assisted by autonomic processes Service processes defined yet flexible Expert system (TROVE) Increased customer self-help tools Suite of serviceability tools and services realized Less time troubleshooting, more time innovating

8. WebSphere Software Serviceability Blueprints - 2004 Problem Discovery/Resolution Trouble Shooting Basics – Getting Started Connection Timeout Thread Hang Class Loader Memory Leak

9. WebSphere Software Serviceability Blueprints - 2004 Talk about the Problem – Listen to the Answer Many times, a problem can be resolved just through good communication Describe the problem in a clear and concise manner. Never assume they understand the details or even the basic issue at hand Listen carefully to what is really being said. Never assume you know what the problem is. Ask questions and keep an open mind Trouble Shooting Basics – Getting Started

10. WebSphere Software Serviceability Blueprints - 2004 Trouble Shooting Basics – Getting Started The Collector Tool Prior to starting any trouble shooting session it is highly advisable to run the collector tool Running the collector tool answers important questions about the hardware and software environment Think of it as analogous to a doctor taking an x-ray Run the collector on the deployment manager and at least one node to get a complete picture of the environment

11. WebSphere Software Serviceability Blueprints - 2004 Connection Timeout Diagnosis A light-weight tracer can be activated to provide additional diagnostic information (*=info:ConnLeakLogic=all) The tracer provides a stack trace indicating the code path holding the application Adds between 1% and 5% overhead to the running system Only outputs data when a problem has occurred, providing instant recognition of the problem Less overhead than standard WebSphere tracing

12. WebSphere Software Serviceability Blueprints - 2004 Connection Timeout Detection Logs will contain ConnectionWaitTimeoutException and the connection pool might become exhausted Raising the number of connections might remedy the situation or just defer the problem if there is a leak How do we determine we have a connection leak? Connection Manager Diagnostics for J2C JDBC Data sources Activated via a trace setting in the WAS admin console

13. WebSphere Software Serviceability Blueprints - 2004 Connection Timeout Resolution Once it has been determined that there is a resource leak in the connections, a code change could be required to resolve the issue A good technique for managing resource in Java is through the se of the try {…} catch {…} finally {…} construct. All resources can be closed in the finally block, assuring they get closed regardless of the code path followed Remember – WebSphere does a good job of managing connections. Don’t be afraid to open and close them as needed

14. WebSphere Software Serviceability Blueprints - 2004 Thread Hang Detection Over time threads might become locked, leading to a system hang Prior to the system being inoperable, threads will start to become exhausted as they are taken from the pool but never returned A thread hang problem could occur because the thread is waiting on other resources and there is an improper timeout or recovery condition in the threading logic

15. WebSphere Software Serviceability Blueprints - 2004 Thread Hand Diagnosis A new Thread Monitor architecture was created to monitor thread pools within WebSphere WebSphere 6.x monitored pools include the Web Container, ORB, and Async Beans thread pools An alarm can be configured for a thread’s life. If the alarm expires before the thread, then it is considered hung The overhead for using this monitoring is negligible (< 1%) Hang notification is sent in 3 different ways StandardOut.log as a RAS message As a JMX event As a PMI event

16. WebSphere Software Serviceability Blueprints - 2004 Thread Hang Resolution Thread hangs could be code errors where deadlocks or busy work is introduced Could also be as a result of a WebSphere thread waiting for another system to complete, but that system never does or does so slowly Resolving these hangs are non-trivial and could require code changes to systems you own or might involve other groups if their system is mis-behaved

17. WebSphere Software Serviceability Blueprints - 2004 Class Loader Detection There are multiple levels of class loaders used to load J2EE applications within WebSphere Detecting a class loader issue can be tricky, to say the least Typically, unexpected behavior will occur in the application, such as new code changes not being picked up or older versions of libraries/jars being used This behavior signals that the wrong class loader is in use, resulting in undesired code being run

18. WebSphere Software Serviceability Blueprints - 2004 Class Loader Diagnosis The Class Loader Viewer provides a graphical display of which classes are being loaded by which class loader The Class Loader Viewer provides search/query capabilities for searching on specific classes A graphical tree display of the class loader hierarchy assists in understand the class loader hierarchy

19. WebSphere Software Serviceability Blueprints - 2004 Class Loader Resolution Resolving a class loader issue relies on an understanding of how the class loader paradigm works in WebSphere v6 The runtime environment of WebSphere Application Server uses the following class loaders to find and load new classes for an application in the following order: (source: Info Center) The bootstrap, extensions, and CLASSPATH class loaders created by the Java virtual machine A WebSphere extensions class loader One or more application module class loaders that load elements of enterprise applications running in the server Zero or more Web module class loaders

20. WebSphere Software Serviceability Blueprints - 2004 Class Loader Resolution There are 2 configuration parameters that directly control class loading in WebSphere Mode: Parent First of Parent Last. The default is Parent First Policy: Module or Application. The default is Module Each web module has its own WAR class loader to load the contents of the WEB-INF/classes and the WEB-INF/lib directories To share resources amongst various web modules, set the policy to Application To have a web module use its WAR class loader set the policy to Module and the mode to Parent Last

21. WebSphere Software Serviceability Blueprints - 2004 Memory Leak Detection Even with garbage collection, memories leaks can still occur within Java Analyzing a memory leak within a runtime environment is non-trivial A memory leak in Java is typically introduced through a coding error, resulting in unused objects being referenced, thereby making them ineligible for garbage collection Really, when we talk about a memory leak in Java we are talking about a resource leak

22. WebSphere Software Serviceability Blueprints - 2004 Memory Leak Diagnosis By using MDD4J, a memory leak will show up as a steady increase in heap consumption with garbage collection reclaiming some of the memory but never all of it The graphical representation of a well behaved JVM heap is a saw tooth pattern where all allocated memory is eventually returned A memory leak will be evident in the graph because over time, garbage collection will not reclaim all of the memory. The valleys of the saw tooth will become smaller over time

23. WebSphere Software Serviceability Blueprints - 2004 Memory Leak Resolution We can resolve the memory leak by using the Memory Dump Diagnostics for Java (MDD4J) tool against 2 memory dumps MDD4J will pin point the objects that are being allocated but never successfully dereferenced Once isolated, the offending area of code will need to be re-worked to eliminate the leak

24. WebSphere Software Serviceability Blueprints - 2004 Further Information WebSphere Application Server Info Center http://www-306.ibm.com/software/webservers/appserv/was/library/ The Collector Tool for z/OS This can be obtained from IBM Support MDD4J This can be obtained from IBM Support

25. WebSphere Software Serviceability Blueprints - 2004 Questions?