WSC Guidelines for a Healthy WebSphere Application Server Runtime on z/OS

1. WSC Guidelines for a Healthy WebSphere Application Server Runtime on z/OS Techdoc TD104172, May 2009 John Hutchinson, hutchjm@us.ibm.com IBM Americas Advanced Technical Support -- Washington Systems Center Gaithersburg, MD, USA

2. a Catalog of Information for WAS on z/OS… High-level guide to documents will help you Configure & Manage WAS on z/OS, based on Washington Systems Center (WSC) experiences TechdocTD104172 Click on Hyperlinks to access documents on the web at the following sites: • Techdocs at http://www.ibm.com/support/techdocs • WPxxxxxx(White Papers) • TDxxxxxxx (Hints, Tips, and Technotes) • PRSxxxx (Presentations and Downloads) • Redbooks (SG24-xxxx) at http://www.redbooks.ibm.com/ • DeveloperWorks athttp://www.ibm.com/developerworks/websphere/ • alphaWorksathttp://www.alphaworks.ibm.com/ Don’t forget: the primary information source for WAS: the Information Center: http://www.ibm.com/software/webservers/appserv/was/library/.

3. Healthy Topics A. Introduction & Education B. Installation Setup & Configuration C. Administration & Operations D. Availability & Recovery E. Security F. Application Deployment & Management G. Performance H. Problem Determination

4. SDK Real Stg A. Introduction Before you start… • Hardware Requirements: 2 Gb Real, several DASD volumes (min) • Software Requirements: • “Program Directory” or “Target System Requirements” in the InfoCenter. • Start with the latest version (6.1.0.24 or 7.0.0.03) if you’re setting up a new server. • Configuration options:Understand the functions & options available. • Use an ND (Network Deployment) configuration for availability, performance, and management qualities. • 1st time: Start with a standalone server to master the fundamentals, but federate it into an ND configuration for practical use. • Web Information Sources:Navigating the IBM Web in search of WebSphere Application Server for z/OS – TD102724 • Saving time with WebSphere Application Server documentation – PRS3546

5. Education Resources Best way to learn about WebSphere is a workshop with hands-on lab exercises, such as: • Wildfire Classes – PRS1778 • WebSphere for z/OS Version 7 for z/OS Workshop (WBSR7) PRS3422 • Security Workshop: WebSphere App. Server for z/OS (WSW07) PRS1438 • WebSphere Process Server V6.1 for z/OS Workshop (ZWPS6) PRS3421 • Enroll via your IBM Rep. who cancontact Judith Ramage (ramagej@us.ibm.com) or call 301.240.3966 • IBM Training courses • ‘Maximizing WebSphere Performance on z/OS’ (OZ850) • IBM Education Assistant(educational modules designed to help you gain a better understanding of WebSphere products) • http://publib.boulder.ibm.com/infocenter/ieduasst/v1r1m0/index.jsp

6. B. WAS Installation • Product code loaded and applied using SMP/E. • See the‘Program Directory’available in the InfoCenter. • See Preventive Service Planning (PSP) site for Service Recommendations and Cross Product Dependencies: https://techsupport.services.ibm.com/server/390.psp390 • WAS V6.1: upgrade ‘WASAS610’ and subset ‘H28W610’ • WAS V7.0: upgrade ‘WASAS700’ and subset ‘H28W700’ • The latest PTFs can also be found at http://www.ibm.com/software/webservers/appserv/zos_os390/support/ • click on ‘Fixes by version’ under the ‘Download’ section.

7. Set up z/OS Components for WAS Several components & services need to be configured for a robust z/OS environment before you can configure WAS: - plus the necessary skills to use them: • UNIX Systems Services (USS) with HFS & zFS. • TCP/IP, including DNS, Resolver, INETD, TN3270 • RRS (Resource Recovery Services), Logger • RACF (or equivalent) • TSO, ISPF, SDSF, & RMF • Optionally, DB2, CICS, MQ, IMS • System Programmer productivity tools (e.g., MXI from Rocket Software.) See “Preparing the base operating system”in the InfoCenter

8. Workstation Tools • 3270 Emulator (IBM Personal Communications) • Telnet client (PuTTY, TeraTerm) • FTP clients (WS_FTP LE) • Web browser (Internet Explorer, Firefox) • PDF Reader (Adobe) • Text File Editor (PFE , Notepad, or SlickEdit) • ASTK (Application Server Toolkit) & zPMT (z/OS Profile Management Tool) • RAD (Rational Application Developer) or WID Each of the hot links above take to you the tool’s download site or home page.

9. Configuring WAS – 1st Steps • For your first application server (if you didn’t attend a workshop), start with a standalone server: Using zPMT and Spreadsheet to build a Quick Standalone - WP100999 • Next build an ND (Network Deployment) cell, which is covered inA 'Top Down' Configuration Approach to WebSphere on Z -WP101030 • Based on more comprehensive white paper: WebSphere z/OS V6 - WSC Sample ND Configuration - WP100653

10. Important HFS Configuration Tips • The ‘smpe_HFS’ should be mounted read-only. • Configure separate HFS/zFS file systems for each node. • Must be owned by the same system as the node is running. • Mounted with “no Automove” attribute. • List in Parmlib BPXPRMxx member, so mounted correctly after IPL. • See SHARE presentation “Avoiding the Potholes on the WebSphere Application Server On Ramp” - PRS2853. • Contains many pointers along with supporting tools (REXX execs, sample JCL, spread-sheets, and examples.)

11. More Configuration Papers Planning & General Guidance: • Planning Test, Production and Maintenance - WP100396 • Problem Avoidance for WAS for z/OS (Ch. 1) -REDP6003 zPMT (z/OS Profile Management Tool): • zPMT Introduction - WP100871 • Using zPMT & Spreadsheet to Build Quick Standalone WP100999 • Introducing the WCT for z/OS - PRS3357 31-bit considerations: • SHRLIBRGNSIZE and Effect on 31-Bit JVM Storage Needs – WP101320 • Space in Pvt Reg. for shared objects may affect LE storage available for a JVM to acquire for the heap or other uses. 64-bit considerations: • 64-bit Addressing Support in WAS for z/OS V6.1 - WP100920 • C/C++ Considerations with 64-bit WAS for z/OS - WP101095 • 64-bit Effect: 5 different ways to look at Applications- WP101121

12. Introducing the WebSphere Configuration Tool (WCT) for z/OS -PRS3357 • Workstation based graphical tool • Create customized jobs to build & migrate a WebSphere Application Server for z/OS environment. • Replaces the ISPF Dialogues & AST. • Eclipse based tool – much “leaner” than the AST. • PMT - Profile Management Tool • Supports WebSphere V 6.1 and V 7 • MMT - Migration Management Tool • See WP101329 • Use with "WebSphere V7 Planning Spreadsheet” -PRS3341 3 Spreadsheets create input to the WCT: • Network Deployment Cell • Standalone Agent, Job Manager & Server • DMZ (Secure Proxy)

13. Hidden Gems: Little Known Features of WAS on z/OS - WP101138(includes a script to clean up common variables- message routing) • Sticking with the script – cleaning up common variable settings • To GMT or not to GMT - using ras_time_local • Keeping track when your applications don't • Putting trace output in its place • Managing your message output • What’s going on? The DISPLAY command • SERVERS and SERVANTS • LISTENERS and CONNECTIONS • TRACE and ERRLOG • JVMHEAP and MODE (31 or 64-bits) • WORK, CLINFO and MDBSTATS • Spreading the work around - stateful session placement • Please stand by - pause/resume listeners • Are your enclaves propagating? Should you care? • Handling large IIOP messages in 64-bit mode • Servant survivor - staying up during a timeout flurry • Throttle up - scaling up to a lot of connected clients

14. Hidden Gems 2: More great but Little Known Features in WebSphere z/OS -WP101464 • In Version 6.1: • Making controller ASIDs reusable • Allowing servants to connect to WLM early • SMF record changes for IBM Getting Started Sub-Capacity Pricing (GSSP) • In Version 7.0: • Starting multiple servants in parallel, and setting minimum and maximum numbers for servants. • Controlling the number of dispatch threads in a servant • Improved tools and diagnostics for troubleshooting. • Ability to dynamically update the dispatch timeout delay • Spinning server output by volume instead of by time • Improved WLM classification wildcarding

15. Other Configuration Options Sysplex Distributorto increase availability & workload balancing WP100312 HTTP Server for session affinity and routing. • Understanding the HTTP Web Server Plugin -PRS1467 • IBM HTTP Server for z/OS Powered by Apache – WP101170 • Extending IHS powered by Apache with Custom Modules–WP101225 • Introducing WebSphere V7 Secure Proxy Server–WP101423 Job Manager Introducing The WebSphere V7 Job Manager for z/OS • WP101341 Heterogeneous Cells spanning different operating system platforms • WP100644 CloudscapeConfiguring the Cloudscape Network Server for WAS for z/OS -TD102368 - Shows how to set up Cloudscapein a generic WebSphere Server.

16. Runtime Migration Migrate your Version 5 or 6.0.2 WAS to V.6.1 or V.7 with the automation scripts detailed in these papers: From V. 5 • Sample Migration from WAS V.5 to V.6 on z/OS - WP100559 To V. 6.1 • Migrating to WAS V.6.1 on z/OS - WP100771. To V. 7 • Migrating to WAS V.7 on z/OS - WP101329 (See Next foil.)

17. Migrating to WAS z/OS V.7 • No need to stop servers during migration in most cases. • Leave existing runtime running, shutting down only at the very end when the new V7 servers are to come up. • Break the main migration job into three smaller jobs • BBOWPRO*, BBOWPRE* and BBOWPOS*. • Advantages: Each job takes less time and less resource • Failure isolation more granular, easier recovery/cleanup steps • Not migrating applications when doing DMGR migration • With many apps the migration could take too long. • Decouples the runtime migration from application migration. • Techdoc:WP101329 • MMT (z/OS Migration management Tool) - Panels • Customized Jobs, Examples, Problems/Solutions, Planning

18. Configuring WAS with other Resource Managers Techdocs to help set up connections to resource managers: DB2 • Enabling WAS for z/OS to use DB2 Universal JDBC Driver - TD101663 CICS • Connecting CICS Transaction Server from WAS for z/OS V 6WP100607 • Using JMS & WAS to Interact with CICS - MQ/CICS BridgeWP100682 IMS • IMS Connectivity in an On Demand Environment - SG24-6794 • IMS Java Guide & Reference -SC18-7821 MQ • WebSphere for z/OS JMS and MDB IVP -WP100424

19. XD FPs Portal ITCAM WPS WebSphere Application Server Extensions to WAS Products configured on top of WAS: • XD (Extended Deployment)- PRS1856 • Mixed Workloads in WebSphere XD V6.0 on z/OS - SG247267 • Scaling for High Availability: XD & WAS for z/OS - REDP3968 • Feature Pack for Web Services - WP101084 • Feature Pack for EJB3 -WP101142 • WAS V7 Feature Pack for SCA -WP101394 • WebSphere Portal Server - TD104289, WP101249, SG24-7459 • ITCAM V6.1 on WebSphere for z/OS - WP101206, SG24-7151 • WPS (see next foil)

20. XD FP WPS Portal ITCAM WebSphere Application Server More product extensions to WAS Process Server – the Flagship of SOA… • WPS (WebSphere Process Server) & WESB (WebSphere ESB): • WPS Wildfire Workshop Material: PRS3421 • WPS ND Deployment: WP101209 • WPS "Easy" ND Deployment: WP101253 • Performing Installation Verification for WPS on z/OS V6.1: WP101218 • WPS & WESB V.6.0 for z/OS Configuration Tools & Samples: PRS2520 • z/OS Getting Started: WPS & WESB V6 SG24-7378 • z/OS: WebSphere Business Process Management • V6.1.2 Production Topologies” RedBook SG24-7703 • V6.2 Production Topologies” RedBook SG24-7733

21. C. Operations & System Administration Day-to-day operation of WebSphere can be administered through variousOperator Interfaces: • WebSphere administrative console (using a web browser) • wsadmin scripts or command-line tools in a USS environment, • MVS Console (usually under TSO and SDSF), • Ant tools, or JMX programs. • See InfoCenter article Where to perform WebSphere Operations

22. Operations: MVS (TSO / SDSF) Consoles • Managing Operator Message Routing - TD103695 • SDSF system displays and MVS commands - TD100589. • Managing LE Options in WebSphere Servers - TD103863 • Creating Dynamic 3270 Screen Size in PCOM - TD102151to increase the number of rows & columns on your 3270 displays:

23. Administration Use the wsadmin tool to automate Configuration Tasks: • WSADMIN Primer (with Jython) - WP101014 • Includes several sample hands-on exercises. • Replaces WSADMIN Primer (with JACL) - WP100421 • Using Jython Scripting Language with WSADMIN - WP100963 • Includes several sample scripts to do common administrative tasks. • Creating new Application Servers in WAS V6 - TD104066 • Includes sample jython script to automatically correct the port assignments to the WSC standard numbering scheme. • Plus many InfoCenter articles…

24. D. Availability • High Availability requires: • Planning • Robust Parallel Sysplex • Reliable parts • Fully tested in a QA server • Redundant components • Network Deployment (ND) • Clusters on Multiple LPARs • Frequent back-ups (& reliable restore procedures) • Techdocs: • Architecting High Availability Using WAS on z/OS - SG24-6850 • Planning Test, Production and Maintenance - WP100396 • Handling Application Dispatch Timeouts - WP101233 • WAS V7 - Dispatch Timeout Improvements - WP101374

25. & Recovery . . . • Rapid Recovery involves many aspects: • Quick diagnosis (automation alerts) • Fix/Restore & Restart damaged parts • “Disaster Recovery” sites • Fully tested Fail-over scenarios • Planning for Disaster Recovery - PRS1137 • Starting DMGR on Another MVS Image - WP100585 • Moving WebSphere Nodes Between MVS Images - PRS1536, and WP100542. • Changing the host names and system names - WP100792

26. E. Security • Global Security enabled by default when you configure WebSphere for z/OS V6.1. • Recommend you use LocalOS to simplify the security setup. • ISPF dialogs & zPMT produce RACF commands to define protection for your WebSphere environment. • WebSphere for z/OS Security class handouts (WSW07) - PRS3422 • Configuring Fine-Grained Security - TD103324 • RACF tips & tools for WebSphere Application Servers - TD101115 • Using SERVAUTH to Protect TCP Port Usage - WP100673 • Enhanced form-based authentication - TD101255 • Sample J2EE App using Form Based Authentication & Style Sheet - PRS3417 • Generic RACF Profiles with WebSphere on z/OS V7 – WP10427

27. Security Techdocs, cont’d SSL, Certificates & Crypto: • Introduction to SSL for z/OS users - PRS1580 • Importing RACF Certificates in WAS 6.1 Distributed - PRS2854 • Using Multiple Certificates with WebSphere for z/OS - TD104044 • Activating z890/z990 Cryptographic Services for WAS - WP100386 • Enabling Web Services H/W Encryption with WAS - WP100808 • Resolving the iKeyman Corrupted Database Message - PRS2855 • SSL Options in WebSphere for z/OS V6.1 – WP101213 • Renewing expiring RACF certificates WAS on z/OS – PRS3584

28. More Security Documents Security-related RedBooks: • WAS for z/OS and Security Integration - REDP-4161 • J2C Security on z/OS - REDP-4202 • Java Messaging Service Security on z/OS - REDP-4203 • z/OS WebSphere and J2EE Security Handbook - SG24-6846 • Java Security and z/OS – The Complete View – SG24-7610

29. F. Application Deployment Sample Applications to help you practice and learn the deployment processes: • SuperSnoop Servlet -TD101815 • PolicyIVP J2EE Application for WAS using DB2 -TD101348 • WebSphere for z/OS JMS and MDB IVP -WP100424 Deployment Tips: • 64-bit Addressing Support in WebSphere for z/OS - WP100920 • Temp Space Shortage Installing Large Apps - TD102662 • Java & System z - DeveloperWorks White Papers - WP101291

30. Application Tuning, Management & Debugging • Moving Applications to WebSphere on z/OS - WP101093 • Enabling the WSAD Distributed Debugger - TD101198 • Enabling the WSAD Application Profiler - TD101199 • Log4j Enhancements for J2EE Applications - TD102335 • JinsightLive for IBM System z–Available on AlphaWorks

31. G. Performance Many Configuration options, Topologies, & Tuning Parms: • Hardware resources and configuration. • Software Levels: z/OS, WAS, & Java (latest releases are best) • Placement, size, & # of servers, plus threading considerations. • Environmental properties and WebSphere variables • Using the IBM HTTP servers • Workload classification • Security settings, and SSL options • SMF, RMF, Performance monitoring & workload simulation • Java tuning, JVM heap size, and garbage collection. • Tracing and other performance trouble-shooting tools. • Application design considerations. • TCP/IP, DB2, CICS, IMS, MQ subsystems.

32. Performance Tuning Papers • Performance Engineering & Tuning for WAS on z/OS -PRS2494 • Performance and tuning tips for WAS on z/OS - TD103036 • Optimizing WebSphere for z/OS Performance - WP100558 • Application Debugging and Profiling - WP100250 • Diagnosing Performance Problems on WAS for z/OS - WP100678 • Large Memory Performance Studies – WP101157 • Classify WAS Control Region in WLM OMVS rules - TD102730 • Finding CPU Usage in your applications - TD102454 • SMF Servlet Filter for WebSphere V6.1 on z/OS - WP101301 • Summarizing SMF 120 records - PRS752 • SMF 120 browser for WAS V7 WP101342 • Threads and excessive CPU Consumption in WebSphere on z/OS - WP101474

33. WLM Advanced Topics for WAS on z/OS - PRS3317 • WLM & WAS Interaction & Dependencies • Control Region / Servant Region Structure • Managing Threads, Servants, Servers & Clusters • Server Failover & Continuous System Operation • Work Classification – Transaction Classes & WLM Rules • Tuning Recommendations & Best Practices • WLM Settings: Service Classes, Report Classes & Resource Groups • SMF Settings and RMF Tools • Monitoring the Health of WebSphere & WLM • RMF Workload Activity Reports & Monitor III • SDSF & MVS Operator Commands for WebSphere & WLM • WLM Commands, Monitors & Tools • Misc. Topics • Capping & Controlling WebSphere Workload License Charges • WLM/WAS Start-up & Routing Options

34. New SMF 120 Records • WebSphere for z/OS creates SMF 120 records. • Issues with prior versions of WebSphere for z/OS: • Insufficient user/request information for Chargeback • Not extendable, Not dynamically controlled • Costly to record • WebSphere Version 7 introduces new subtype-9 • Dynamically enabled/disabled • More information for chargeback • Bytes transferred, Elapsed Times, CP, zAAP, zIIP times • Low overhead • Extendable with User inserted sections • “Overview of the WAS SMF Record 120-9”– WP101342 • WAS V7 SMF Summary Browser available from: • https://www14.software.ibm.com/webapp/iwm/web/preLogin.do?source=zosos390 (Select “SMF Browser for WAS for z/OS V5 and V6”)

35. Performance & Capacity Planning Education Available: • IBM Training course ‘Maximizing WebSphere Performance on z/OS’ (OZ850). Capacity Planning • Contact your IBM Rep to use the zPSG, zSeries processor sizing for new applications. For zAAP capacity projections, see • Obtaining the zAAP Usage Estimation Information - WP100431 • zAAP Estimation w/ Java5 & WAS for z/OS V6.1 - TD103460

36. H. Problem Determination Good “PD” isn’t an exact science; • It’s based on a thorough understanding of WebSphere & z/OS, experiences (usually unsuccessful), pattern recognition, contacts & ability to use many tools. • Your best “tools” are your brain, ability to search many information sources, and communication with other subject-matter “experts.” Tools: • Browsers, Editors, FTP clients, telnet clients, • ISPF, SDSF, MXI, Commands, SPUFI, Skills: • UNIX, vi, shell scripting, jython, jacl, • MVS, TSO, ISPF, SDSF

37. PD Documentation InfoCenter: Troubleshooting and support • Overview and new features • How do I?... Troubleshooting • Debugging applications • Adding logging and tracing to your application • Diagnosing problems (using diagnosis tools) • Accessing the Support site Redbooks & Redpapers: • Problem Determination for WebSphere for z/OSSG24-6880 • Problem Determination Methodology for WebSphereREDP6001 • Problem Symptoms in WebSphere for z/OS REDP6002 • Problem Avoidance for WebSphere App. Server REDP6003 • WAS for z/OS Problem Determination Means & ToolsREDP6880

38. Summary • There is a great deal of information available. • Trick is to find the right document quickly! • Become familiar with the information sources. • InfoCenter • Techdocs • Take the time to READ! • Experiment with your own system. • Know whom to call ;-) • Turn this: into This:

39. Questions ? Suggestions for future Techdocs? • Send an e-mail to: hutchjm@us.ibm.com or dbagwell@us.ibm.com