Understanding IBM Maximo Architecture - Configuration and Performance

1. Understanding IBM Maximo Architecture - Configuration and Performance

2. Overview • J2EE/Java Application Architecture • The TPAE Architecture • Maximo Reliance Pyramid • Network • LAN / WAN • Resolving Issues • Hardware • Minimum Requirements • Common Problems • Soft and Hard Partitioning • Services • J2EE Services • Database Services • Report Services • Maximo • Fine Tuning • Questions

3. Considerations for this presentation Because of the approach of this presentation, some technologies will be covered from different perspectives (Example Network tuning and Maximo tuning for the network) Every environment is different and exact tuning solutions require and analysis of a specific environment

4. J2EE Application Architecture Java 2 Platform Enterprise Edition Application Architecture

5. Quick overview of Java Technologies Java code is written in a text style editor in semi English terms. Source text files (.java) are compiled using a java compiler (javac) to become class files (.class) Class files are really a collection of tokens that execute in a Java Virtual Machine (JVM) JVMs are written for each computer environment (OS, Processor). Since it is the JVM that is running on the computer equipment, Java code does not have to be recompiled for different computers. The same code runs in different JVMs

6. Quick overview of Application Servers Java code can execute in a JVM without the use of an Application Server Application Servers were designed to simplify JVM management and provide additional functionality Application Servers provide deployment, logging, memory management and other services Additional functionality provided by most Application Servers includes JDBC pooling, JMS Queues, and Load Balancing

7. Technology Relationships of Java Programs The Application Server starts => The JVM starts => The Java code starts So Maximo relies on the JVM to run which relies on the Application Server to run which relies on the Operating System to run Java Code (Maximo and others) Contains the application logic Java Virtual Machine (IBM, Sun, and others) Executes the Java Code Application Server (IBM WebSphere or Oracle WebLogic) Manages the JVMs

8. External Technology Relationships Java code can have reliance on external technologies in addition to the Java relationships. In Maximo’s case, it relies on external database technologies and may rely on external report technologies. Maximo does not use Application Server JDBC pooling Java Code Contains the application logic Java Virtual Machine Executes the Java Code Application Server (IBM WebSphere or Oracle WebLogic) Manages the JVMs External Report Technologies (Actuate) Manages the reporting for the application External DB Technologies (DB2, Oracle, SS) Manages the data from the application

9. Resolving J2EE configuration issues Each layer of technology relies on the layer it resides in J2EE Layers can be reviewed separate from external technologies External technologies such as database and report can be reviewed separate from J2EE Both J2EE and external technologies may have common reliance such as network

10. The TPAE Architecture Tivoli Process Automation Engine (TPAE) Architecture

11. What is TPAE? • The Tivoli Process Automation Engine provides basic services to applications that are built on it’s architecture • Functionality such as database connectivity, email listeners, cron tasks, workflow, security and much more is built into TPAE and provided to applications • Applications that use TPAE have a single point of troubleshooting for core issues

12. Tips for finding performance bottlenecks The IBM Maximo Reliance Pyramid

13. The Maximo Reliance Pyramid

14. The Approach Used For Troubleshooting • Reliance on other technologies complicates • To simplify troubleshooting, start with the least reliant technology and build towards the most reliant • Using the bottom up approach to the reliance pyramid, troubleshooting starts with network and progresses upward. • Maximo • Services • Hardware • Network

15. Networks – The Ground Rules Maximo uses about 32 kbps in bandwidth per concurrent user Maximo performs best in 50 ms or less latency It takes thousands of packets to display a single User Interface (UI) screen 1000 * 50 ms = 50,000 ms or 50 seconds

16. Networks – Bandwidth Bandwidth is the amount of data that can be transmitted simultaneously Using a 56 kbps connection to transmit 512 kbps of data will…. fill up your “tubes” Maximo uses approximately 32 kbps per user. Multiply 32 * number of concurrent users to determine required available bandwidth

17. Networks – Latency Latency is the amount of time in milliseconds that it takes for a packet to reach its destination and a receipt to be sent back to the sender When too much data is sent through a connection, it is put in a queue and transmitted as bandwidth becomes available Queued data takes longer to transmit because of the wait time and results in higher latency Since latency is a key factor in Maximo performance, proper bandwidth must be allocated in order to ensure maximum performance

18. Maximo On The Network Maximo Application Server Report Server Database Server

19. Networks – Wide Area Networks (WAN)

20. Networks – Quick Answers Two simple commands to help determine performance ping tracert Senator Ted Stevens explains network communication best

21. Networks – Objects vs. Packets Two objects can be related to packets: Object 1 = 51 bytes and can be placed in 1 packet Object 2 = 2,330 bytes and can be placed in 2 packets How does this translate into performance? It takes 100 copies of object 1 to make up 1 copy of object 2 Object 1 takes 98 more packets to transfer and cover the same area as object 2 At 50 ms latency, object 1 will fill the area in 5 seconds while object 2 will only take 1/10 of a second

22. Networks – Maximo vs. Web Pages Images and objects (Application Designer) Web Pages use larger images for static content Maximo breaks images into smaller objects Validation (Ajax type technology) Web pages typically validate at submit time Maximo validates and secures at field level during input User Interface and Data Manipulation (ad hoc search) Web pages typically focus on a specific job Maximo provides a high level of customization capability and user flexibility to access and manipulate large quantities of data

23. Networks – Optimization Technologies Citrix / Terminal Server (Client Emulation) Junniper, Expand, & similar – (Single Device Acceleration / Caching, HTTP Header Compression) Riverbed & similar – (Multi-device Acceleration / Caching, Compression, HTTP Headers)

24. The Maximo Reliance Pyramid - Networks You are here When network tuning is complete, the next step is Hardware

25. Hardware is measurable & relatively easy to define Maximo requires a minimum of 900 MHz processors, enough memory to support the Java JVM, and network connectivity (a NIC card) If these minimums are not met, Maximo may still run but is not supportable Check documentation for specific release requirements Hardware

26. Hardware - Operating System Relationships Hardware is the physical equipment (processor, memory, storage devices, monitors) that software runs on Operating systems manage the hardware and how it is used by software programs All calls and uses of hardware are processed by the operating system If the operating system runs poorly then the software running in it will run poorly

27. Hardware – Common Windows OS Problems • Mapped drives requiring network access for logs & attached documents • When building/deploying EAR files, remote source files may cause the build/deploy process to perform poorly • Operating System swap space and memory management • Disk space / quotas • Configuration of the Network Interface Card (NIC) • Antivirus software on the server set too aggressively • Windows defaults to a more open security architecture than Unix but can be locked down

28. Hardware – Common Unix OS Problems • The #1 OS problem experienced in Unix is privileges! • Unix defaults to a more secure security architecture than Windows does but privileges can be granted. If access to required files is denied, Maximo may not run properly • Mounted drives requiring network access for logs and attached documents • Operating System temp space and memory management • Unix Configuring X-Windows, XVFB, and the display variable on the server • Disk space / quotas for logs & attached documents • Unix controls upper limits on all types of connections such as DB connections, open files, logins etc… These can prevent proper function. (set “ulimit” for open files and memory)

29. Hardware – Partitioning • Physical servers can be divided into physical partitions. Example: 4 processors, 4 GB memory, can be 4 separate hard partitions acting as 4 separate computers, each one independent (IBM LPAR, Sun Domain, etc) • Physical partitioning usually applies to large Unix based systems • Soft partitions such as VM Ware and Microsoft VM are more PC oriented. VM Ware is not supported for production environments until release 7.1.1.4

30. The Maximo Reliance Pyramid - Hardware You are here When hardware tuning is complete, the next step is Services

31. Services – How they affect Maximo • Maximo is a software product written in Java that pulls together many types of technology to empower users • If a service is not performing well, that functionality of Maximo will not perform well • Tuning services is heavily reliant on recommendations from the manufacturer of that product • Services include J2EE, JVM, Database, Reporting and others that Maximo relies on

32. Services – How they affect Maximo • If database queries run slowly then Maximo performance will degrade due to waiting for database response • If the J2EE server (IBM WebSphere or BEA WebLogic) does not perform well, Maximo running inside the J2EE server cannot perform well • If the JRE / JVM that the J2EE server uses does not perform well, Maximo cannot perform well • If the report server does not perform well, reports may be slow and other areas of Maximo may be affected

33. Services – Database - How they affect Maximo • DB2 has a cost driven optimizer , executing the RUNSTATS utility will update the catalog and improve performance • Maximo does not use bind variables. Oracle will perform better if cursor_sharing is set to similar or force • In Oracle, more SGA is better, Oracle for Windows may have memory limitations • SQL Server uses memory page level locking by default. High concurrent user counts can cause database locks as a result. SQL Server will perform better if row level locking is enabled • Any database platform will perform better if indexing has been tuned for your particular use

34. Services – J2EE Services - How they affect Maximo • If the J2EE server (IBM WebSphere or BEA WebLogic) or the JVM supporting Maximo does not perform well, Maximo cannot perform well • WebSphere should have cache settings and TCP/IP settings tuned to perform well • WebLogic should have threads tuned to perform well • HTTP servers and JVMs be deployed in sufficient numbers to support incoming requests and should be load balanced • Documents 1261874 & 1261853 WebSphere Tuning - http://www-1.ibm.com/support/docview.wss?rs=3214&uid=swg21261874&loc=en_US WebLogic Tuning - http://www-1.ibm.com/support/docview.wss?rs=3214&uid=swg21261853&loc=en_US

35. The Maximo Reliance Pyramid - Services You are here When service tuning is complete, the next step is Maximo

36. Maximo Tuning – Common Performance Problems • Network Issues described earlier • Too many users on too few JVMs • Minimal training for end users causing improper use • Configuration choices at implementation time • Memory leaks • Databases not properly tuned • Log Overkill

37. Maximo Tuning – Network Properties • Maximo 6.1 built in compression – (Software Compression / Gzip) • Maximo 6.1 built in caching – (Software Caching / max-age) • Document numbers 1262009 & 1292557 http://www-1.ibm.com/support/docview.wss?rs=3214&context=SSLKT6&uid=swg21262009&loc=en_US http://www-1.ibm.com/support/docview.wss?rs=3214&context=SSLKT6&uid=swg21292557&loc=en_US

38. Maximo Tuning – Concurrent Users • Maximo should be clustered and load balanced to maintain less than 50 concurrent users per instance • 50 users is a guideline not a hard number. 50 users logged in and not doing anything is not the same as 50 users run in a Load Runner script hammering the system • Performance testing scripts should include wait times, different users, and different tasks to ensure a more real world experience • Be prepared to add more instances if performance is not as targeted. Document 1329219 http://www-01.ibm.com/support/docview.wss?rs=3214&context=SSLKT6&uid=swg21329219&loc=en_US

39. Maximo Tuning – Minimal Training • Maximo is very flexible and often has more than one way to accomplish a task • Untrained users often choose an approach that may be less than optimal • Common tasks should be reviewed by knowledgeable staff to fine tune processes • Maximo searching is very powerful but if misused, can impact performance for all users

40. Maximo Tuning – Configuration Choices • Search functionality defaults to the most flexible searching method but burdens the database server. Search methodologies include: • Wildcard (default) • Exact • Text • None • User loads, type of work and user training may dictate changing the default search methodology • A common implementation approach is to create catch all screens for users. More smaller screens can improve performance over larger crowded screens

41. Maximo Tuning – Memory Leaks • All programs have bugs • Some memory leaks may be related to invalid program data (example parent/child locations set as the same value causing a loop) • Some bugs may be related to custom coding • Memory and MBO’s will continue to grow and not release if there is a memory leak – Documents 1291250 & 1326774 • 07 Apr 2008 11:14:22:622 [INFO] WORKORDER: mbosets (272), mbos (306) • 07 Apr 2008 11:15:22:651 [INFO] WORKORDER: mbosets (278), mbos (312) • 07 Apr 2008 11:16:22:690 [INFO] WORKORDER: mbosets (380), mbos (432) • 07 Apr 2008 11:17:24:708 [INFO] WORKORDER: mbosets (435), mbos (489) • 07 Apr 2008 11:18:24:928 [INFO] WORKORDER: mbosets (686), mbos (812) • 07 Apr 2008 11:19:25:101 [INFO] WORKORDER: mbosets (847), mbos (939) • 07 Apr 2008 11:20:26:399 [INFO] WORKORDER: mbosets (942), mbos (1028) http://www-1.ibm.com/support/docview.wss?rs=3214&uid=swg21291250&loc=en_US http://www-1.ibm.com/support/docview.wss?rs=3214&uid=swg21326774&loc=en_US

42. Maximo Tuning – Databases • Maximo is shipped for generic use. A specific implementation requires specific index and database tuning • Tuning the database will involve a good understanding of the database environment as well as the ways Maximo is used in a given environment • If Maximo is more focused on purchasing, tuning work order tables and indexes will probably not help • Use Maximo logging properties to identify problems • Document 1291250 http://www-1.ibm.com/support/docview.wss?rs=3214&uid=swg21291250

43. Maximo Tuning – Database (SQL Server 2005) • SQL Server 2005 tuning parameters • mxe.db.disableservercursor=Y • mxe.db.resultsettype=TYPE_FAST_FORWARD • mxe.db.fetchsizeuse=Y • mxe.db.fetchsize=40 • mxe.db.optionuse = Y • mxe.db.optionnum=1000 • mxe.db.sqlserverPrefetchRows=200 • Note: For Maximo releases > 6.1& SQL Server 2000, these values are different • Documents 1268567, 1296072, 1313428 http://www-1.ibm.com/support/docview.wss?rs=3214&uid=swg21268567 http://www-1.ibm.com/support/docview.wss?rs=3214&uid=swg21296072 http://www-1.ibm.com/support/docview.wss?rs=3214&uid=swg21313428

44. Maximo Tuning – < 6.2 Database (SQL Server 2000) • Use SQL Server tuning parameters to manage cursor usage • mxe.db.disableservercursor=Y • mxe.db.resultsettype=TYPE_SCROLL_INSENSITIVE • mxe.db.fetchsizeuse=Y • Note: For Maximo releases > 6.1& SQL Server 2000, these values are different • Documents 1268567,1295983, 1313428 http://www-1.ibm.com/support/docview.wss?rs=3214&uid=swg21268567 http://www-1.ibm.com/support/docview.wss?rs=3214&uid=swg21295983 http://www-1.ibm.com/support/docview.wss?rs=3214&uid=swg21313428

45. Maximo Tuning – > 6.1 Database (SQL Server 2000) • Use SQL Server tuning parameters to manage cursor usage • mxe.db.disableservercursor=Y • mxe.db.resultsettype=TYPE_SCROLL_INSENSITIVE • mxe.db.fetchsizeuse=Y • mxe.db.fetchsize=40 • mxe.db.rowcount=15000 • mxe.db.optionuse=N • Note: For Maximo releases < 6.2& SQL Server 2000, these values are different • Documents 1268567,1295983, 1313428 http://www-1.ibm.com/support/docview.wss?rs=3214&uid=swg21268567 http://www-1.ibm.com/support/docview.wss?rs=3214&uid=swg21295983 http://www-1.ibm.com/support/docview.wss?rs=3214&uid=swg21313428

46. Maximo Tuning – Oracle • Use Oracle logging properties to identify problems • Ensure that Oracle has enough SGA • Ensure that Oracle is set to: Cursor_Sharing=similar • Ensure that Oracle is upgraded to 9.2.0.8 or later • Ensure that Oracle is set to Cost Based optimization http://www-1.ibm.com/support/docview.wss?rs=3214&uid=swg21262180 http://www-1.ibm.com/support/docview.wss?rs=3214&uid=swg21262959

47. Maximo Tuning – Logging • Setting verbosity 9 or debug mode makes finding a problem in a log like a needle in a haystack • Setting verbosity 9 or debug mode causes performance problems, impacts disk space and may cause the log to roll before the issue can be captured • Minimizing logging can impact problem resolution times. Typical logging should be set at verbosity 3 and per document 1291250 for general runtime. • Use common sense when logging for a suspect issue • Debug Logging - http://www-1.ibm.com/support/docview.wss?rs=3214&uid=swg21291250&loc=en_US

48. Maximo – Tuning and Troubleshooting Use the Maximo debug parameters to ensure that SQL is performing well mxe.db.logSQLTimeLimit (5.2 P02A and above) mxe.db.logSQLPlan (6.0 P01 and above - Oracle) mxe.db.sqlTableScanExclude (6.0 P01 and above - Oracle) Use the debug parameters to ensure that memory is performing well mbocount (5.2 P05 and above - different implementations) mxe.db.fetchResultLogLimit (5.2 P02A and above) Document numbers 1291250 & 1326774 http://www-01.ibm.com/support/docview.wss?rs=3214&context=SSLKT6&uid=swg21291250&loc=en_US http://www-01.ibm.com/support/docview.wss?rs=3214&context=SSLKT6&uid=swg21326774&loc=en_US

49. This link is the key to finding good information on Maximo WebSphere Tuning - 1261874 Gzip / max-age – 1262009 Oracle performance - 1262959 & 1262180 max-age - 1292557 SQL Server - 1268567 SQL Server 2000 - 1295983 & 1313428 SQL Server 2005 - 1296072 Debug logging - 1291250 Type the document reference number into the search criteria Important Information to Take Away http://www-1.ibm.com/software/sysmgmt/products/support/IBMMaximoAssetManagement.html Or go to http://www.ibm.com and enter the following in the search field (include the quotes) "IBM Maximo Asset Management Support"

50. Questions