PeopleTools Application Server BEA Tuxedo Internals

1. PeopleTools Application Server & BEA Tuxedo Internals david.kurtz@go-faster.co.uk Go-Faster Consultancy Ltd. http://www.go-faster.co.uk

2. 23 January 2002 2 Who am I? Independent Consultant Performance Tuning PeopleSoft / Oracle

3. 23 January 2002 3 Rules of Engagement Interrupt me: if you think I have got something wrong if you have a question if you can�t hear me I put this slide in to make sure I say thisI put this slide in to make sure I say this

4. 23 January 2002 4 BEA Tuxedo Historical Overview Tuxedo Internal Architecture PeopleSoft 3-tier client implementation How does PeopleSoft Internet Architecture (PIA) fit in Configuration issues Monitoring

5. 23 January 2002 5 Brief Historical Background In the beginning monolithic IBM MainframeIBM Mainframe

6. 23 January 2002 6 Client Server Two-tier client-server

7. 23 January 2002 7 3-Tier Client Server �1st Generation�

8. 23 January 2002 8 3-Tier Client Server �2nd Generation� - Tuxedo

9. 23 January 2002 9 Where did Tuxedo come from? Bell Labs - 1993 Transactions for UniX Extended for Distributed Operations The key words here are distributed operations. Different parts of a transaction occur on different nodesThe key words here are distributed operations. Different parts of a transaction occur on different nodes

10. 23 January 2002 10 What is Tuxedo? It is the knots on the ends of the string! Imagine a telephone make of string and tin cans. Every child has made one If a tuxedo application where that telephone, the tuxedo bits would be the knots on the end of the stringImagine a telephone make of string and tin cans. Every child has made one If a tuxedo application where that telephone, the tuxedo bits would be the knots on the end of the string

11. 23 January 2002 11 PeopleSoft 2-Tier Client PeopleSoft windows client is written in C Imagine cutting up the two tier clientPeopleSoft windows client is written in C Imagine cutting up the two tier client

12. 23 January 2002 12 PeopleSoft 3-Tier Client

13. 23 January 2002 13 PeopleSoft 2-Tier Client Client Connects Directly to the Database 1 database connection per window inactivity timeout

14. 23 January 2002 14 PeopleSoft 2-Tier Client Client Connects Directly to the Database 1 database connection per window inactivity timeout What happens in 3-Tier?

15. 23 January 2002 15 PS/Tuxedo Internal Structure 1st Application Server Process Started Bulletin Board Liaison (BBL) Reads PSTUXCFG configuration file Shared Memory Segment Bulletin Board (BB)

16. 23 January 2002 16 PS/Tuxedo Internal Structure Workstation Listener (WSL) Listens for connection request from clients on a specified port Client initially contacts the WSL

17. 23 January 2002 17 PS/Tuxedo Internal Structure Workstation Listener (WSL) Client needs to know where to find the WSL Machine and port specified in Configuration Manager

18. 23 January 2002 18 PS/Tuxedo Internal Structure Workstation Handler (WSH) handles traffic from client after connection Client is assigned to a port on the WSH WSH port range can be constrained to match a firewall Additional WSH can be spawned if necessary Compress Messages

19. 23 January 2002 19 PS/Tuxedo Internal Structure

20. 23 January 2002 20 How does it work?

21. 23 January 2002 21 How does PIA work? Java Client & PIA use Jolt Java Class Library Jolt Listener maps Java functions to Tuxedo Messages BEA Jolt connects Java clients to applications built using the BEA TUXEDO system. The TUXEDO system provides a set of modular services, each offering specific functionality related to the application as a whole. For example, a simple banking application might have services such as INQUIRY, WITHDRAW, TRANSFER, and DEPOSIT. In PeopleSoft it is the functionality of the client Typically, service requests are implemented in C or COBOL as a sequence of calls to a program library. The Jolt Server implementation acts as a proxy for the Jolt client, invoking the TUXEDO service on behalf of the client. The BEA Jolt 1.2 Server accepts requests from the Jolt clients and maps those requests into TUXEDO service requests. Diagram is taken from BEA Jolt ManualBEA Jolt connects Java clients to applications built using the BEA TUXEDO system. The TUXEDO system provides a set of modular services, each offering specific functionality related to the application as a whole. For example, a simple banking application might have services such as INQUIRY, WITHDRAW, TRANSFER, and DEPOSIT. In PeopleSoft it is the functionality of the client Typically, service requests are implemented in C or COBOL as a sequence of calls to a program library. The Jolt Server implementation acts as a proxy for the Jolt client, invoking the TUXEDO service on behalf of the client. The BEA Jolt 1.2 Server accepts requests from the Jolt clients and maps those requests into TUXEDO service requests. Diagram is taken from BEA Jolt Manual

22. 23 January 2002 22 Jolt Internal Structure

23. 23 January 2002 23 New structure for PIA (Apache Webserver)

24. 23 January 2002 24 What is the difference between the web servers? Apache Separate Servlet Engine Separate encryption module (Raven) Weblogic Contains an integral servlet container

25. 23 January 2002 25 Progressive Connection Concentration Typical picture 35,000 users 1000 concurrent connections 1000 users active in the last n minutes where n is the Java + JSH timeout therefore need 1000 threads but maybe only 4 JVMs 100 JSH assumes 10 clients per JSH 10-20 PSAPPSRV processes each PSAPPSRV connects to database 1 database

26. 23 January 2002 26 Configuration

27. 23 January 2002 27 Configuration Files psappsrv.cfg contains configurable parameters psappsrv.val validation file for configurable values psappsrv.ubx configuration template for Tuxedo domain psappsrv.ubb configuration file for Tuxedo domain PSTUXCFG compiled configuration file read by BBL at domain startup

28. 23 January 2002 28 psappsrv.ubx -v- psappsrv.ubb psappsrv.ubx & psappsrv.cfg PeopleSoft file make changes to these files difficult to read psappsrv.ubb Tuxedo file never make changes to this file easier to read this file is compiled by tmloadcf to generate domain configuration file PSTUXCFG variables fully resolved

29. 23 January 2002 29 psappsrv.ubb Five (or Six) Sections Resources resource limits -> BB sizing Machine Resource limits. Environment (Network Connection between Machines) Groups Logical Split (BASE, APPSRV, JREPGRP, JSLGRP) Servers Number of servers, Command line Services Timeout, Load, Priority

30. 23 January 2002 30 Application Server Configuration Issue Move quick PSAPPSRV services into a second server (PSQCKSRV) (y/n)? <=PT7.58 this question is misleading services are additionally advertised on PSQCKSRV as well as PSAPPSRV so they are advertised on more than one queue >=PT7.59 and PT8.1 services are de-advertised from PSAPPSRV quick services not used in PIA, only IC* services so not a problem

31. 23 January 2002 31 PeopleTools <=7.58 Quick Service

32. 23 January 2002 32 Inside the Bulletin Board Without Load Balancing (default) Table of servers process where each service is advertised Read and populated as a stack WSL looks for first free server where request can be enqueued This can cause problems

33. 23 January 2002 33 Load Balancing Load Balancing is about choosing the queue upon which to place the message by default all requests have the same load

34. 23 January 2002 34 Load Balancing Trace Add -r to application server command line CLOPT="-r -e APPQ.stderr -A -- -C psappsrv.cfg -D GP81O81 -S PSAPPSRV� -r enables trace to file �stderr� (not channel 2) -e overrides the output filename process trace file with Tuxedo utility �txrpt� obtain average service time make load proportional to average service time

35. 23 January 2002 35 Load Balancing Trace Output from TXRPT SVCNAME 18p-19p TOTALS Num/Avg Num/Avg --------------- -------- ------- SqlRequest 16/0.10 16/0.10 MgrClear 5/0.04 5/0.04 RamList 1/0.43 1/0.43 SamGetParmsSvc 1/0.50 1/0.50 --------------- ------- ------- TOTALS 23/0.12 23/0.12

36. 23 January 2002 36 Load Balancing load = execution time (seconds) x 100

37. 23 January 2002 37 Multiple Queues on Busy System You can have many server processes on a single queue BEA recommend no more that 10 server processes per queue If you have queue contention it is likely to impede the WSH/JSH processes enqueuing requests. You are unlikely to observe queuing with PQ. So, if you want more than 10 PSAPPSRV processes you should use two APPQ queues In, which case you should to enable load balancing and you must calculate the load. Less significant with PIA because services different.

38. 23 January 2002 38 Multiple Queues on Busy System

39. 23 January 2002 39 Multiple Queues on Busy System psadmin will not correctly calculate MAXACCESSERS MAXSERVICES tmadmin->bbs will report maximum number of services count of all services advertised on each server process and if you have multiple work station listeners MAXWSCLIENTS

40. 23 January 2002 40 Service Priority Priority is about queue jumping servers de-queue the request 9 out of 10 de-queued on the basis of priority 1 out 10 is taken from the front of the queue

41. 23 January 2002 41 Service Priority All PeopleSoft Services have same priority specified in SERVICES section in psappsrv.ubb therefore there is no queue jumping no reason to believe that it would be advantageous

42. 23 January 2002 42 Tuning Tip Financials On-line journal edit & post - remote call Service initiates a Cobol process Process can take minutes RemoteCall services can block APPQ Move RemoteCall service to a separate queue

43. 23 January 2002 43 1 domain on many nodes Multi-Node Domain

44. 23 January 2002 44 Multi-Node Domain Distinct Bulletin Board Liaison process Master Node TLISTEN process Listens for incoming requests BRIDGE Sends & Receives messages between nodes in a load balancing environment configurable bridge load add to remote queues limiting number of messages sent across the bridge

45. 23 January 2002 45 Multi-Node Domain DO ATTEMPT THIS WITHOUT BEA CONSULTANCY At UBS PT 7.5x susceptible to network errors integrated into HP service guard ultimately this was removed from production because of transient network errors between the nodes causing the domain to �partition�

46. 23 January 2002 46 So what is going on? tmadmin Tuxedo command line interface also available within psadmin

47. 23 January 2002 47 tmadmin commands pq - printqueues amount of work queued and waiting not always accurate

48. 23 January 2002 48 tmadmin commands psr - printservers reports activity of individual server processes

49. 23 January 2002 49 tmadmin commands psc - printservices each service on each server

50. 23 January 2002 50 tmadmin commands Lots of other tmadmin commands documented in the Tuxedo documentation Use them to dynamically reconfigure the domain dynamically resize the domain start/shutdown extra servers monitoring scripts tmadmin -r for read only mode

51. 23 January 2002 51 Java Administrative Applet

52. 23 January 2002 52 Conclusion So now you know BEA Tuxedo/Jolt is a messaging protocol How the messages are moved around How PeopleSoft has used it How the PIA has been constructed Options for Tuning and Sizing Load Balancing Multi-Node domains Monitoring Scripts BEA Documentation is on your CD http://e-docs.bea.com

53. 23 January 2002 53 More Information about PT8 Application Server PeopleSoft 2000 Conference: Americas 8006: PeopleTools 8.1 Application Server 8010: Inside the Application Server Buzz Voelker PeopleTools Application Server Team

54. 23 January 2002 54 Final Caveat PeopleSoft will not support any of the changes that I have described in this presentation Everything described in this presentation has been done, in production, at UBS If you want to use a multi-node domain you are strongly recommended to arrange support/consultancy direct from BEA.

55. 23 January 2002 55 Questions?

56. PeopleTools Application Server & BEA Tuxedo Internals david.kurtz@go-faster.co.uk Go-Faster Consultancy Ltd. http://www.go-faster.co.uk