PeopleTools Application Server & BEA Tuxedo Internals

1. PeopleTools Application Server & BEA Tuxedo Internals david.kurtz@go-faster.co.uk Go-Faster Consultancy Ltd. http://www.go-faster.co.uk

2. Who am I? • Independent Consultant • Performance Tuning • PeopleSoft / Oracle

3. Rules of Engagement • Interrupt me: • if you think I have got something wrong • if you have a question • if you can’t hear me

4. BEA Tuxedo • Historical Overview • Tuxedo Internal Architecture • PeopleSoft 3-tier client implementation • How does PeopleSoft Internet Architecture (PIA) fit in • Configuration issues • Monitoring

5. Data Access (DBMS) Business Logic Presentation Brief Historical Background • In the beginning • monolithic

6. SQL Data Access (DBMS) Business Logic Presentation Raw Data LAN Client Server Client Server • Two-tier client-server

7. Data Access (DBMS) Request GUI + Application Logic Response Business Logic LAN Client Server 3-Tier Client Server • “1st Generation”

8. Presentation Business Logic Data Management Service 1 Service 2 ... Service n TUXEDO Messaging SQL Presentation Logic DBMS 1 ... DBMS n TUXEDO Clients TUXEDO Servers Database Servers 3-Tier Client Server • “2nd Generation” - Tuxedo

9. Where did Tuxedo come from? • Bell Labs - 1993 • Transactions for • UniX • Extended for • Distributed • Operations

10. What is Tuxedo? • It is the knots on the ends of the string!

11. Presentation Logic Panel Load Panel Save Field Change etc... PeopleSoft 2-Tier Client

12. Tuxedo Client Tuxedo Server Panel Load Panel Save Field Change etc... PeopleSoft 3-Tier Client Presentation Logic Server Side Client Side

13. PeopleSoft 2-Tier Client • Client Connects Directly to the Database • 1 database connection per window • inactivity timeout DATABASE

14. PeopleSoft 2-Tier Client • Client Connects Directly to the Database • 1 database connection per window • inactivity timeout • What happens in 3-Tier? DATABASE

15. PS/Tuxedo Internal Structure • 1st Application Server Process Started • Bulletin Board Liaison (BBL) • Reads PSTUXCFG configuration file • Shared Memory Segment • Bulletin Board (BB) BB DATABASE PSTUXCFG BBL

16. PS/Tuxedo Internal Structure • Workstation Listener (WSL) • Listens for connection request from clients on a specified port • Client initially contacts the WSL WSL BB DATABASE BBL

17. PS/Tuxedo Internal Structure • Workstation Listener (WSL) • Client needs to know where to find the WSL • Machine and port specified in Configuration Manager WSL BB BBL

18. PS/Tuxedo Internal Structure • Workstation Handler (WSH) • handles traffic from client after connection • Client is assigned to a port on the WSH • WSH port range can be constrained to match a firewall • Additional WSH can be spawned if necessary • Compress Messages WSL WSH BB DATABASE BBL

19. PSAUTH AUTHQ APPQ PSAPPSRV QCKQ PSQCKSRV QRYQ PSQRYSRV APIQ PSAPISRV SAMQ PSSAMSRV PS/Tuxedo Internal Structure WSL WSH BB DATABASE BBL

20. PSAUTH AUTHQ APPQ PSAPPSRV QCKQ PSQCKSRV QRYQ PSQRYSRV APIQ PSAPISRV SAMQ PSSAMSRV How does it work? WSHQ WSL WSH BB DATABASE BBL

21. How does PIA work? • Java Client & PIA use Jolt • Java Class Library • Jolt Listener maps Java functions to Tuxedo Messages

22. PSAUTH AUTHQ APPQ PSAPPSRV QCKQ PSQCKSRV QRYQ PSQRYSRV APIQ PSAPISRV JSH SAMQ PSSAMSRV Jolt Internal Structure JSHQ JSL JREPSRV WSL WSH BB JREPOSITORY DATABASE BBL

23. Disk (static objects) New structure for PIA (Apache Webserver) Sessionid = “xy1k…vm8010 http 1.1 address+cookie (stateless) Cookie Java JVM Servlet Engine Servlet thread IS stateful Servlet Container Servlet Webserver Servlet Thread Servlet Thread HTTPD Client Servlet Thread Client mod_jserv Port 8010 Tuxedo Application Server JSL JSH JSH JSH

24. What is the difference between the web servers? • Apache • Separate Servlet Engine • Separate encryption module (Raven) • Weblogic • Contains an integral servlet container

25. Progressive Connection Concentration 35000 users • Typical picture • 35,000 users • 1000 concurrent connections • 1000 users active in the last n minutes • where n is the Java + JSH timeout • therefore need 1000 threads • but maybe only 4 JVMs • 100 JSH • assumes 10 clients per JSH • 10-20 PSAPPSRV processes • each PSAPPSRV connects to database • 1 database 1000 connections 4 JVMs 1000 Java Threads 100 JSHs 10-20 PSAPPSRVs 1 database

26. Configuration psappsrv.val psadmin (configuration dialogue) psappsrv.cfg ubbgen (PS Utility) psappsrv.ubx psappsrv.ubb tmloadcf (tuxedo process) PSTUXCFG BBL (domain startup)

27. psappsrv.cfg contains configurable parameters psappsrv.val validation file for configurable values psappsrv.ubx configuration template for Tuxedo domain psappsrv.ubb configuration file for Tuxedo domain PSTUXCFG compiled configuration file read by BBL at domain startup Configuration Files

28. psappsrv.ubx & psappsrv.cfg PeopleSoft file make changes to these files difficult to read psappsrv.ubb Tuxedo file never make changes to this file easier to read this file is compiled by tmloadcf to generate domain configuration file PSTUXCFG variables fully resolved psappsrv.ubx -v- psappsrv.ubb

29. psappsrv.ubb • Five (or Six) Sections • Resources • resource limits -> BB sizing • Machine • Resource limits. Environment • (Network • Connection between Machines) • Groups • Logical Split (BASE, APPSRV, JREPGRP, JSLGRP) • Servers • Number of servers, Command line • Services • Timeout, Load, Priority

30. Application Server Configuration Issue • Move quick PSAPPSRV services into a second server (PSQCKSRV) (y/n)? • <=PT7.58 this question is misleading • services are additionally advertised on PSQCKSRV as well as PSAPPSRV • so they are advertised on more than one queue • >=PT7.59 and PT8.1 • services are de-advertised from PSAPPSRV • quick services not used in PIA, only IC* services • so not a problem

31. APPQ PSAPPSRV QCKQ PSQCKSRV PeopleTools <=7.58 • Quick Service WSL WSH BB DATABASE BBL

32. Inside the Bulletin Board • Without Load Balancing (default) • Table of servers process where each service is advertised • Read and populated as a stack • WSL looks for first free server where request can be enqueued • This can cause problems PSAPPSRV PSQCKSRV PSQCKSRV PSQCKSRV PSAPPSRV PSAPPSRV PSAPPSRV

33. 50 50 50 APPQ PSAPPSRV 50 50 50 50 50 50 50 QCKQ PSQCKSRV Load Balancing • Load Balancing is about choosing the queue upon which to place the message • by default all requests have the same load

34. Load Balancing Trace • Add -r to application server command line • CLOPT="-r -e APPQ.stderr -A -- -C psappsrv.cfg -D GP81O81 -S PSAPPSRV” • -r enables trace to file ‘stderr’ (not channel 2) • -e overrides the output filename • process trace file with Tuxedo utility ‘txrpt’ • obtain average service time • make load proportional to average service time

35. Load Balancing Trace • Output from TXRPT SVCNAME 18p-19p TOTALS Num/Avg Num/Avg --------------- -------- ------- SqlRequest 16/0.10 16/0.10 MgrClear 5/0.04 5/0.04 RamList 1/0.43 1/0.43 SamGetParmsSvc 1/0.50 1/0.50 --------------- ------- ------- TOTALS 23/0.12 23/0.12

36. 78 386 135 APPQ PSAPPSRV 50 6 30 3 63 3 7 QCKQ PSQCKSRV Load Balancing • load = execution time (seconds) x 100 599 116

37. Multiple Queues on Busy System • You can have many server processes on a single queue • BEA recommend no more that 10 server processes per queue • If you have queue contention it is likely to impede the WSH/JSH processes enqueuing requests. You are unlikely to observe queuing with PQ. • So, if you want more than 10 PSAPPSRV processes you should use two APPQ queues • In, which case you should to enable load balancing and you must calculate the load. • Less significant with PIA because services different.

38. APPQ1 PSAPPSRV APPQ2 PSAPPSRV Multiple Queues on Busy System WSL WSH BB DATABASE BBL

39. Multiple Queues on Busy System • psadmin will not correctly calculate • MAXACCESSERS • MAXSERVICES • tmadmin->bbs • will report maximum number of services • count of all services advertised on each server process • and if you have multiple work station listeners • MAXWSCLIENTS

40. 80 50 50 50 50 50 PSQCKSRV Service Priority • Priority is about queue jumping • servers de-queue the request • 9 out of 10 de-queued on the basis of priority • 1 out 10 is taken from the front of the queue QCKQ

41. Service Priority • All PeopleSoft Services have same priority • specified in SERVICES section in psappsrv.ubb • therefore there is no queue jumping • no reason to believe that it would be advantageous

42. Tuning Tip • Financials • On-line journal edit & post - remote call • Service initiates a Cobol process • Process can take minutes • RemoteCall services can block APPQ • Move RemoteCall service to a separate queue

43. BB APPQ PSAPPSRV BBL BRIDGE BBL WSL WSH WSL WSH BB APPQ PSAPPSRV Multi-Node Domain • 1 domain on many nodes DBBL BRIDGE TLISTEN DATABASE TLISTEN

44. Multi-Node Domain • Distinct Bulletin Board Liaison process • Master Node • TLISTEN process • Listens for incoming requests • BRIDGE • Sends & Receives messages between nodes • in a load balancing environment • configurable bridge load add to remote queues • limiting number of messages sent across the bridge

45. Multi-Node Domain • DO ATTEMPT THIS WITHOUT BEA CONSULTANCY • At UBS • PT 7.5x • susceptible to network errors • integrated into HP service guard • ultimately this was removed from production because of transient network errors between the nodes causing the domain to ‘partition’

46. So what is going on? • tmadmin • Tuxedo command line interface • also available within psadmin

47. tmadmin commands • pq - printqueues • amount of work queued and waiting • not always accurate

48. tmadmin commands • psr - printservers • reports activity of individual server processes

49. tmadmin commands • psc - printservices • each service on each server

50. tmadmin commands • Lots of other tmadmin commands • documented in the Tuxedo documentation • Use them to • dynamically reconfigure the domain • dynamically resize the domain • start/shutdown extra servers • monitoring scripts • tmadmin -r for read only mode