560 likes | 794 Views
PeopleTools Application Server & BEA Tuxedo Internals. david.kurtz@go-faster.co.uk Go-Faster Consultancy Ltd. http://www.go-faster.co.uk. Who am I?. Independent Consultant Performance Tuning PeopleSoft / Oracle. Rules of Engagement. Interrupt me: if you think I have got something wrong
E N D
PeopleTools Application Server & BEA Tuxedo Internals david.kurtz@go-faster.co.uk Go-Faster Consultancy Ltd. http://www.go-faster.co.uk
Who am I? • Independent Consultant • Performance Tuning • PeopleSoft / Oracle
Rules of Engagement • Interrupt me: • if you think I have got something wrong • if you have a question • if you can’t hear me
BEA Tuxedo • Historical Overview • Tuxedo Internal Architecture • PeopleSoft 3-tier client implementation • How does PeopleSoft Internet Architecture (PIA) fit in • Configuration issues • Monitoring
Data Access (DBMS) Business Logic Presentation Brief Historical Background • In the beginning • monolithic
SQL Data Access (DBMS) Business Logic Presentation Raw Data LAN Client Server Client Server • Two-tier client-server
Data Access (DBMS) Request GUI + Application Logic Response Business Logic LAN Client Server 3-Tier Client Server • “1st Generation”
Presentation Business Logic Data Management Service 1 Service 2 ... Service n TUXEDO Messaging SQL Presentation Logic DBMS 1 ... DBMS n TUXEDO Clients TUXEDO Servers Database Servers 3-Tier Client Server • “2nd Generation” - Tuxedo
Where did Tuxedo come from? • Bell Labs - 1993 • Transactions for • UniX • Extended for • Distributed • Operations
What is Tuxedo? • It is the knots on the ends of the string!
Presentation Logic Panel Load Panel Save Field Change etc... PeopleSoft 2-Tier Client
Tuxedo Client Tuxedo Server Panel Load Panel Save Field Change etc... PeopleSoft 3-Tier Client Presentation Logic Server Side Client Side
PeopleSoft 2-Tier Client • Client Connects Directly to the Database • 1 database connection per window • inactivity timeout DATABASE
PeopleSoft 2-Tier Client • Client Connects Directly to the Database • 1 database connection per window • inactivity timeout • What happens in 3-Tier? DATABASE
PS/Tuxedo Internal Structure • 1st Application Server Process Started • Bulletin Board Liaison (BBL) • Reads PSTUXCFG configuration file • Shared Memory Segment • Bulletin Board (BB) BB DATABASE PSTUXCFG BBL
PS/Tuxedo Internal Structure • Workstation Listener (WSL) • Listens for connection request from clients on a specified port • Client initially contacts the WSL WSL BB DATABASE BBL
PS/Tuxedo Internal Structure • Workstation Listener (WSL) • Client needs to know where to find the WSL • Machine and port specified in Configuration Manager WSL BB BBL
PS/Tuxedo Internal Structure • Workstation Handler (WSH) • handles traffic from client after connection • Client is assigned to a port on the WSH • WSH port range can be constrained to match a firewall • Additional WSH can be spawned if necessary • Compress Messages WSL WSH BB DATABASE BBL
PSAUTH AUTHQ APPQ PSAPPSRV QCKQ PSQCKSRV QRYQ PSQRYSRV APIQ PSAPISRV SAMQ PSSAMSRV PS/Tuxedo Internal Structure WSL WSH BB DATABASE BBL
PSAUTH AUTHQ APPQ PSAPPSRV QCKQ PSQCKSRV QRYQ PSQRYSRV APIQ PSAPISRV SAMQ PSSAMSRV How does it work? WSHQ WSL WSH BB DATABASE BBL
How does PIA work? • Java Client & PIA use Jolt • Java Class Library • Jolt Listener maps Java functions to Tuxedo Messages
PSAUTH AUTHQ APPQ PSAPPSRV QCKQ PSQCKSRV QRYQ PSQRYSRV APIQ PSAPISRV JSH SAMQ PSSAMSRV Jolt Internal Structure JSHQ JSL JREPSRV WSL WSH BB JREPOSITORY DATABASE BBL
Disk (static objects) New structure for PIA (Apache Webserver) Sessionid = “xy1k…vm8010 http 1.1 address+cookie (stateless) Cookie Java JVM Servlet Engine Servlet thread IS stateful Servlet Container Servlet Webserver Servlet Thread Servlet Thread HTTPD Client Servlet Thread Client mod_jserv Port 8010 Tuxedo Application Server JSL JSH JSH JSH
What is the difference between the web servers? • Apache • Separate Servlet Engine • Separate encryption module (Raven) • Weblogic • Contains an integral servlet container
Progressive Connection Concentration 35000 users • Typical picture • 35,000 users • 1000 concurrent connections • 1000 users active in the last n minutes • where n is the Java + JSH timeout • therefore need 1000 threads • but maybe only 4 JVMs • 100 JSH • assumes 10 clients per JSH • 10-20 PSAPPSRV processes • each PSAPPSRV connects to database • 1 database 1000 connections 4 JVMs 1000 Java Threads 100 JSHs 10-20 PSAPPSRVs 1 database
Configuration psappsrv.val psadmin (configuration dialogue) psappsrv.cfg ubbgen (PS Utility) psappsrv.ubx psappsrv.ubb tmloadcf (tuxedo process) PSTUXCFG BBL (domain startup)
psappsrv.cfg contains configurable parameters psappsrv.val validation file for configurable values psappsrv.ubx configuration template for Tuxedo domain psappsrv.ubb configuration file for Tuxedo domain PSTUXCFG compiled configuration file read by BBL at domain startup Configuration Files
psappsrv.ubx & psappsrv.cfg PeopleSoft file make changes to these files difficult to read psappsrv.ubb Tuxedo file never make changes to this file easier to read this file is compiled by tmloadcf to generate domain configuration file PSTUXCFG variables fully resolved psappsrv.ubx -v- psappsrv.ubb
psappsrv.ubb • Five (or Six) Sections • Resources • resource limits -> BB sizing • Machine • Resource limits. Environment • (Network • Connection between Machines) • Groups • Logical Split (BASE, APPSRV, JREPGRP, JSLGRP) • Servers • Number of servers, Command line • Services • Timeout, Load, Priority
Application Server Configuration Issue • Move quick PSAPPSRV services into a second server (PSQCKSRV) (y/n)? • <=PT7.58 this question is misleading • services are additionally advertised on PSQCKSRV as well as PSAPPSRV • so they are advertised on more than one queue • >=PT7.59 and PT8.1 • services are de-advertised from PSAPPSRV • quick services not used in PIA, only IC* services • so not a problem
APPQ PSAPPSRV QCKQ PSQCKSRV PeopleTools <=7.58 • Quick Service WSL WSH BB DATABASE BBL
Inside the Bulletin Board • Without Load Balancing (default) • Table of servers process where each service is advertised • Read and populated as a stack • WSL looks for first free server where request can be enqueued • This can cause problems PSAPPSRV PSQCKSRV PSQCKSRV PSQCKSRV PSAPPSRV PSAPPSRV PSAPPSRV
50 50 50 APPQ PSAPPSRV 50 50 50 50 50 50 50 QCKQ PSQCKSRV Load Balancing • Load Balancing is about choosing the queue upon which to place the message • by default all requests have the same load
Load Balancing Trace • Add -r to application server command line • CLOPT="-r -e APPQ.stderr -A -- -C psappsrv.cfg -D GP81O81 -S PSAPPSRV” • -r enables trace to file ‘stderr’ (not channel 2) • -e overrides the output filename • process trace file with Tuxedo utility ‘txrpt’ • obtain average service time • make load proportional to average service time
Load Balancing Trace • Output from TXRPT SVCNAME 18p-19p TOTALS Num/Avg Num/Avg --------------- -------- ------- SqlRequest 16/0.10 16/0.10 MgrClear 5/0.04 5/0.04 RamList 1/0.43 1/0.43 SamGetParmsSvc 1/0.50 1/0.50 --------------- ------- ------- TOTALS 23/0.12 23/0.12
78 386 135 APPQ PSAPPSRV 50 6 30 3 63 3 7 QCKQ PSQCKSRV Load Balancing • load = execution time (seconds) x 100 599 116
Multiple Queues on Busy System • You can have many server processes on a single queue • BEA recommend no more that 10 server processes per queue • If you have queue contention it is likely to impede the WSH/JSH processes enqueuing requests. You are unlikely to observe queuing with PQ. • So, if you want more than 10 PSAPPSRV processes you should use two APPQ queues • In, which case you should to enable load balancing and you must calculate the load. • Less significant with PIA because services different.
APPQ1 PSAPPSRV APPQ2 PSAPPSRV Multiple Queues on Busy System WSL WSH BB DATABASE BBL
Multiple Queues on Busy System • psadmin will not correctly calculate • MAXACCESSERS • MAXSERVICES • tmadmin->bbs • will report maximum number of services • count of all services advertised on each server process • and if you have multiple work station listeners • MAXWSCLIENTS
80 50 50 50 50 50 PSQCKSRV Service Priority • Priority is about queue jumping • servers de-queue the request • 9 out of 10 de-queued on the basis of priority • 1 out 10 is taken from the front of the queue QCKQ
Service Priority • All PeopleSoft Services have same priority • specified in SERVICES section in psappsrv.ubb • therefore there is no queue jumping • no reason to believe that it would be advantageous
Tuning Tip • Financials • On-line journal edit & post - remote call • Service initiates a Cobol process • Process can take minutes • RemoteCall services can block APPQ • Move RemoteCall service to a separate queue
BB APPQ PSAPPSRV BBL BRIDGE BBL WSL WSH WSL WSH BB APPQ PSAPPSRV Multi-Node Domain • 1 domain on many nodes DBBL BRIDGE TLISTEN DATABASE TLISTEN
Multi-Node Domain • Distinct Bulletin Board Liaison process • Master Node • TLISTEN process • Listens for incoming requests • BRIDGE • Sends & Receives messages between nodes • in a load balancing environment • configurable bridge load add to remote queues • limiting number of messages sent across the bridge
Multi-Node Domain • DO ATTEMPT THIS WITHOUT BEA CONSULTANCY • At UBS • PT 7.5x • susceptible to network errors • integrated into HP service guard • ultimately this was removed from production because of transient network errors between the nodes causing the domain to ‘partition’
So what is going on? • tmadmin • Tuxedo command line interface • also available within psadmin
tmadmin commands • pq - printqueues • amount of work queued and waiting • not always accurate
tmadmin commands • psr - printservers • reports activity of individual server processes
tmadmin commands • psc - printservices • each service on each server
tmadmin commands • Lots of other tmadmin commands • documented in the Tuxedo documentation • Use them to • dynamically reconfigure the domain • dynamically resize the domain • start/shutdown extra servers • monitoring scripts • tmadmin -r for read only mode