850 likes | 1.17k Views
PeopleSoft Performance Metrics. nalin.n.patel@abbey.com david.kurtz@go-faster.co.uk. Abbey key facts 1. Sixth largest bank by assets in the UK Founded in 1944 Currently have approximately 18m customers 741 branches across UK. Abbey key facts 2.
E N D
PeopleSoftPerformance Metrics nalin.n.patel@abbey.com david.kurtz@go-faster.co.uk
Abbey key facts 1 • Sixth largest bank by assets in the UK • Founded in 1944 • Currently have approximately 18m customers • 741 branches across UK
Abbey key facts 2 • Abbey's main offices are in London, Milton Keynes, Bradford, Glasgow and Belfast. • We have around 26,000 people (full time equivalent) • We have about 1.8 million shareholders • Assets at 30 June 2004 - £171 billion • Personal Financial Services trading profit before tax for 6 months to 30 June 2004 - £340 million
History of PeopleSoft at Abbey • PeopleSoft HRMS acquired for recruitment in 1994 • Implemented PeopleSoft HRMS in 1997 • Recruitment, Personnel & Training • Paylink used to send data from HRMS to payroll • Workflow and self-service with v7.5 in 2000 • JAVA HTML Clients • PeopleSoft HRMS upgraded to 8 SP1 in 2001 • Implemented PeopleSoft Payroll in August 2003 • Project initiated to upgrade to HCM 8.8
Current Platform • AppServer runs on SUN E4500 • Database runs on SUN E10000 • Both boxes are shared with other applications • Tier 1 mirrored disks • Oracle 9.2.04
Why PeopleSoft Payroll ? • Integrated HRMS • Common infrastructure • Web enabled • Automate administrative functions • Manager Self-Service • Absence and maternity input • Employee Self-Service • Overtime input • On-line payslips • Real-time data input • Increase system availability
PeopleSoft Payroll Implementation • Development commenced in January 2002 • In-house IT Project team • Project delays due to re-scoping and internal re-structure • Streamed payroll during parallel run tests • Went live with payroll and absence in August 2003 • 30,000 staff and 7,000 pensioners • 12 streams introduced in February 2004 • Introduced hash partition in July 2004 due to increased run times • Identify and calculate taking 2.5 hours • But we had to tune it…
Agenda • What is Performance Tuning? • On-Line • Batch • What has been achieved at Abbey? • PeopleSoft Performance Monitor • The shape of things to come! • Questions?
Rules of Engagement • If you can’t hear me - say so now! • Feel free to ask on-topic questions during the presentation. • This presentation (with notes) will be available on the Conference website on Customer Connection and on • www.go-faster.co.uk
Independent consultant System Performance Tuning Oracle Databases Unix Tuxedo PeopleSoft Apps Book www.psftdba.com Who am I?
Aphorism • Performance is exactly what the user perceives it to be. No more, no less. • Poor performance is when the user’s perception does not match their expectation.
Performance Tuning is a search for lost time. Optimizing Oracle Performance Cary Millsap & Jeff Holt Aphorism • See • ‘Yet Another Performance Profiling Method’ (1999) • by Anjo Kolk, Shari Yamaguchi & Jim Viscusi • http://www.oraperf.com/whitepapers.html
When you have eliminated the impossible, whatever remains, however improbable, must be the truth. A Study in Scarlet, Arthur Conan-Doyle Aphorism • It is a capital mistake to theorise before one has data. The Memoirs of Sherlock Holmes, Arthur Conan-Doyle
Aphorism • Detection is, or ought to be, an exact science. It should be be treated in the same cold and unemotional manner. • The Sign of Four, Arthur Conan-Doyle
PIA (On-Line) Performance • Architecture • brief overview • Sources of Metrics • Graphical Techniques • Tracing • Techniques
It isn’t always the database! • Understand architecture • Web front end • Network • Application server • Database • What can you measure?
Java Servlet (presentation logic) Tuxedo Application Server (application logic) DBMS (application data & meta-data http / https Tuxedo Message SQL PeopleSoft Internet Architecture • PIA is 4-tier architecture • Each component can contribute to response time • Need to measure each component
Sources of Metrics (1) • Browser • Proxy Server • 3rd Party software • Local agent on PC • PeopleSoft Ping • From PT8.19, PT8.4 • Web Server • Access Log • Weblogic & Apache • Websphere does not generate a log
Sources of Metrics (2) • BEA/Tuxedo • Service Trace • TMADMIN • LogFence 4 • Oracle Database • SQL Trace (event 10046) • PeopleTools SQL Trace • Large overhead
Hint • All the data collected on a time base. • Therefore, make sure that the clocks on all machines are synchronised at least with each other. • PC clocks are particularly liable to drift.
Java Servlet (presentation logic) Tuxedo Application Server (application logic) DBMS (application data & meta-data http / https Tuxedo Message SQL Sources of Metrics Proxy Server Access Log Webserver Access Log
W3C Standard format for access log www.w3.org/pub/WWW/TR/logfile.html Used by Weblogic, IIS Apache is different WebSphere doesn’t have an access log! Web/Proxy Access Log
Web/Proxy Access Log • Fields • Date, Time • Time went log entry emitted • (End time not start time) • Time Taken • accurate to OS time units (1/100ths or 1/1000ths) • only 1s on Apache • Message size bytes • Client IP addresses & DNS names • URI stem & Query
How to Enable WebLogic 5.1 Log #weblogic.properties #required weblogic.httpd.enableLogFile=true weblogic.httpd.logFileName=access.log weblogic.httpd.logFileFormat=extended #recommended weblogic.httpd.logRotationType=date weblogic.httpd.logRotationPeriodMins=1440 weblogic.httpd.logRotationBeginTime=11-24-2000-00:00:00
Sample WebLogic Access Log #Version: 1.0 #Fields: date time time-taken bytes c-ip c-dns cs-method sc-status cs-uri-stem cs-uri-query 2002-03-15 21:51:03 0.11 2103 10.0.0.2 go-faster-1 GET 200 /peoplesoft8/signin.css - 2002-03-15 21:51:04 0.05 1186 10.0.0.2 go-faster-1 GET 200 /peoplesoft8/images/secondaryPSFTLogo.gif - 2002-03-15 21:51:10 3.414 8101 10.0.0.2 go-faster-1 POST 200 /servlets/iclientservlet/peoplesoft8/ ICType=Panel&Menu=ADMINISTER_WORKFORCE_(GBL)&Market=GBL&PanelGroupName=PERSONAL_DATA&RL=&target=main2
Apache Access Log #httpd.conf LogFormat "%{%Y.%m.%d %H:%M:%S}t|%T|%B|%u|%h|%{User-Agent}i|%>s|%m|%U|%q" monitoring CustomLog logs/access.log monitoring
Sample Apache Access Log 2002.02.26 09:57:06|0|275|-|127.0.0.1|Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Q312461)|200|GET|/peoplesoft8/cache/PT_NEXTTAB_ENG_1.gif| 2002.02.26 09:57:17|0|31847|-|127.0.0.1|Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Q312461)|200|POST|/servlets/iclientservlet|?ICType=Panel&Menu=ADMINISTER_WORKFORCE_(GBL)&Market=GBL&PanelGroupName=JOB_DATA 2002.02.26 09:57:26|0|30959|-|127.0.0.1|Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Q312461)|200|POST|/servlets/iclientservlet|?ICType=Panel&Menu=ADMINISTER_WORKFORCE_(GBL)&Market=GBL&PanelGroupName=JOB_DATA
What does the access log tell us? • A date and time for every request • Who made the request? • How long did it take to serve? • What was requested? • Which component (panel group)? • Indicate possible bandwidth problems.
Download Bandwidth #version: 1.0 #Fields: date time time-taken bytes c-ip c-dns cs-method sc-status cs-uri-stem cs-uri-query 2002-09-10 11:29:52 0.359 15832 141.93.249.74 141.93.249.74 GET 200 /XXXXXXXX/images/topPSFTLogo.gif -
Java Servlet (presentation logic) Tuxedo Application Server (application logic) DBMS (application data & meta-data http / https Tuxedo Message SQL Sources of Metrics Tuxedo Service Trace
BEA/Tuxedo Service Trace • Edit psappsrv.ubx file • CLOPT=“-r -e APPQ.stderr…” • -r enables trace to stderr file • -e qualifies name of stderr file
Edit psappsrv.cfg • Add a new variable to the trace section • Thus service trace can be enabled/disabled from configuration file • New variable will appear in the configuration dialogue [Trace] TuxedoServiceTrace=-r
Edit psappsrv.ubx • Change every CLOPT entry in psappsrv.ubx. • File name is set manually. • Same as queue name RQADDR="APPQ" CLOPT="{$Trace\TuxedoServiceTrace} -e {LOGDIR}{FS}{MACH}_{$Domain Settings\Domain ID}_APPQ.stderr {$PSAPPSRV\Spawn Server} -s@..{FS}psappsrv.lst -- -C {CFGFILE} -D {$Domain Settings\Domain ID} -S PSAPPSRV"
Resultant psappsrv.ubb • When domain is configured .ubb file is generated. All variables are fully resolved. CLOPT=”-r -e D:\ps\hr88\appserv\hr88\LOGS\HR88_APPQ.stderr -p 1,600:3,10 -s@..\psappsrv.lst -s@..\psqcksrv.lst -sICQuery -sSqlQuery:SqlRequest -- -C psappsrv.cfg -D HR88 -S PSAPPSRV"
Sample Tuxedo Service Trace • Each service called to domain is logged SERVICE PID SDATE STIME EDATE ETIME ------- --- ----- ----- ----- ----- @ICScript 390 1014717320 4586495 1014717322 4588397 @ICScript 390 1014717322 4588467 1014717322 4588537 @ICScript 390 1014717322 4588557 1014717323 4589259 @ICScript 390 1014717332 4598021 1014717332 4598171 @ICScript 390 1014717355 4621745 1014717361 4627554 @ICScript 390 1014717365 4631329 1014717366 4632511 @ICScript 390 1014717375 4641944 1014717377 4643306 @ICPanel 390 1014717380 4646761 1014717381 4647783 @ICPanel 390 1014717388 4654202 1014717391 4657567 @ICPanel 390 1014717396 4662614 1014717401 4667271
Tuxedo Service Trace • Service - Name of Service • Pid - An OS PID of listener or handler • Sdate/Edate - Start/End Date • Seconds since 00:00hrs GMT 1st January 1970. • Stime/Etime • Start/End Time in OS time units (1/ 100ths or 1/1000ths) • Hence calculate accurate service duration • Time on Server - not queuing
Txrpt Utility SERVICE SUMMARY REPORT SVCNAME 9a-10a TOTALS Num/Avg Num/Avg --------------- -------- ------- ICScript 8/1.43 8/1.43 ICPanel 6/3.47 6/3.47 JavaMgrGetObj 1/0.04 1/0.04 GetCertificate 1/0.73 1/0.73 --------------- ------- ------- TOTALS 16/2.06 16/2.06
Java Servlet (presentation logic) Tuxedo Application Server (application logic) DBMS (application data & meta-data http / https Tuxedo Message SQL Sources of Metrics Tuxedo tmadmin script
BEA/Tuxedo TMADMIN Utility • BEA Interactive command line interface • Can be included in scripts • -r for read only in monitoring scripts • (this is a different –r) • pq (printqueue) • psr (printserver) • pclt (printclient)
PQ • What queues in domain • How many servers on queue • How many requests queued • (How much work on queue) Prog Name Queue Name # Serve Wk Queued # Queued Ave. Len Machine --------- ------------------- --------- -------- -------- ------- JSL.exe 00095.00200 1 - 0 - GO-FASTER+ JREPSVR.exe 00094.00250 1 - 0 - GO-FASTER+ PSSAMSRV.exe SAMQ 1 - 0 - GO-FASTER+ BBL.exe 54455 1 - 0 - GO-FASTER+ WSL.exe 00001.00020 1 - 0 - GO-FASTER+ PSAPPSRV.exe APPQ 1 - 0 - GO-FASTER+
PSR • What servers exist? • How many requests have they handled? • What are they doing right now? Prog Name Queue Name Grp Name ID RqDone Load Done Current Service --------- ---------- -------- -- ------ --------- --------------- BBL.exe 54455 GO-FAST+ 0 3 150 ( IDLE ) PSAPPSRV.exe APPQ APPSRV 1 9 450 ICPanel PSSAMSRV.exe SAMQ APPSRV 100 0 0 ( IDLE ) WSL.exe 00001.00020 BASE 20 0 0 ( IDLE ) JSL.exe 00095.00200 JSLGRP 200 0 0 ( IDLE ) JREPSVR.exe 00094.00250 JREPGRP 250 5 250 ( IDLE )
PCLT • Who is connected? • Are they doing something? • When did they connect? LMID User Name Client Name Time Status Bgn/Cmmt/Abrt --------------- --------------- --------------- -------- ------- ------------- GO-FASTER-1 NT WSH 0:21:14 IDLE 0/0/0 GO-FASTER-1 NT JSH 0:21:13 IDLE 0/0/0 GO-FASTER-1 NT tmadmin 0:00:00 IDLE 0/0/0 GO-FASTER-1 PS JavaClient 0:02:30 BUSY/W 0/0/0
Java Servlet (presentation logic) Tuxedo Application Server (application logic) DBMS (application data & meta-data http / https Tuxedo Message SQL Sources of Metrics Oracle SQL*Trace
Oracle SQL Trace • Tuning tool • Session trace • Format it with TKPROF • (Or the www.HOTSOS.com profiler) • Find long running SQL statements • Total SQL Time for session call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Parse 72289 173.06 175.45 7 14654 94 0 Execute 77140 64.02 67.37 205495 766343 170611 78630 Fetch 50599 33.20 34.28 17302 971624 80146 143218 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 200028 270.28 277.10 222804 1752621 250851 221848
Small Application Servers • Create small application with just a single handler process • Enable trace on that process • Trace sample transaction (only one user) sys.dbms_system.set_sql_trace_in_session(<sid>,<serial>,TRUE);
Java Servlet (presentation logic) Tuxedo Application Server (application logic) DBMS (application data & meta-data http / https Tuxedo Message SQL Sources of Metrics Proxy Server Access Log Tuxedo tmadmin script Tuxedo Service Trace Oracle SQL*Trace Webserver Access Log
Graphical Analysis • Don’t drown in numbers • Draw a picture • Load Data into Database • SQL_Loader (Oracle 9i External Table) • Process Data • Matching, Aggregation • Oracle Analytic Function • Load data into Excel • Draw Graph
Sample SQL*Loader File LOAD DATA INFILE 'APPQ.stderr' REPLACE INTO TABLE txrpt WHEN (1) = '@' FIELDS TERMINATED BY WHITESPACE TRAILING NULLCOLS (service "substr(:service,2)" -- remove leading @ ,pid ,stimestamp ":stimestamp/86400+2/24+TO_DATE('01011970','DDMMYYYY')" ,stime ":stime/100" ,queue ”’APPQ’" ,etime ":etime/100")