670 likes | 833 Views
Performance Metrics: What to Measure in the PeopleSoft 8 Environment. David Kurtz Go-Faster Consultancy Ltd. david.kurtz@go-faster.co.uk www.go-faster.co.uk. Agenda. What is Performance Tuning? On-Line Batch What has been achieved? Questions?. Rules of Engagement.
E N D
Performance Metrics:What to Measure in the PeopleSoft 8 Environment David Kurtz Go-Faster Consultancy Ltd. david.kurtz@go-faster.co.uk www.go-faster.co.uk Go-Faster Consultancy Ltd.
Agenda • What is Performance Tuning? • On-Line • Batch • What has been achieved? • Questions? Go-Faster Consultancy Ltd.
Rules of Engagement • If you can’t hear me - say so now! • Feel free to ask on-topic questions during the presentation. • This presentation (with notes) will be available on the Tech SIG page on Customer Connection and on • www.go-faster.co.uk Go-Faster Consultancy Ltd.
Who am I? • DBA • Independent consultant • Performance tuning • UK Oracle User Group • Unix SIG Chair • www.ukoug.org Go-Faster Consultancy Ltd.
Aphorism • Performance is exactly what the user perceives it to be. No more, no less. • Poor performance is when the user’s perception does not match their expectation. Go-Faster Consultancy Ltd.
Aphorism • Performance Tuning is a search for lost time. Go-Faster Consultancy Ltd.
When you have eliminated the impossible, whatever remains, however improbable, must be the truth. A Study in Scarlet, Arthur Conan-Doyle Aphorism • It is a capital mistake to theorise before one has data. The Memoirs of Sherlock Holmes, Arthur Conan-Doyle Go-Faster Consultancy Ltd.
Aphorism • Detection is, or ought to be, an exact science. It should be be treated in the same cold and unemotional manner. • The Sign of Four, Arthur Conan-Doyle Go-Faster Consultancy Ltd.
PIA (On-Line) Performance • Architecture • brief overview • Sources of Metrics • Graphical Techniques • Tracing • Techniques Go-Faster Consultancy Ltd.
It isn’t always the database! • Understand architecture • Web front end • Network • Application server • Database • What can you measure? Go-Faster Consultancy Ltd.
Java Servlet (presentation logic) Tuxedo Application Server (application logic) DBMS (application data & meta-data http / https Tuxedo Message SQL PeopleSoft Internet Architecture • PIA is 4-tier architecture • Each component can contribute to response time • Need to measure each component Go-Faster Consultancy Ltd.
Sources of Metrics (1) • Browser • Proxy Server • 3rd Party software • Local agent on PC • Web Server • Access Log Go-Faster Consultancy Ltd.
Sources of Metrics (2) • BEA/Tuxedo • Service Trace • TMADMIN • Oracle Database • SQL_Trace • PSFT Application Server Trace Go-Faster Consultancy Ltd.
Hint • All the data collected on a time base. • Therefore, make sure that the clocks on all machines are synchronised at least with each other. • PC clocks are particularly liable to drift. Go-Faster Consultancy Ltd.
Java Servlet (presentation logic) Tuxedo Application Server (application logic) DBMS (application data & meta-data http / https Tuxedo Message SQL Sources of Metrics Proxy Server Access Log Webserver Access Log Go-Faster Consultancy Ltd.
Standard format for access log www.w3.org/pub/WWW/TR/logfile.html Apache is different Web/Proxy Access Log Go-Faster Consultancy Ltd.
Web/Proxy Access Log • Fields • Date, Time (finish not start) • Time Taken • accurate to OS time units (1/100ths or 1/1000ths) • only 1s on Apache • Message size bytes • Client IP addresses & DNS names • URI stem & Query Go-Faster Consultancy Ltd.
How to Enable WebLogic Trace #weblogic.properties #required weblogic.httpd.enableLogFile=true weblogic.httpd.logFileName=access.log weblogic.httpd.logFileFormat=extended #recommended weblogic.httpd.logRotationType=date weblogic.httpd.logRotationPeriodMins=1440 weblogic.httpd.logRotationBeginTime=11-24-2000-00:00:00 Go-Faster Consultancy Ltd.
Sample WebLogic Access Log #Version: 1.0 #Fields: date time time-taken bytes c-ip c-dns cs-method sc-status cs-uri-stem cs-uri-query 2002-03-15 21:51:03 0.981 8057 10.0.0.2 go-faster-1 GET 200 /servlets/iclientservlet/peoplesoft8/signon.html - 2002-03-15 21:51:03 0.11 2103 10.0.0.2 go-faster-1 GET 200 /peoplesoft8/signin.css - 2002-03-15 21:51:04 0.05 1186 10.0.0.2 go-faster-1 GET 200 /peoplesoft8/images/secondaryPSFTLogo.gif - 2002-03-15 21:51:10 3.414 8101 10.0.0.2 go-faster-1 POST 200 /servlets/iclientservlet/peoplesoft8/ ICType=Panel&Menu=ADMINISTER_WORKFORCE_(GBL)&Market=GBL&PanelGroupName=PERSONAL_DATA&RL=&target=main2 Go-Faster Consultancy Ltd.
Apache Access Log #httpd.conf LogFormat "%{%Y.%m.%d %H:%M:%S}t|%T|%B|%u|%h|%{User-Agent}i|%>s|%m|%U|%q" monitoring CustomLog logs/access.log monitoring Go-Faster Consultancy Ltd.
Sample Apache Access Log 2002.02.26 09:57:06|0|275|-|127.0.0.1|Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Q312461)|200|GET|/peoplesoft8/cache/PT_NEXTTAB_ENG_1.gif| 2002.02.26 09:57:17|0|31847|-|127.0.0.1|Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Q312461)|200|POST|/servlets/iclientservlet|?ICType=Panel&Menu=ADMINISTER_WORKFORCE_(GBL)&Market=GBL&PanelGroupName=JOB_DATA 2002.02.26 09:57:26|0|30959|-|127.0.0.1|Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Q312461)|200|POST|/servlets/iclientservlet|?ICType=Panel&Menu=ADMINISTER_WORKFORCE_(GBL)&Market=GBL&PanelGroupName=JOB_DATA Go-Faster Consultancy Ltd.
What does the access log tell us? • A date and time for every request • Who made the request? • How long did it take to serve? • What was requested? • Which component (panel group)? • Indicate possible bandwidth problems. Go-Faster Consultancy Ltd.
Download Bandwidth #version: 1.0 #Fields: date time time-taken bytes c-ip c-dns cs-method sc-status cs-uri-stem cs-uri-query 2002-09-10 11:29:52 0.359 15832 141.93.249.74 141.93.249.74 GET 200 /XXXXXXXX/images/topPSFTLogo.gif - Go-Faster Consultancy Ltd.
Java Servlet (presentation logic) Tuxedo Application Server (application logic) DBMS (application data & meta-data http / https Tuxedo Message SQL Sources of Metrics Tuxedo Service Trace Go-Faster Consultancy Ltd.
BEA/Tuxedo Service Trace • Edit psappsrv.ubx file • CLOPT=“-r -e APPQ.stderr…” • -r enables trace to stderr file • -e qualifies name of stderr file Go-Faster Consultancy Ltd.
Edit psappsrv.cfg • Add a new variable to the trace section • Thus service trace can be enabled/disabled from configuration file • New variable will appear in the configuration dialogue [Trace] TuxedoServiceTrace=-r Go-Faster Consultancy Ltd.
Edit psappsrv.ubx • Change every CLOPT entry in psappsrv.ubx. • File name is set manually. • Same as queue name RQADDR="APPQ" CLOPT="{$Trace\TuxedoServiceTrace} -e {LOGDIR}{FS}APPQ.stderr {$PSAPPSRV\Spawn Server} -s@..{FS}psappsrv.lst -- -C {CFGFILE} -D {$Domain Settings\Domain ID} -S PSAPPSRV" Go-Faster Consultancy Ltd.
Resultant psappsrv.ubb • When domain is configured .ubb file is generated. All variables are fully resolved. CLOPT=”-r -e D:\ps\hr8\appserv\hr8d\LOGS\APPQ.stderr -p 1,600:3,10 -s@..\psappsrv.lst -s@..\psqcksrv.lst -sICQuery -sSqlQuery:SqlRequest -- -C psappsrv.cfg -D HR8D -S PSAPPSRV" Go-Faster Consultancy Ltd.
Sample Tuxedo Service Trace • Each service called to domain is logged SERVICE PID SDATE STIME EDATE ETIME ------- --- ----- ----- ----- ----- @ICScript 390 1014717320 4586495 1014717322 4588397 @ICScript 390 1014717322 4588467 1014717322 4588537 @ICScript 390 1014717322 4588557 1014717323 4589259 @ICScript 390 1014717332 4598021 1014717332 4598171 @ICScript 390 1014717355 4621745 1014717361 4627554 @ICScript 390 1014717365 4631329 1014717366 4632511 @ICScript 390 1014717375 4641944 1014717377 4643306 @ICPanel 390 1014717380 4646761 1014717381 4647783 @ICPanel 390 1014717388 4654202 1014717391 4657567 @ICPanel 390 1014717396 4662614 1014717401 4667271 Go-Faster Consultancy Ltd.
Tuxedo Service Trace • Service - Name of Service • Pid - An OS PID of listener or handler • Sdate/Edate - Start/End Date • Seconds since 00:00hrs GMT 1st January 1970. • Stime/Etime • Start/End Time in OS time units (1/ 100ths or 1/1000ths) • Hence calculate accurate service duration • Time on Server - not queuing Go-Faster Consultancy Ltd.
Txrpt Utility SERVICE SUMMARY REPORT SVCNAME 9a-10a TOTALS Num/Avg Num/Avg --------------- -------- ------- ICScript 8/1.43 8/1.43 ICPanel 6/3.47 6/3.47 JavaMgrGetObj 1/0.04 1/0.04 GetCertificate 1/0.73 1/0.73 --------------- ------- ------- TOTALS 16/2.06 16/2.06 Go-Faster Consultancy Ltd.
Java Servlet (presentation logic) Tuxedo Application Server (application logic) DBMS (application data & meta-data http / https Tuxedo Message SQL Sources of Metrics Tuxedo tmadmin script Go-Faster Consultancy Ltd.
BEA/Tuxedo TMADMIN Utility • BEA Interactive command line interface • Can be included in scripts • -r for read only in monitoring scripts • pq (printqueue) • psr (printserver) • pclt (printclient) Go-Faster Consultancy Ltd.
PQ • What queues in domain • How many servers on queue • How many requests queued • (How much work on queue) Prog Name Queue Name # Serve Wk Queued # Queued Ave. Len Machine --------- ------------------- --------- -------- -------- ------- JSL.exe 00095.00200 1 - 0 - GO-FASTER+ JREPSVR.exe 00094.00250 1 - 0 - GO-FASTER+ PSSAMSRV.exe SAMQ 1 - 0 - GO-FASTER+ BBL.exe 54455 1 - 0 - GO-FASTER+ WSL.exe 00001.00020 1 - 0 - GO-FASTER+ PSAPPSRV.exe APPQ 1 - 0 - GO-FASTER+ Go-Faster Consultancy Ltd.
PSR • What servers exist? • How many requests have they handled? • What are they doing right now? Prog Name Queue Name Grp Name ID RqDone Load Done Current Service --------- ---------- -------- -- ------ --------- --------------- BBL.exe 54455 GO-FAST+ 0 3 150 ( IDLE ) PSAPPSRV.exe APPQ APPSRV 1 9 450 ICPanel PSSAMSRV.exe SAMQ APPSRV 100 0 0 ( IDLE ) WSL.exe 00001.00020 BASE 20 0 0 ( IDLE ) JSL.exe 00095.00200 JSLGRP 200 0 0 ( IDLE ) JREPSVR.exe 00094.00250 JREPGRP 250 5 250 ( IDLE ) Go-Faster Consultancy Ltd.
PCLT • Who is connected? • Are they doing something? • When did they connect? LMID User Name Client Name Time Status Bgn/Cmmt/Abrt --------------- --------------- --------------- -------- ------- ------------- GO-FASTER-1 NT WSH 0:21:14 IDLE 0/0/0 GO-FASTER-1 NT JSH 0:21:13 IDLE 0/0/0 GO-FASTER-1 NT tmadmin 0:00:00 IDLE 0/0/0 GO-FASTER-1 PS JavaClient 0:02:30 BUSY/W 0/0/0 Go-Faster Consultancy Ltd.
Java Servlet (presentation logic) Tuxedo Application Server (application logic) DBMS (application data & meta-data http / https Tuxedo Message SQL Sources of Metrics Oracle SQL*Trace Go-Faster Consultancy Ltd.
Oracle SQL Trace • Tuning tool • Session trace • Format it with TKPROF • Find long running SQL statements • Total SQL Time for session call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Parse 72289 173.06 175.45 7 14654 94 0 Execute 77140 64.02 67.37 205495 766343 170611 78630 Fetch 50599 33.20 34.28 17302 971624 80146 143218 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 200028 270.28 277.10 222804 1752621 250851 221848 Go-Faster Consultancy Ltd.
Small Application Servers • Create small application with just a single handler process • Enable trace on that process • Trace sample transaction (only one user) sys.dbms_system.set_sql_trace_in_session(<sid>,<serial>,TRUE); Go-Faster Consultancy Ltd.
Java Servlet (presentation logic) Tuxedo Application Server (application logic) DBMS (application data & meta-data http / https Tuxedo Message SQL Sources of Metrics Proxy Server Access Log Tuxedo tmadmin script Tuxedo Service Trace Oracle SQL*Trace Webserver Access Log Go-Faster Consultancy Ltd.
Graphical Analysis • Don’t drown in numbers • Draw a picture • Load Data into Database • SQL_Loader (Oracle 9i External Table) • Process Data • Matching, Aggregation • Oracle Analytic Function • Load data into Excel • Draw Graph Go-Faster Consultancy Ltd.
Sample SQL*Loader File LOAD DATA INFILE 'APPQ.stderr' REPLACE INTO TABLE txrpt WHEN (1) = '@' FIELDS TERMINATED BY WHITESPACE TRAILING NULLCOLS (service "substr(:service,2)" -- remove leading @ ,pid ,stimestamp ":stimestamp/86400+2/24+TO_DATE('01011970','DDMMYYYY')" ,stime ":stime/100" ,queue ”’APPQ’" ,etime ":etime/100") Go-Faster Consultancy Ltd.
Tips about Graphs • Connect Excel directly to a database • Aggregate data carefully • Too little and excel draws it badly because it adds a black border around an area • Too much and you lose details • Pivot tables are useful • but Excel97 buggy • Scatters and Trendlines • joining dots can obscure data • Excel supports 32000 points per series Go-Faster Consultancy Ltd.
Example: CPU Utilisation Go-Faster Consultancy Ltd.
Example: Apache Servlet Times Go-Faster Consultancy Ltd.
Longest http conversations Go-Faster Consultancy Ltd.
Example: Tuxedo Service Time Go-Faster Consultancy Ltd.
tmadmin - PCLT - Connect Client Sessions Go-Faster Consultancy Ltd.
tmadmin - PCLT - Busy/Wait Users Go-Faster Consultancy Ltd.
tmadmin - PQ - Queue Length Go-Faster Consultancy Ltd.