Packaged Application Tuning

1. Packaged Application Tuning David Kurtz Go-Faster Consultancy Ltd. david.kurtz@go-faster.co.uk www.go-faster.co.uk There are many packaged applications on the market today. Some of them (SAP, PeopleSoft, Siebel etc) are designed to run on any database platform. I've head salesmen call this 'Platform Agnostic'. Other people think that this leads to equally bad performance on all platforms! The application, if it can be customised at all, my be hidden away in an vendor specific design tool, the SQL will conform to a lowest-common-demoninator standard (no outer-joins, no UNION ALL, no special features that Oracle developed to improve performance). If you capture a piece of SQL from a trace, it may not be coded explicity, but generated by some part of the application from meta-code. Even if it is explicitly code it can be extremely difficult for a developer to find. Even adding an index is a form of application upgrade when the application maintains its own version of the database catalogue. As a DBA working with PeopleSoft this is the challenge that I face daily. If this sounds familiar to you come and listen this presentation. I will discuss the method and techniques that I use to monitor, trace and then improve performance. The key to solving this problem lies in combining database knowledge with application technical knowledge. There are many packaged applications on the market today. Some of them (SAP, PeopleSoft, Siebel etc) are designed to run on any database platform. I've head salesmen call this 'Platform Agnostic'. Other people think that this leads to equally bad performance on all platforms! The application, if it can be customised at all, my be hidden away in an vendor specific design tool, the SQL will conform to a lowest-common-demoninator standard (no outer-joins, no UNION ALL, no special features that Oracle developed to improve performance). If you capture a piece of SQL from a trace, it may not be coded explicity, but generated by some part of the application from meta-code. Even if it is explicitly code it can be extremely difficult for a developer to find. Even adding an index is a form of application upgrade when the application maintains its own version of the database catalogue. As a DBA working with PeopleSoft this is the challenge that I face daily. If this sounds familiar to you come and listen this presentation. I will discuss the method and techniques that I use to monitor, trace and then improve performance. The key to solving this problem lies in combining database knowledge with application technical knowledge.

2. Packaged Application Tuning Who am I? What is Tuning? What is a Packaged Application? What techniques could you use? Experimental Session Are you, the membership, interested in discussing Application specific issues? Little bit of theory Brief demonstration The rest of it is composed of war stories I’ll talk about the system in which they were used What it brought us and What we the problemsLittle bit of theory Brief demonstration The rest of it is composed of war stories I’ll talk about the system in which they were used What it brought us and What we the problems

3. Rules of Engagement If you can’t hear me - say so now! Feel free to ask on-topic questions during the presentation. This presentation (with notes) is available on the conference website (after the conference) and on www.go-faster.co.uk

4. Who am I? DBA Independent consultant Performance tuning PeopleSoft UKOUG Unix SIG Chair If you’ve been to these gatherings you will have seen me before Nowadays I am an independent consultant, but I still specialise in performance tuning I specialise in PeopleSoft ERP systems. So I am going to use PeopleSoft as the basis of my examples. purely because that is what I know about.If you’ve been to these gatherings you will have seen me before Nowadays I am an independent consultant, but I still specialise in performance tuning I specialise in PeopleSoft ERP systems. So I am going to use PeopleSoft as the basis of my examples. purely because that is what I know about.

5. Aphorism Performance is exactly what the user perceives it to be. No more, no less. I have realised that over the years that I have collected a number of catchphrases and quotes. When you are tackling performance problems you need to keep an eye on what you are trying to achieve. Often a user will report that the system is slow, so you have measure what the use is experiencing The availability and response time that a system produces are sum of all the components in the chain. Database - Application Server - Network - PC It takes time for requests to be transmitted over a network, through application server, be processed by the database and returned. There is a certain amount of processing on the client and that takes time too. If any component fails then the system fails. If you go to a cash point and you cant draw money, you don’t care whether the central mainframe is down, or the network, or the cash point itself is down, or even if it have run out of money. The bottom line is that the system is not available, and you don’t get your money. The users can only observe one thing. Their own PC One of the problems with performance is that it can all be a bit touchy feely. Users will only ever say two things: 1) The system is slow 2) The system is OK You need some sort of metric. Performance Level Agreements as a part of the SLA. The only thing that means anything to a user is time. How long do they spend waiting for the hourglass? Perhaps it also means that there is another component of your system to tune - your usersI have realised that over the years that I have collected a number of catchphrases and quotes. When you are tackling performance problems you need to keep an eye on what you are trying to achieve. Often a user will report that the system is slow, so you have measure what the use is experiencing The availability and response time that a system produces are sum of all the components in the chain. Database - Application Server - Network - PC It takes time for requests to be transmitted over a network, through application server, be processed by the database and returned. There is a certain amount of processing on the client and that takes time too. If any component fails then the system fails. If you go to a cash point and you cant draw money, you don’t care whether the central mainframe is down, or the network, or the cash point itself is down, or even if it have run out of money. The bottom line is that the system is not available, and you don’t get your money. The users can only observe one thing. Their own PC One of the problems with performance is that it can all be a bit touchy feely. Users will only ever say two things: 1) The system is slow 2) The system is OK You need some sort of metric. Performance Level Agreements as a part of the SLA. The only thing that means anything to a user is time. How long do they spend waiting for the hourglass? Perhaps it also means that there is another component of your system to tune - your users

6. Aphorism Performance Tuning is a search for lost time. There is only one currency here. Time Where are you loosing how much of your time? There is only one currency here. Time Where are you loosing how much of your time?

7. Aphorism When you have eliminated the impossible, whatever remains, however improbable, must be the truth. A Study in Scarlet, Arthur Conan-Doyle In order to improve the performance of a system (and by system I mean anything not just a computer system) you have to understand how it work. Answering that questions is not trivial. Because when you do the analysis properly and you may get an unexpected answer. The answer might be counter-intuitive, it may be so peculiar that you might have rejected it out of hand. This is experimental science In order to improve the performance of a system (and by system I mean anything not just a computer system) you have to understand how it work. Answering that questions is not trivial. Because when you do the analysis properly and you may get an unexpected answer. The answer might be counter-intuitive, it may be so peculiar that you might have rejected it out of hand. This is experimental science

8. Aphorism Detection is, or ought to be, an exact science. It should be be treated in the same cold and unemotional manner. The Sign of Four, Arthur Conan-Doyle I’ve had a presentation will all this Sherlock Holmes stuff for several years. And I put some of it into this presentation a long time ago now. In October 2002 the Royal Society of Chemistry (who are a serious bunch of scientists) very publicly awarded a gold medal to Holmes (and not his creator Conan-Doyle) because of his contributions to science. Why? Obviously it was a publicity stunt. Done because the character of Holmes is a positive scientific role model. The myth embodies a rigorous, logical, scientific techniques for investigation. So there is a serious point here, and I am not simply padding out this presentation. I’ve had a presentation will all this Sherlock Holmes stuff for several years. And I put some of it into this presentation a long time ago now. In October 2002 the Royal Society of Chemistry (who are a serious bunch of scientists) very publicly awarded a gold medal to Holmes (and not his creator Conan-Doyle) because of his contributions to science. Why? Obviously it was a publicity stunt. Done because the character of Holmes is a positive scientific role model. The myth embodies a rigorous, logical, scientific techniques for investigation. So there is a serious point here, and I am not simply padding out this presentation.

9. What is a Packaged Application Bought in from a software vendor. You didn’t have any say in how it was developed. You cannot (easily) change the code. You may not even be able to see the code. It may be designed to run on other databases as well as Oracle. Too often a packaged application is a black box. I am not here selling a product. I am not even here selling consultancy (well maybe a little bit). I am selling an idea. The idea is that if you have something that you don’t fully understand, then take some time to probe it. Conduct experiments, take measurements, and conjecture how a system works. Too often a packaged application is a black box. I am not here selling a product. I am not even here selling consultancy (well maybe a little bit). I am selling an idea. The idea is that if you have something that you don’t fully understand, then take some time to probe it. Conduct experiments, take measurements, and conjecture how a system works.

10. What Packaged Applications are there? ERP Oracle, SAP, PeopleSoft Baan, J D Edwards ... CRM and Siebel What others? I would say that these are the most common packaged applications.I would say that these are the most common packaged applications.

11. It isn’t always the database! Understand architecture Web front end Network Application server Database What can you measure?

12. PeopleSoft Internet Architecture PIA is 4-tier architecture Each component can contribute to response time Need to measure each component This is where I start to use PeopleSoft specific examples. In the beginning PeopleSoft was a two-tier system. A windows client interacted directly with the database. Today on the application designer and data mover still do that. In 1997 PeopleTools 7.0 introduced a full three-tier windows client that used the Tuxedo Application server. In 2000 PeopleTools 8.x appeared with a fully internet client. The database still stores the application and the application server is responsible for extracting the relevant application objects from the database as well as the application data. That hans’t really changed. The application server bundles the component (the thing we used to call the panel group) will all the html, JavaScript fragments and graphics and send that to the servlet. The servlet, which is closely bound with the webserver unpacks that message, sends the page to the browser and rest of those objects are loaded by the browser from the webserver. The Servlet also holds session information for each connected user. There is a lot of work going on at different tiers in the chain. Depending on circumstances different components can contribute differently to response times. Our objective is to measure all of this. (see www.go-faster.co.uk for detailed PIA/Tuxedo architectural overview) This is where I start to use PeopleSoft specific examples. In the beginning PeopleSoft was a two-tier system. A windows client interacted directly with the database. Today on the application designer and data mover still do that. In 1997 PeopleTools 7.0 introduced a full three-tier windows client that used the Tuxedo Application server. In 2000 PeopleTools 8.x appeared with a fully internet client. The database still stores the application and the application server is responsible for extracting the relevant application objects from the database as well as the application data. That hans’t really changed. The application server bundles the component (the thing we used to call the panel group) will all the html, JavaScript fragments and graphics and send that to the servlet. The servlet, which is closely bound with the webserver unpacks that message, sends the page to the browser and rest of those objects are loaded by the browser from the webserver. The Servlet also holds session information for each connected user. There is a lot of work going on at different tiers in the chain. Depending on circumstances different components can contribute differently to response times. Our objective is to measure all of this. (see www.go-faster.co.uk for detailed PIA/Tuxedo architectural overview)

13. Sources of Metrics Browser Proxy Server 3rd Party software Local agent on PC Web Server Access Log There are lots of things to measure in the PeopleTools 8. Each tier in the diagram on the pervious slide produces its own set of metrics. The key is to relate different sets. In the light of what I have just said about perception, the most important place to measure is as close to the client as possible. That can be difficult. There is 3rd party software that will put agents on the clients. Page detailer is a part of the Websphere suite. One technique that I have used is to introduce a proxy server. Most webservers can be configured as proxy servers. So I put a proxy server physically close to the clients that I want to measure. Similarly the web server can also record all the same requests but now their duration will be the time at the webserver. This time that the request took to actually process. The difference in these duration is the time lost across the network. There are lots of things to measure in the PeopleTools 8. Each tier in the diagram on the pervious slide produces its own set of metrics. The key is to relate different sets. In the light of what I have just said about perception, the most important place to measure is as close to the client as possible. That can be difficult. There is 3rd party software that will put agents on the clients. Page detailer is a part of the Websphere suite. One technique that I have used is to introduce a proxy server. Most webservers can be configured as proxy servers. So I put a proxy server physically close to the clients that I want to measure. Similarly the web server can also record all the same requests but now their duration will be the time at the webserver. This time that the request took to actually process. The difference in these duration is the time lost across the network.

14. Sources of Metrics (2) BEA/Tuxedo Service Trace TMADMIN Oracle Database SQL_Trace PSFT Application Server Trace

15. Sources of Metrics So now you can use pairs of measures to measure the segment between the points of measurement The difference between proxy server and web server is network time The difference between Webserver log and Tuxedo Log is servlet time + tuxedo queue time Tuxedo queuing can be estimated from Tuxedo tmadmin scripts. An average queue of 2 requests on a queue suggests a queuing time of twice the average service time Oracle trace can only be used on a statistical basis. What proportion of Tuxedo Service time is SQL Important tip - if these traces are coming off difference servers get them time synchronised at least with each other if not a proper time source. So now you can use pairs of measures to measure the segment between the points of measurement The difference between proxy server and web server is network time The difference between Webserver log and Tuxedo Log is servlet time + tuxedo queue time Tuxedo queuing can be estimated from Tuxedo tmadmin scripts. An average queue of 2 requests on a queue suggests a queuing time of twice the average service time Oracle trace can only be used on a statistical basis. What proportion of Tuxedo Service time is SQL Important tip - if these traces are coming off difference servers get them time synchronised at least with each other if not a proper time source.

16. Web/Proxy Access Log Standard format for access log www.w3.org/pub/WWW/TR/logfile.html Apache is different The access log will record all the requests including their duration as measured at the proxy server. There is a standard format for webserver logs. Weblogic conforms to this stanard, Apache does not. There are a number of things that you can records.The access log will record all the requests including their duration as measured at the proxy server. There is a standard format for webserver logs. Weblogic conforms to this stanard, Apache does not. There are a number of things that you can records.

17. Web/Proxy Access Log Fields Date, Time Time Taken accurate to OS time units (1/100ths or 1/1000ths) only 1s on Apache Message bytes IP addresses & DNS names URI stem & Query

18. Apache Access Log #httpd.conf LogFormat "%{%Y.%m.%d %H:%M:%S}t|%T|%B|%u|%h|%{User-Agent}i|%>s|%m|%U|%q" monitoring CustomLog logs/access.log monitoring But apache is no longer supported in 8.4But apache is no longer supported in 8.4

19. Sample Apache Access Log 2002.02.26 09:57:06|0|275|-|127.0.0.1|Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Q312461)|200|GET|/peoplesoft8/cache/PT_NEXTTAB_ENG_1.gif| 2002.02.26 09:57:17|0|31847|-|127.0.0.1|Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Q312461)|200|POST|/servlets/iclientservlet|?ICType=Panel&Menu=ADMINISTER_WORKFORCE_(GBL)&Market=GBL&PanelGroupName=JOB_DATA 2002.02.26 09:57:26|0|30959|-|127.0.0.1|Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Q312461)|200|POST|/servlets/iclientservlet|?ICType=Panel&Menu=ADMINISTER_WORKFORCE_(GBL)&Market=GBL&PanelGroupName=JOB_DATA Pipe symbols have been used as separators because it’s very unlikely to appear elsewhere in the trace.Pipe symbols have been used as separators because it’s very unlikely to appear elsewhere in the trace.

20. What does the access log tell us? A date and time for every request Who made the request? How long did it take to serve? What was requested? Which component (panel group)? So to summarise the enhanced access log will tell you about activity at the web or proxy server. Everything you need to know about a request.So to summarise the enhanced access log will tell you about activity at the web or proxy server. Everything you need to know about a request.

21. BEA/Tuxedo Service Trace Edit psappsrv.ubx file CLOPT=“-r -e APPQ.stderr…” -r enables trace to stderr file -e qualifies name of stderr file The Tuxedo service trace will log every service that goes via the application server. There is almost, but not quite, a one-to-one relationship between web server and app server requests. The difference between web server service time and app server service time is the time spent by the servlet on the JVM. Using SQL trace it is possible to aggregate SQL time over a period. You can determine the proportion of SQL time in Tuxedo time. The Tuxedo service trace will log every service that goes via the application server. There is almost, but not quite, a one-to-one relationship between web server and app server requests. The difference between web server service time and app server service time is the time spent by the servlet on the JVM. Using SQL trace it is possible to aggregate SQL time over a period. You can determine the proportion of SQL time in Tuxedo time.

22. Sample Tuxedo Service Trace Each service called to domain is logged SERVICE PID SDATE STIME EDATE ETIME ------- --- ----- ----- ----- ----- @ICScript 390 1014717320 4586495 1014717322 4588397 @ICScript 390 1014717322 4588467 1014717322 4588537 @ICScript 390 1014717322 4588557 1014717323 4589259 @ICScript 390 1014717332 4598021 1014717332 4598171 @ICScript 390 1014717355 4621745 1014717361 4627554 @ICScript 390 1014717365 4631329 1014717366 4632511 @ICScript 390 1014717375 4641944 1014717377 4643306 @ICPanel 390 1014717380 4646761 1014717381 4647783 @ICPanel 390 1014717388 4654202 1014717391 4657567 @ICPanel 390 1014717396 4662614 1014717401 4667271 The trace is a text file,which is easy to work with.The trace is a text file,which is easy to work with.

23. Tuxedo Service Trace Service - Name of Service Pid - An OS PID of listener or handler Sdate/Edate - Start/End Date Seconds since 00:00hrs GMT 1st January 1970. Stime/Etime Start/End Time in OS time units (1/ 100ths or 1/1000ths) Hence calculate accurate service duration Time on Server - not queuing The fields in the service traceThe fields in the service trace

24. BEA/Tuxedo TMADMIN Utility BEA Interactive command line interface Can be included in scripts -r for read only in monitoring scripts pq (printqueue) psr (printserver) pclt (printclient)

25. PQ What queues in domain How many servers on queue How many requests queued (How much work on queue) Prog Name Queue Name # Serve Wk Queued # Queued Ave. Len Machine --------- ------------------- --------- -------- -------- ------- JSL.exe 00095.00200 1 - 0 - GO-FASTER+ JREPSVR.exe 00094.00250 1 - 0 - GO-FASTER+ PSSAMSRV.exe SAMQ 1 - 0 - GO-FASTER+ BBL.exe 54455 1 - 0 - GO-FASTER+ WSL.exe 00001.00020 1 - 0 - GO-FASTER+ PSAPPSRV.exe APPQ 1 - 0 - GO-FASTER+

26. PSR What servers exist? How many requests have they handled? What are they doing right now? Prog Name Queue Name Grp Name ID RqDone Load Done Current Service --------- ---------- -------- -- ------ --------- --------------- BBL.exe 54455 GO-FAST+ 0 3 150 ( IDLE ) PSAPPSRV.exe APPQ APPSRV 1 9 450 ICPanel PSSAMSRV.exe SAMQ APPSRV 100 0 0 ( IDLE ) WSL.exe 00001.00020 BASE 20 0 0 ( IDLE ) JSL.exe 00095.00200 JSLGRP 200 0 0 ( IDLE ) JREPSVR.exe 00094.00250 JREPGRP 250 5 250 ( IDLE )

27. PCLT Who is connected? Are they doing something? When did they connect? LMID User Name Client Name Time Status Bgn/Cmmt/Abrt --------------- --------------- --------------- -------- ------- ------------- GO-FASTER-1 NT WSH 0:21:14 IDLE 0/0/0 GO-FASTER-1 NT JSH 0:21:13 IDLE 0/0/0 GO-FASTER-1 NT tmadmin 0:00:00 IDLE 0/0/0 GO-FASTER-1 PS JavaClient 0:02:30 BUSY/W 0/0/0

28. Oracle SQL Trace Tuning tool Session trace Format it with TKPROF Find long running SQL statements Total SQL Time for session call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Parse 72289 173.06 175.45 7 14654 94 0 Execute 77140 64.02 67.37 205495 766343 170611 78630 Fetch 50599 33.20 34.28 17302 971624 80146 143218 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 200028 270.28 277.10 222804 1752621 250851 221848

29. Small Application Servers Create small application with just a single handler process Enable trace on that process Trace sample transaction (only one user) sys.dbms_system.set_sql_trace_in_session(<sid>,<serial>,TRUE);

30. Sources of Metrics So now you can use pairs of measures to measure the segment between the points of measurement The difference between proxy server and web server is network time The difference between Webserver log and Tuxedo Log is servlet time + tuxedo queue time Tuxedo queuing can be estimated from Tuxedo tmadmin scripts. An average queue of 2 requests on a queue suggests a queuing time of twice the average service time Oracle trace can only be used on a statistical basis. What proportion of Tuxedo Service time is SQL Important tip - if these traces are coming off difference servers get them time synchronised at least with each other if not a proper time source. So now you can use pairs of measures to measure the segment between the points of measurement The difference between proxy server and web server is network time The difference between Webserver log and Tuxedo Log is servlet time + tuxedo queue time Tuxedo queuing can be estimated from Tuxedo tmadmin scripts. An average queue of 2 requests on a queue suggests a queuing time of twice the average service time Oracle trace can only be used on a statistical basis. What proportion of Tuxedo Service time is SQL Important tip - if these traces are coming off difference servers get them time synchronised at least with each other if not a proper time source.

31. Graphical Analysis Don’t drown in numbers Draw a picture Load Data into Database SQL_Loader (Oracle 9i External Table) Process Data Matching, Aggregation Oracle Analytic Function Load data into Excel Draw Graph My tactic for dealing with almost any log file is the same. Load it into a database, manipulate the data with SQL - you may have to match entries in different logsMy tactic for dealing with almost any log file is the same. Load it into a database, manipulate the data with SQL - you may have to match entries in different logs

32. Sample SQL*Loader File LOAD DATA INFILE 'APPQ.stderr' REPLACE INTO TABLE txrpt WHEN (1) = '@' FIELDS TERMINATED BY WHITESPACE TRAILING NULLCOLS (service "substr(:service,2)" -- remove leading @ ,pid ,stimestamp ":stimestamp/86400+2/24+TO_DATE('01011970','DDMMYYYY')" ,stime ":stime/100" ,queue ”’APPQ’" ,etime ":etime/100")

33. Tips about Graphs Connect Excel directly to a database Aggregate data carefully Too little and excel draws it badly because it adds a black border around an area Too much and you lose details Pivot tables are useful but Excel97 buggy Scatters and Trendlines joining dots can obscure data Excel supports 32000 points per series

34. Example: CPU Utilisation

35. Example: Apache Servlet Times

36. Longest http conversations

37. Example: Tuxedo Service Time

38. tmadmin - PCLT - Connect Client Sessions

39. tmadmin - PCLT - Busy/Wait Users

40. tmadmin - PQ - Queue Length

41. tmadmin - PSR - print server processes

42. Relating different metrics produces new information

43. Batch Performance Sources of Metrics Graph

44. Sources of Metrics Process Scheduler Request Table every scheduled process recorded on this table start and end time Application Traces SQL Trace

45. Graphical Analysis

46. Oracle SQL*Trace & TKPROF Enable SQL Trace on batch process Process trace file with TKPROF report Top n statements sort by Parse/Fetch/Execute CPU/Elapsed Time statement timings/execution plan call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Parse 36 0.02 0.01 0 0 0 0 Execute 36 55.09 86.80 0 2973787 3 1 Fetch 0 0.00 0.00 0 0 0 0 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 72 55.11 86.81 0 2973787 3 1

47. Triggers Process Scheduler Status Change On-Connect Enable Trace Archiving data during purge routines Altering session parameters

48. Process Scheduler CREATE OR REPLACE trigger sysadm.set_trace before update of runstatus on sysadm.psprcsrqst for each row when (new.runstatus = 7 and old.runstatus != 7 and new.prcstype IN('Application Engine','COBOL SQL', 'SQR Process','SQR Report','SQR Report For WF Delivery'))

49. On-Connect Triggers create or replace trigger sysadm.psqrysrv_session_trigger after logon on sysadm.schema declare l_program VARCHAR(64); begin SELECT program INTO l_program FROM v$session WHERE sid IN(SELECT sid FROM v$mystat WHERE rownum = 1); IF substr(l_program,1,8) = 'PSQRYSRV' OR INSTR(UPPER(l_program),'PSNVS') > 0 THEN sys.dbms_session.set_sql_trace(true); ...

50. For example… sys.dbms_session.set_sql_trace(true); ALTER SESSION SET EVENTS '10053 trace name context forever, level 1’; ALTER SESSION SET EVENTS '10128 trace name context forever, level 2’; ALTER SESSION SET optimizer_mode=FIRST_ROWS; ALTER SESSION SET optimizer_index_caching=100; ALTER SESSION SET optimizer_index_cost_adj=1; ALTER SESSION SET sort_area_size = 100000; sys.dbms_application_info.set_module('nVision','psqrysrv_session_trigger fired');

51. Aphorism The problems change from release to release, but the methods by which they are investigated remain the same. Jonathan Lewis (author of Practical Oracle 8i)

52. Questions?

53. Aphorism You know my methods, apply them. A Study in Scarlet, Arthur Conan-Doyle I will leave you with a final piece of Sherlock Holmes and my thanks.I will leave you with a final piece of Sherlock Holmes and my thanks.

54. Packaged Application Tuning David Kurtz Go-Faster Consultancy Ltd. david.kurtz@go-faster.co.uk www.go-faster.co.uk