T4 Tools Overview: Installation, Usage, and Advanced Topics

1. Getting Started With T4 Tom Cafarella OpenVMS Boot Camp September, 2011

2. Outline • T4 History • What is T4? • What are its components? • How is it distributed? • What is its support status? • Advanced Topics • How to install it • How to run it

3. T4 History • Idea conceived by Steve Lieman in 2001 • For troubleshooting customer performance problems. • Visual element missing from existing free tools • After-market products difficult to access inside HP • Prototype written in DCL by Tom Cafarella in 2001 • Most T4 original code and TLViz visualizer written by Ian Megarity. • Now maintained by Kernel Tools team and Office of Customer Programs. • See Mandar Chitale this week with questions • Used heavily by Performance Team • See Rafiq Ahamed K and Kevin Jenkins this week.

4. What is T4 ? • It’s the preferred system-level data collector for OpenVMS Engineering customer performance engagements. • And a low-overhead, basic performance tool for our customers. • Data collector only! • To visualize data, use its PC companion tool, TLViz • TLViz and Case Studies sessions after this one • It’s an “umbrella collector” • Uses other low-level data collectors

5. What is T4 ? • Runs on OpenVMS Integrity and Alpha only • Versions V7.3 through V8.4 and beyond • A single PCSI kit contains all versions • It is officially UNSUPPORTED • but many of our customers depend on it • and so do we, when performance issues arise

6. T4’s Niche • Mainly a performance troubleshooting tool • Both within OpenVMS Engineering and at our customer sites. • Run it all the time • Keep history • You never know when issues will arise • Always need to know what the baseline is • So we can see what changed. • Not designed for capacity planning • But could be used for that in a basic way.

7. T4 Data  Powerful Visual Stories Dramatic proof of higher throughput following upgrade

8. How do you run T4? • @T4$SYS:T4$CONFIG • Submits batch job that runs for one full day • … and schedules batch job for the next day • Spawns a process for each component collector • At end of day, converts data files to CSV files • CSV files packaged into few .ZIP’s for xfer to PC • Data files packaged into a separate .ZIP file • Analysis done on PC using TLViz

9. T4 Collection • Intent is to provide a low-overhead comprehensive, but not exhaustive overview of system statistics. • Comprised of several individual collectors • And your own!

10. Worrying about T4 Overhead?It’s Not a Problem T4 coexists with other high powered performance management and capacity planning tools T4 defaults  low overhead – typically less than 1/2 of 1% with 60 second sampling High resolution possible at 10 seconds or less with reasonable overhead

11. T4 Collectors MONITOR or TDC XFC (SHOW MEM / CACHE) Dedicated lock manager (Busy %, Request) TCPIP traffic system wide (pkts, mbs) Network adapter (0,1, or more) Login/Logout extraction from accounting log FCMon (if Fibrechannel disks present) VevaMon (if EVA present) SPL (Spinlock Tracing) (if T4$EXPERT non-zero)

12. T4 Components • Collectors written especially for T4 • T4$LCK73_MON.EXE • Lock Management statistics • T4$NET_MON.EXE • Data-link level network statistics • T4$XFC_MON, T4$XFC_POST and T4$XFC_STAT • XFC statistics

13. T4 Components • Collectors written especially for T4 (continued) • T4$ACLG.EXE • Accounting login statistics • T4$FC_MONITOR.EXE • Fibrechannel statistics • Runs if Fibrechannel devices present on system • T4$SPL_TL.EXE • Spinlock usage statistics • Runs when the logical name T4$EXPERT defined to any non-null value

14. T4 Components Collectors provided by other engineering groups • TCPIP$EXAMPLES:TCPIP$TCP_MON.EXE • TCP network statistics • VEVAMON.EXE • EVA performance statistics • Runs if EVA is located on the system • VEVAMON_V2_3_ALPHA.EXE and VEVAMON_V2_3_IA64.EXE

15. T4 Components • Other collectors • T4$RDB2CSV.EXE • Rdb statistics • Designed to run if Rdb present on the system • Tried it for a while, but was “not ready for prime time” • You may see the image, but it has been disabled. • T4MEM.EXE • Memory usage statistics. • Being considered for future inclusion

16. Advanced T4 Topics • Add your own collector • Use T4$EXPERT logical name • Multiple concurrent collections • Run a “one-off” collection starting NOW • Use @T4$NOW to collect today’s stats “so far” • Use T4ABORT to cleanly stop collection ahead of schedule • Keep a historical archive of collected data • Customize T4 command procedures • Use T4EXTR to convert any MONITOR file to T4 format • Collecting RMS statistics

17. T4 Style Collectors Add your own! • Easy to add your own collector to T4 and have it display along with existing stats. • Simply have it output a CSV file in “T4 style” • Slides describing this technique will be posted soon at T4 website. • Then use T4APRC utility to combine your data with other system data. • Follow T4APRC example in T4$COLLECT.COM

18. Advanced T4 Collection • Sample rate • 60 seconds default, 10 second minimum • Can go down to 1 second if T4$EXPERT logical name set to non-null value. • DEFINE /SYSTEM /EXEC T4$EXPERT 1 • Will also enable SPL (Spinlock Tracing) collection if desired.

19. Multiple T4 collections • Overlap 2 or more T4 collections • Ad hoc trouble shooting at higher resolution • Runs for short duration, • Launch only when needed • Or, zoom in on known peak period • E.g. you know the system tends to get especially busy between 9:30 and 11:30

20. Special T4 Collections • Start “NOW” & do not resubmit • Input parameters when launching T4 session • Quick response to slowdown • Running quick ad hoc Benchmarking tests • Getting the feel for T4 launch procedure • Estimating data storage requirements • Couple this with higher resolution time

21. Process Accumulated Statistics NOW • Use the @T4$NOW feature to capture stats from currently running T4 collection. • Creates a composite CSV file entitled *_xxxx.CSV, where xxxx is time of day. • T4 collection continues uninterrupted. • Lets you see what’s happened so far today. • May also want to do a MONITOR playback on the T4 MONITOR.DAT file for a quicker look. • Restriction: Only composite file created • No DISK, SCS, FCM or EVA data • Issue: @T4$SYS:T4$NOW

22. Stopping T4 Gracefully • T4ABORT (utility included in kit) • Stops collection • Produces standard CSV files • Gracefully terminates all collectors • Let’s you select which T4 sessions to stop • Stop high resolution • Leave daily history running • Issue: • $ SET COMMAND T4$SYS:T4$ABORT.CLD • $ T4ABORT /ID=pid

23. Initiating T4 Historical Archive Define system logical name T4$DATA Choose disk & directory for output files carefully – make sure sufficient disk quota Avoid using the system disk Test run to gauge size of output

24. T4$CONFIG.COM and T4$COLLECT.COM • DCL – so it’s customizable to your needs • Can comment out collectors • Can add your own integrated collectors • Response, throughput, database, … • Can adjust who receives mail • Can transfer resulting files (e.g. FTP) • Can change the MONITOR command • Change Processes list

25. RMS statistics Turn on RMS statistics for selected files MONITOR RMS Then T4EXTR /CLASS=(NOALL,RMS) Recommend keeping these separate

26. Where to get T4 and TLViz kits • OpenVMS Website: • http://h71000.www7.hp.com/openvms/products/t4/ • T443-1.zip file. It contains: • T4 V4.3-1PCSI kit • T4_README.TXT • Starting with V8.4, same files available in SYS$ETC:T4_V43_KIT.EXE • Self-extracting image file. Simply RUN it to produce above files. • Contains an extra fix to correct FCMON problem.

27. Quick “How To Install & Run T4” Guide • Obtain T4 kit • PRODUCT INSTALL T4 • DEFINE/SYS/EXEC T4$SYS SYS$SYSDEVICE:[VMS$COMMON.T4$SYS] • DEFINE/SYS/EXEC T4$DATA disk:[directory] • Not system disk! • Create batch queue • @T4$SYS:T4$CONFIG.COM • Answer prompts

28. Questions and Feedback • Thank you for your attention! • Send questions/feedback/suggestions to T4@HP.COM • Working on V4.4!