Building an MXG PDB from NTSMF Data

1. Building an MXG PDB from NTSMF Data Chuck Hopf chuck@chopf.com www.chopf.com/geekland.shtml

2. Building an MXG PDB from NTSMF Data • ‘Classic’ MXG PDB Structure • Why it doesn’t work • What needed to change • How it is implemented for NTSMF • Using BLDNTPDB

3. Classic MXG PDB Structure • Read ALL the data and place it in a PDB with possibly CICSTRAN on the side • Run daily - daily contains detail of ALL data read • WEEKBLD consolidates dailies into weeklies • MONTHBLD consolidates weeklies into monthlies

4. Classic MXG PDB Structure • DASD storage required: • 8 daily PDB • 5 weekly PDB • enough WORK space for the largest dataset (STEPS or TYPE74)

5. Why It Doesn’t Work • 40 GB of SMF data daily • Needs 160GB of WORK space (SORTWK and WORK) • 320GB of space for daily datasets • 1+ Terror Bytes for weekly datasets • Time - Daily jobs that run for more than a day are not practical

6. What Needed to Change • Multiple Parallel Threads • Run week-to-date • Run month-to-date • Eliminate weekly/monthly jobs • Exploit summarization macros where needed to reduce volume

7. How Is It Implemented for NTSMF? • BLDNTPDB - a very large and complex collection of SAS MACROs that allows the user to choose how the PDB should be structured. • Can use the classic MXG structure or... • Can use virtually any structure the user chooses

8. Using BLDNTPDB • Parameters and Meanings • Examples of use • Future Directions • ?????????????????????

9. Using BLDNTPDB - Parameters and Meanings • NTSMF=C:\MXG\NTSMF\NTSMF.SMF - where is your RAW NTSMF data? • WEEKSTRT=MON - on what day of the week does your week start? The normal MXG default is Monday. This controls when weekly processing takes place. • RUNNTINT=YES - should we run the NTINTRV summarization?

10. Parameters and Meanings • ASUMDUR=HOUR - what level of summarization should we apply? Valid values are QTRHOUR, HALFHOUR, HOUR, SHIFT, DATE, WEEK, MONTH • KEEPERS=null - A list of datasets to be ‘kept’ (usually not used) • DROPPERS=null - A list of datasets to be ‘dropped’ (built but not kept in the output PDB.)

11. Parameters and Meanings • ZEROOBS=null - Similar to DROPPERS but datasets are created with 0 observations • MAXDRVS=26 - Maximum number of logical drives/physical drive (limits and array size.) • RERUN=null - If YES is specified forces the rewriting of an old PDB USE WITH CAUTION

12. Parameters and Meanings • MON=C:\MXG\MON • TUE=C:\MXG\TUE • WED=C:\MXG\WED • THU=C:\MXG\THU • FRI=C:\MXG\FRI • SAT=C:\MXG\SAT • SUN=C:\MXG\SUN • Dataset placement for DAILY PDBs. If a null string is used, the dataset WILL NOT BE BUILT.

13. Parameters and Meanings • WEEK=C:\MXG\WEEK • WEEK2=C:\MXG\WEEK2 • WEEK3=C:\MXG\WEEK3 • WEEK4=C:\MXG\WEEK4 • WEEK5=C:\MXG\WEEK5 • Placement for weekly datasets. If a null string is used, the dataset WILL NOT BE BUILT.

14. Parameters and Meanings • TREND=C:\MXG\TREND • MONTH=C:\MXG\MONTH • MONTH1=null • Placement for monthly and TREND datasets. A null string results in the dataset NOT BEING BUILT.

15. Parameters and Meanings • RUNDAY=YES - Run the BUILD process daily and build the specified daily databases • RUNWEEK=NO - YES means run the weekly BUILD process if it is the day specified by WEEKDATE and WTD means build WTD datasets daily • RUNMONTH=NO - YES means run the monthly BUILD process on the 2nd day of the month and MTD means build MTD datasets daily • RUNTREND=WEEKLY - Build the TREND datasets when the WEEKLY BUILD is run. If DAILY is specified, build the TREND datasets when the DAILY datasets are built.

16. Parameters and Meanings • RPTRAW=NO - Build a table of the distribution of the RAW NTSMF record types • RPTDAY=PDB - Create reports for the most recent PDB at the hourly • RPTWEEK=NO - Create reports for the WEEKLY PDB if it has just been • RPTMONTH=NO - Create reports for the MONTHLY database if it has just been created • RPTTREND=NO - Create reports for the TREND database if it has just been updated

17. Parameters and Meanings • GRAFDAY=PDB - Create reports for the most recent PDB at the hourly • GRAFWEEK=NO - Create reports for the WEEKLY PDB if it has just been • GRAFMNTH=NO - Create reports for the MONTHLY database if it has just been created • GRAFTRND=NO - Create reports for the TREND database if it has just been updated

18. Parameters and Meanings • Any combination of RUNxxxx, RPTxxxx, and GRAFxxxx specifications may be used. You can do DAILY, WEEKLY, and MONTHLY runs in a single execution as well as creating reports and graphs at each level.

19. Parameters and Meanings • Null strings passed as some parameters means ‘dont do this!’

20. Examples of Use • Classic MXG Structure • PDB for each day of week, 5 weeks, monthly with week run on Monday • %BLDNTPDB(RUNDAY=YES, RUNWEEK=YES, RUNMNTH=YES, RUNTRND=WEEKLY);

21. Examples of Use • Single daily PDB no WEEKLY/MONTHLY/TREND • %BLDNTPDB(RUNDAY=YES,MON=, TUE=, WED=, THU=, FRI=, SAT=, SUN=);

22. Examples of Use • Single daily PDB with week-to-date, month-to-date, and trending done daily • %BLDNTPDB(RUNDAY=YES, RUNWEEK=WTD,RUNMNTH=MTD, RUNTRND=DAILY,MON=,TUE=,WED=, THU=, FRI=, SAT=, SUN=);

23. A Few Words About MXG 16.04 and Later • Each dataset now has an up to 6 character mnemonic identifier. In the case of NTSMF data it starts with NT and then has 4 unique characters. • This is used to identify what datasets to KEEP DROP or set to ZEROOBS. • YOU MUST REVIEW THE 16.04 CHANGES and DOCUMENTATION!

24. Examples of Use • Build and keep only the datasets needed for NTINTRV • %BLDNTPDB(KEEPERS=NTBROW NTCACH NTFTP NTMEM NTNBTC NTNBUI NTNETS NTNWLI NTNWLN NTNWLS NTOBJ NTPAGE NTPDSK NTPROC NTPROR NTRDIR NTSERV NTSYST NTTCP);

25. Examples of Use • Build only the datasets needed for NTINTRV and DROP after NTINTRV is created: • %BLDNTPDB(DROPPERS=NTBROW NTCACH NTFTP NTMEM NTNBTC NTNBUI NTNETS NTNWLI NTNWLN NTNWLS NTOBJ NTPAGE NTPDSK NTPROC NTPROR NTRDIR NTSERV NTSYST NTTCP);

26. Examples of Use • Build the datasets needed for NTINTRV but build the PROCESS dataset with zero observations: • %BLDNTPDB(KEEPERS=NTBROW NTCACH NTFTP NTMEM NTNBTC NTNBUI NTNETS NTNWLI NTNWLN NTNWLS NTOBJ NTPAGE NTPDSK NTPROR NTRDIR NTSERV NTSYST NTTCP, ZEROOBS=NTPROC);

27. Examples of Use • Reporting can be added to any execution: • %BLDNTPDB(xxxxxxxxxxxxxxx, RPTDAY=YES);

28. Examples of Use • Reporting can be done against any PDB: • %BLDNTPDB(xxxxxxxxxxxxxxx, RPTDAY=MON);

29. Examples of Use • GRAPHS can be created for daily, weekly and trend datasets (CPU usage only at this time - more to come.) • %BLDNTPDB(xxxxxxxxxxxxx, GRAFDAY=YES, GRAFWEEK=YES);

30. Examples of Use - A Case Study Suppose you want the following: Run the PDB DAILY keeping all datasets Maximum of 5 logical drives/physical drive Start the week on Sunday Keep a WTD/MTD/and TREND datasets Run reports of Daily usage Create Graphs of Daily usage

31. Examples of Use- A Case Study %BLDNTPDB(MON=, TUE=,WED=,THU=,FRI=, SAT=,SUN=,WEEK2=,WEEK3=, WEEK4=,WEEK5=, RUNDAY=YES,WEEKSTRT=SUN, RUNWEEK=WTD,RUNMNTH=MTD, RUNTRND=DAILY,MAXDRVS=5, RPTDAY=PDB,GRAFDAY=PDB);

32. Future Directions • Largely up to YOU! • My knowledge of the data is limited • My experience with the data is limited • I need to know what YOU want/need

33. Future Directions • What reports are needed • Data sources • Data elements • Levels of summarization

34. Future Directions • What graphs are needed • What data sources • What kinds of graphs • What levels of summarization

35. Future Directions • Currently on the development list: • Selection of datasets to place in the WEEKLY/MONTHLY databases - a WEEKKEEP= operand or something along those lines • More automated summarization (this is a tough one because it supposes knowledge of the datasets and variables that is difficult to have programmatically.)

36. Future Directions • Currently on the Development List • Ability to limit variables in WTD/MTD datasets using DROP= • Ability to output reports as HTML

37. ???????????