Best Practices in Batch Tuning Empowering IT Staff to Reduce CPU and Run-Time in the Enterprise

1. Best Practices in Batch TuningEmpowering IT Staff to Reduce CPU and Run-Time in the Enterprise

2. Agenda • Did You Know • Presentation Goal • Importance of Batch Tuning • Common Problems Encountered • Solving the Problem • Remediation Options • Summary

3. Are you taking any steps to retain IT pros who are nearing traditional retirement age? The Center on Aging & Work / Workplace Flexibility has used U.S. Census data to calculate that an average of 4.6 adults will turn 65 each minute in 2007. In 2025, an average of 8.0 adults will turn 65 each minute. (McNamara, 2006). No steps taken 60% Flexible schedules 19% Part-time work options 12% Contracting retired employees 9% Increased compensation 9% In 2012, there could be as many as 21 million vacant jobs, but just 17 million workers to fill those posts. Computing Technology Industry Association Inc. Training or professional 9% development programs Bonuses for staying on 8% Job sharing 5% Delayed retirement plans 4% Other 4% Source: ComputerWorld Vital Signs survey of 233 IT professional, May-June 2007. Multiple responses allowed. Did You Know! Where have all the Baby Boomers gone!

4. Ford –Launched project Everest to replace mainframe-based Ford Supplier Network procurement system with new ERP-based system. After 4 years and multiple problems, Everest is scrapped. Cost: $400M Did You Know! Who Said the Mainframe is DEAD! Ford fails to climb Everest Innovate eats McDonalds’ lunch • McDonald’s -Launched project Innovate to replace multiple custom-built applications (including mainframe accounting systems) with global, ERP-based platform. After a year of R&D and spiraling costs, Innovate is killed. Cost: $170M http://www.roughtype.com/archives/2006/01/down_the_drain.php

5. A recent survey* in the UK showed that among 50 mid-to-large size enterprises using mainframes… 93% still depend on the mainframe to run mission-critical applications 50% say it is too risky to migrate off mainframe 28% tried and failed to migrate off the mainframe 12% had limited success with a migration off the mainframe 8% had tried and succeeded with a migration off the mainframe *Source: Mainframe still reufusing to die. Survey, Ireland News, 2/28/07 Did You Know! Who Said the Mainframe is DEAD!

6. Overall Presentation Goal Explore the key factors that often limit the focus on batch tuning, and discuss the technology framework and methods that effectively assist performance and capacity staff as well as empower other key staff members in the identification and resolution of batch tuning opportunities. Discuss methods for identifying the major sources of batch inefficiencies that recover the greatest batch elapsed time, CPU and I/O savings quickly, and with minimum effort.

7. Why Is Batch Tuning Still Important? • Corporate growth and new business needs have caused the introduction of complex applications • Most Data Centers are operating 24x7 with online availability nearly 100%. To accommodate this, the Batch window process must be reduced • More than 50% of critical data processing is done via the batch process • Batch processing continues to grow while batch windows continue to shrink

8. Common Problems of Batch Tuning Practice? • New forms of z/OS transactions have lead the performance analyst to concentrate efforts on WLM • Lack of efficient JCL coding practices and Standards Enforcement • Serial design of most batch jobs, along with I/O intensity, significantly contributes to the elapsed runtime of batch jobs

9. Common Problems of Batch Tuning Practice? • Existing Performance Analysis Tools: • Can be extremely complex to Use • Complicate data analysis with a multiplicity of reports/options/software • Fail to consider batch workload characterization/profiling • Ignore JCL construct factors • The Cost of Remediation • Batch jobs, although operating correctly, have not benefited from advanced technology, coding practices or processing methods. • The typical thinking is: “If it ain't broke, don't fix it”!.

10. DATA IN MEMORY (DIM) OPTIMIZE I/O VIO Block sizes (connect time) Hiperbatch Use cache where possible BatchPipes Level of granularity of serialization (Reduce ENQ time) VLF, modules and Execs Buffering Data Base Buffers Sort Parameters (DFSORT Hipersorting) VSAM Index and/or Data, BLSR SYTEM & TECHNOLOGY Exploit new addressing scheme 64 bits Workload Manager DASD Technology (ESCON Channels, Stripping) Coupling Facility (JES) TMM DB2 & IMS Subsystems Batch Tuning – Areas to Investigate OTHERS Data not used Excessive backups Parallelism (Job Split) Redundant Sort Job classes scheme Operator involvement (Automation) Job Priority

11. Iteration Goal Metrics Select Workload Analyze Imp. Changes Results Common Problems of Batch Tuning Practice? • Unsystematic Approach • Inaccurate Metrics • Unrepresentative Workload • Inappropriate Evaluation Technique • Ignoring Significant JCL and JOBS Design Factors • Improper presentation of results. 1 2 3 4 5 6

12. The Challenge Iteration Goal Metrics Select Workload Analyze Imp. Changes Results How to Identify specific opportunities by other staff members, e.g. Development, Operations, Production Control How to minimize data correlation and analysis How to Model the Batch Timeline and Workload Characterization How to streamline the remediation phase, minimizing cost and maximizing payoff

13. System wide focus of performance analysis • Applications complexities • Performance analysis is a highly iterative process and it is difficult to implement a practice focused on batch workload • Problem data for the batch job population is practically impossible for a specialist to gather and maintain • Users and application support rely on infrastructure for performance studies JCL Inefficiencies Copy Syndrome Deployment Cycle Target System Traditional Perf. Analysis Perf. & Capacity Planning Analysts Developers Production Control Sys. Programmers Batch Performance Analysis Dimensions Batch Knowledgebase

14. A Batch Performance tool should provide: • Conservative savings in the 15% to 20% range in CPU MIPS and elapsed runtime. Large organizations should see millions in savings and a return on investment measured in months.     • Target is Application/System based versus Source code based.  This means lower complexity and fast time to value. • Knowledge Base should provide the combined expertise of a Capacity Analyst and a Production Control Analyst. • Recommendations should be precise, prioritized, and easy to understand.

15. A Batch Performance tool should provide: • Majority of changes are JCL related and they typically follow the 80/20 rule.  80% or more of the benefits are seen in 20% or less of the changes. • Deliverables should not only facilitate fine tuning of what needs to run but help identify what can be eliminated.  The savings in eliminating redundant sorts, or the creation of files that are not used, is a 100% efficiency gain for those components eliminated! • Should not be overhead intensive.  Analysis should be based on history. • The approach should be proactive and embrace a strategy to preserve the integrity of the changes.  This is much more resource efficient than automating the same change every time an inefficient component is executed.

16. CPU Savings • Jobs Sorting Already-Sorted Data • Jobs Performing Unnecessary Passes Over Data • Jobs Executing Slow Utilities • Jobs with Inefficient DFDSS/FDR Executions • Jobs with Inefficient DB2 Utility Execution • Jobs Failing During Execution • Unnecessary Passes Over Data

17. CPU Savings • Data Created But Not Referenced • Data Sets with Inefficient Block-Size • VSAM Inefficiency Matrix • VSAM Inefficient NSR Buffering • VSAM Candidates for Batch LSR • DFSMS-compressed Data Set Inefficiency Matrix • DFSMShsm Data Set Migration Thrashing • Inefficient Cobol Compile Options

18. Elapsed Time Savings • Jobs with Allocation Delays • Jobs Waiting for Initiator • Jobs with Excessive Program Run-Time • Jobs Delayed Prior to Init. Selection Phase • Data Set with ENQ Conflicts • Sort Inefficiencies Matrix

19. DISK Space Savings • Data Sets with Inefficient Block-Size • Over-Allocated DASD Data Sets • Data Created But Not Used • VSAM Inefficiency Matrix

20. Workload Selection Workload Selection– Network Flowchart and Critical Path Analysis in a Batch Study A critical Path is that sequence of activities (Jobs) that determine the total elapsed time for a given batch workload to reach a defined point. It identifies which of the activities in a batch workload govern the timing of work during the batch Window.

21. Workload Selection Workload Selection– Network Flowchart and Critical Path Analysis in a Batch Study • A systematic and reliable study requires a consistent compilation of data from comparable runs. Ex1: • Daily Jobs, plot data from a month, (20 samples) • Weekly Jobs, plot the data from three months (13 samples) • Verify if the end of month run implies extra work • Average job execution times from number of batch cycles  Watch Deviation ! • Produce exceptional reports after each batch window • Trend analysis and monthly reports • Modeling the effect of changes. Focus on new CP is path is projected to change • Identify Steps that can be moved to another Job (backups, reorgs, etc.) 1. IBM Redbook SYSTEM/390 MVS Parallel Sysplex Batch Performance

22. Workload Selection Workload Selection– Network Flowchart and Critical Path Analysis in a Batch Study

23. Workload Selection

24. Job Analysis Example #1 • The problem: • TEMPORARY DATASET CREATED BUT NOT REFERENCED AFTERWARD • Exactly where it occurs: • JOB GZDTP### • PGM=TBR017 STEP= SEL0301 PROCSTEP= • EXEC=000.11.50 CPU=0.02.49 EXCP#=353K • DD=SYSUT2 MODE=OUTPUT ACCESS=QSAM EXCP#=238K • DSN=SYS10020.T235730.RA000.ZMZL0200.R0149276 • How to correct it: • DATA IS WRITTEN INTO A TEMPORARY DATASET BUT DOES NOT SEEM TO BE USED AFTERWARDS. THE CREATION OF THE DATA CAN PROBABLY BE ELIMINATED

25. Job Analysis Example #2 • The problem: • IDCAMS REPRO EXECUTION FOLLOWED BY COMPLEMENTARY SORT • Exactly where it occurs: • JOB ZMZL0200 • PGM=IDCAMS STEP=STEP03 PROCSTEP= • EXEC=000.04.01 CPU=0.00.49 EXCP#=62K • DD=UNLOAD MODE=OUTPUT ACCESS=QSAM DSN=DM.MPROD.DATA33 • EXCP#=31k • PGM=ICEMAN STEP=SM03 PROCSTEP= • EXEC=000.02.09 CPU=0.00.27 EXCO#=69K • DD=SORTING MODE: INPUT ACCESS=BSAMD DSN=DM.MPROD.DATA33 • EXCP#=31K • How to correct it: • A SINGLE SORT STEP CAN USUALLY REPLACE THE TWO STEPS OF IDCAMS REPRO AND SORT. IMPLEMENTING THE SINGLE SORT STEP WILL PROVIDE SIGNIFICANT SAVINGS IN ELAPSED TIME, CPU TIME, AND I/O ACTIVITY

26. What About DB2? • Improve DB2 Application Efficiency • Increase DB2 Parallelism • Improve Efficiency of DB2 Housekeeping • Eliminate DB2 Contentions

27. DB2 Analysis Example #1 • The problem: • A READ-ONLY DB2 THREAD DOES NOT EXPLOIT DB2 QUERY PARALLELISM • Exactly where it occurs: • JOB=MDIKIG04 STEP=STEP020 PROCSTEP= PGM=IKJEFT1B • DB2-ID=DBP2 PLAN=MDPPROD TYPE=TSO-BACKGROUND • PACKAGE: VKB_DVP2.COLPROD.MDIKIG • START-OF-THREAD: 2010/01/18 20.28.09 • END-OF-THREAD: 2010/01/18 21.16.10 • How to correct it: • THE APPLICATION PERFORMS EXTENSIVE READ-ONLY OPERATIONS, BUT DB2 HAS NOT USED THE OPTION TO PROCESS THE QUERY IN PARALLEL. PARALLEL DB2 OPERATIONS CAN SIGNIFICANTLY REDUCE THE ELAPSED TIME OF EXTENSIVE QUERIES. IF THIS IS AN IMPORTANT THREAD THAT MUST RUN FASTER, CONSIDER ENABLING DB2 PARALLELISM FOR THIS APPLICATION.

28. DB2 Analysis Example #2 • The problem: • A DB2 THREAD HAS ENCOUNTERED INTERNAL LIMITATIONS DURING RID LIST PROCESSING. • Exactly where it occurs: • JOB=MDIKIG04 STEP=STEP020 PROCSTEP= PGM=IKJEFT1B • DB2-ID=DBP2 PLAN=MDPPROD TYPE=TSO-BACKGROUND • PACKAGE: VKB_DVP2.COLPROD.MDIKIG • START-OF-THREAD: 2010/01/12 19.08.02 • END-OF-THREAD: 2010/01/13 22.11.10 • RID LIST PROCESSING: #SUCCESSES=12 #FAILURES= 237

29. DB2 Analysis Example #2 • How to correct it: • RID LIST PROCESSING CAN HELP REDUCE I/O RESOURCE CONSUMPTION AND ELAPSED TIME. HOWEVER, IF THERE IS NOT ENOUGH RID LIST STORAGE, RESOURCES ARE ACTUALLY WASTED (THE APPLICATION HAS ALREADY CONSUMED RESOURCES AND, UPON FAILURE, MUST REVERT TO A TABLE SPACE SCAN.) • IT IS SUGGESTED THAT THE DB2 ADMINISTRATOR CONSIDER THE FOLLOWING:  • 1. IF THE PACKAGE WAS BOUND WITH A SMALL AMOUNT OF DATA: THE ACCESS PATH THAT WAS CHOSEN MIGHT BE INEFFICIENT AND MIGHT HAVE CAUSED THE FAILURE. RUNSTATS AND REBIND MAY SOLVE THE PROBLEM. • 2. CHANGING THE APPLICATION PATH TO INHIBIT RID LIST PROCESSING (LIST PREFETCH ACCESS PATH).  • 3. REVIEWING THE APPLICATION LOGIC.

30. Excel SUBGAF Hlq.. REP%ID%.D%DATE%.T%TIME% My Computer (.TXT file) JOBHIST Graphical Analysis Facility (GAF) A PC-based Graphical Analysis Facility (GAF) provides a set of pre-defined Excel spreadsheets of some Global reports

31. GAF - Examples

32. Remediation Techniques and Options (EXAMPLE) Reduce MIPS Consumed Lower HW/SW Cost Success Factors Cost Factors

33. Remediation Techniques and Options • Implement gradual changes using a systematic approach (minimize the level of risk) • Consider JCL refurbishing (< cost) • Focus improvement on the Jobs in the Critical Path • Fine-tune the base model • Coordinate the implementation of changes • Verify results (adjust base line), start again

34. Summary • Tuning the Batch environment can result in significant CPU and elapsed time savings • The traditional approach to batch tuning makes it difficult to identify batch application inefficiencies, hinders participation by application development teams, and limits the effectiveness of results • Identifies and deciphers the batch inefficiencies so that application development and support personnel are empowered to locate and correct batch issues.

35. Interested in learning more: • Schedule an on-site meeting and live demonstration • for your entire IT staff • FOR MORE INFORMATION PLEASE CONTACT: • E-mail: sea@seasoft.com • Tel: 800.272.7322 • Web: www.seasoft.com