FOCUS Quality Assurance

1. FOCUS Roundtable: Advances in Advances in Advances in Advances in Advances in Advances in Advances in Advances in Advances in Advances in Advances in Advances in FOCUS Quality Assurance 10 of them stacked up. One move by the arrows every 60 seconds Joel Starkman Director of Operations June, 2008

2. Advances in FOCUS Quality AssuranceCustom-Built Testing System Envelope • Catalogs and executes all tests across the entire company • Captures results and compares to approved base results • Web-based interface to view results, drill down to code, intelligent cross-platform analysis • Runs 5000-10000 test-focexecs every night on every platform on several different daily gens of the product - mainframe (z/OS and CMS), plus Linux, AS400, Windows, and many Unix platforms • Many test-focexecs contain 5+ tests each. Many others generate hundreds or thousands of test variations via Dialog Manager loops • Date and datetime combinations spanning centuries • Thousands of /format variations everywhere formats can appear • New IBI invention to run interactive tests in batch (FOCREPLAY) All combined, about 100,000 tests every night • QA and Programming staff examine results daily • Discrepancies are reported and fixed for the next cycle • Overnight coverage via staff located world-wide

3. Advances in FOCUS Quality AssuranceNew Testing Concepts • Random test-generation and test combinations • Hits combinations we could never predict • Random execution paths simulate live sessions • Non-predictable results, different every day, ok as long as it finishes cleanly • Real customer applications simulated in-house • Parallel runs of random test combination against prior release • Compare results to each other rather than to approved base results The FOCUS language has combinations. How can you possibly test every one of them?

4. Intelligent Testing System WHERE Generator SU Scenarios MODIFY, MAINTAIN, HLI Parser Stress Tests Customer Applications SU Stress Mechanism TABLE Syntax Generator System Stress Tests New QA Tools and MechanismsAll created in the last 3 years

5. Intelligent Testing System WHERE Generator SU Scenarios MODIFY, MAINTAIN, HLI Parser Stress Tests Customer Applications SU Stress Mechanism TABLE Syntax Generator System Stress Tests New QA Tools and MechanismsSU Scenarios

6. SU Scenarios MODIFY, MAINTAIN, HLI New QA Tools and MechanismsSU Scenarios • Plays every logical combination (scenario) that forces two or more users to collide on specific records or database paths with conflicting intentions to Include, Update and Delete that record or path • Results in FOCCURRENT return codes that identify the conflict, or predictably affects a TABLE that is processing at the same time • Plays all of these against each other: • MAINTAIN • MODIFY • TABLE • HLI • Commit

7. Intelligent Testing System WHERE Generator SU Scenarios MODIFY, MAINTAIN, HLI Parser Stress Tests Customer Applications SU Stress Mechanism TABLE Syntax Generator System Stress Tests New QA Tools and MechanismsParser Stress Tests

8. Parser Stress Tests New QA Tools and MechanismsParser Stress Tests • Use every FOCUS keyword in every possible position in the TABLE syntax to uncover language parser inconsistencies TABLE FILE BY SUM BY/BY_BY AS 'BY' BY BY PRINT BY COMPUTE BY = BY/BY; BY BY BY BY SUBTOTAL ON BY SUBHEAD "BY <BY " END

9. Intelligent Testing System WHERE Generator SU Scenarios MODIFY, MAINTAIN, HLI Parser Stress Tests Customer Applications SU Stress Mechanism TABLE Syntax Generator System Stress Tests New QA Tools and MechanismsTABLE Syntax Generator

10. TABLE Syntax Generator New QA Tools and MechanismsTABLE Syntax Generator • Randomly generates valid TABLE syntax • Random keyword selection and order, within syntactic rules • Random number of keywords, fields, sorting, prefixes, selection criteria, computations, text, summary lines, SET environments • Generates thousands of different combinations every day • Compares current release to any other release

11. Intelligent Testing System WHERE Generator SU Scenarios MODIFY, MAINTAIN, HLI Parser Stress Tests Customer Applications SU Stress Mechanism TABLE Syntax Generator System Stress Tests New QA Tools and MechanismsCustomer Applications

12. Customer Applications New QA Tools and MechanismsCustomer Applications Customer passes application to Information Builders Customer environment is duplicated or closely simulated on IBI hardw Database is snapshot of actual data or contrived data values Application is validated on current gen of next release – bugs fixed 2 8 7 6 4 3 1 5 How the counting happens Eight separate boxes with numbers 1 through 8 are overlaid in the right order and just quickly Appear sequentially. customer applications running in-house today! Customer informed of pass/fail results and migration adjustments

13. Intelligent Testing System WHERE Generator SU Scenarios MODIFY, MAINTAIN, HLI Parser Stress Tests Customer Applications SU Stress Mechanism TABLE Syntax Generator System Stress Tests New QA Tools and MechanismsWHERE Generator

14. WHERE Generator New QA Tools and MechanismsWHERE Generator • Generates thousands of complex, syntactically valid WHERE syntax combinations • Parenthetical expressions, calculations, subroutine calls, random field selections, varied comparison operators • Dynamically incorporated into standard TABLE syntax and executed

15. Intelligent Testing System WHERE Generator SU Scenarios MODIFY, MAINTAIN, HLI Parser Stress Tests Customer Applications SU Stress Mechanism TABLE Syntax Generator System Stress Tests New QA Tools and MechanismsIntelligent Testing System

16. Intelligent Testing System New QA Tools and MechanismsIntelligent Test System • Randomly combines cataloged individual tests into consecutive executions, looking for unpredicted interactions • Runs twice - against current and previous releases • Compares results, extracts suspicious activity, isolates the causal test, reruns separately, reports analysis to QA for further investigation • Daily alters test code and test environments on the fly to expose tests to originally-unintended conditions

17. Intelligent Testing System WHERE Generator SU Scenarios MODIFY, MAINTAIN, HLI Parser Stress Tests Customer Applications SU Stress Mechanism TABLE Syntax Generator System Stress Tests New QA Tools and MechanismsSU Stress Mechanism

18. SU Stress Mechanism New QA Tools and MechanismsSU Stress Mechanism Objective: Crash the sink machine or damage a database, and capture the evidence for evaluation • Random number of tasks per user, random task selection based on weighted frequencies, random inter-task and intra-task time delays, random data selection. Conflicts are unpredictable, like real applications. • Randomly performs Includes, Updates, Deletes, TABLE extracts, volume loads, intentional delays, user out to lunch or crash, HLI via COBOL • Settable # of simultaneous users; users drop off – new ones start up • Run it for 10 minutes or 10 days • Tracks every activity, generates traces, analyzed after run completes Tens to 100’s of simultaneous users randomly perform a large variety of tasks against sink machine databases in real time

19. Intelligent Testing System WHERE Generator SU Scenarios MODIFY, MAINTAIN, HLI Parser Stress Tests Customer Applications SU Stress Mechanism TABLE Syntax Generator System Stress Tests New QA Tools and MechanismsSystem Stress Tests

20. System Stress Tests New QA Tools and MechanismsSystem Stress Tests • Concentrate specifically on how FOCUS interacts with the operating system environment • Expect files to be sized, stored, opened, closed, appended, removed, etc. properly at expected times with expected return codes • Disks, directories, pds’s, temp areas, etc. to be accessible and utilized as expected • FOCUS’s awareness of other environmental conditions and changes that affect its processing, with proper messaging and handling

21. Intelligent Testing System WHERE Generator SU Scenarios MODIFY, MAINTAIN, HLI Parser Stress Tests Customer Applications SU Stress Mechanism TABLE Syntax Generator System Stress Tests Advances in FOCUS Quality Assurance Thank You !