Don’t Just “Test”… Validate!!

1. Don’t Just “Test”… Validate!!

2. Agenda • Overview of Testing versus Validation • Common Activities and Responsibilities • Justification for Validation • Discussion

3. Install Set For Year End Production Environment Design and Coding Testing Traditional View of Testing Project Life Cycle

4. Hiring Freeze Install Set For Year End Design and Coding Changes to Requirements Time & Resource Constraints Production Environment Testing Project Life Cycle

5. Risk FactorsExamples of Project Risk Factors SEI identifies at least 10 project risk factors: • Over-ambitious Schedule • Under-ambitious Budget • Over-ambitious or unrealistic expectations • Undefined or misunderstood obligations • Inadequate software sizing estimate • Unsuitable or lack of development process model • Continuous requirements change • Unsuitable organizational structure • Inadequate software development plan • Lack of political support Assess these factors in relationship to the testing effort

6. Factors Affecting Testing Short Discussion: What have you seen?

7. Testing Objectives Did we build the system right? Maximize System Quality Minimize Testing Effort

8. What Is Validation? Providing documented evidence with high degree of assurance the system will consistently produce a product or perform a process which meets the predetermined specifications and quality attributes

9. Critical Success Factors Risk to Business Validation Business Needs Factors Driving Validation

10. Relationship of Risk to Validation Risk tells you • What to address first • What to address most thoroughly Degree of Validation Degree of Risk

11. Validation Objectives Did we build the right system? Maximize System Quality Meet Business Needs

12. Goals of Validation • Control of system development: • Requirements meaningful, specified, and approved • Effective process capability of developer • Existence of and conformance to SOPs • Effective, appropriate, documented testing • Configuration management & change control • Control of system use: • Current accuracy and reliability • Continued accuracy and reliability • Management awareness and control • Auditability • Data integrity • Reviewer independence and credentials

13. What Makes Up A System? Peripherals Documentation Procedures Infrastructure Software Hardware Trained Personnel The Computer System The Business Environment

14. Testing (traditionally) Encompasses… Documentation Procedures Infrastructure Software Hardware Trained Personnel The Computer System

15. Validation Encompasses… Peripherals Documentation Procedures Infrastructure Software Hardware Trained Personnel The Computer System The Business Environment

16. Marketing Design Project Orientation SDLC Construction Test Construction Test Execution Elaboration Test Evaluation Transition Test Planning Inception Test Strategy Test Scope QC Process Transition Executing Initiation Planning Closure Customer Requirement Idea Generation General Release Support Retirement Operations Implementation CMM Quality Assurance ProgramProgram Owner AccountabilitiesQA Accountabilities occur at every touch point Product Process Project Management Process

17. QA & QC Coming Together Project Orientation Initiation Transition Executing Planning Closure Quality Assurance Process Audits Process Consultation Defect Analysis Measurement Analysis Continuous Improvement Standards Development Risk Management Defect Management Test Planning Static Testing Dynamic Testing Test Automation Test & Quality Control Defect Resolution Metrics Test Execution Metrics Project Planning Support Defect Cost Analysis Benchmarking Forecasting Measurement & Analysis

18. Strategy Evaluate Design Execute Common Activities Each Level is Composed of These Activities

19. Strategy Evaluate Design Execute Strategy Development • Determine Validation Strategy • Perform Risk Assessment • Identify Critical Success Factors • Set Validation Objectives • Define Validation Activities • Estimate Time & Resources • Document Validation Strategy

20. Factor Extent to which ... Correctness … system must satisfy stated requirements Authorization … processing requires management authorization Data Integrity … data stored by the system must be accurate Service Levels … schedules must be met Access Control … access must be restricted Methodology … development plan must be followed Reliability … system cannot fail during operation Ease of use … effort is required to learn and use system Maintainability … effort is required to fix errors in the system Portability … system can operate in multiple environments Performance … functions must perform within a specified time Identify Critical Success Factors

21. Strategy Evaluate Design Execute Design • Develop/obtain tools for conducting Validation Activities • Deliverables & Phase Checklists • Procedures describing reviews & inspections • Validation Matrices • Test Cases • Automated Testing Tools • Audit plans

22. Examples of Validation Objectives # Validation Objective Priority Completion Criteria 1 Customer orders processed correctly High Enter customer order information as described in the validation test cases and verify the results are correct 2 System is ready to use High No open defects of type SEVERE or HIGH Performance criteria are met Users are trained, documentation is ready 3 Hold is placed on inventory when QC test fails Med Enter FAIL test results for product Verify MRP II indicates status as HOLD Verify product is segregated in warehouse 4 Increased through-put Low Measure increase by time study 5 Labels printed as req’d Low Print labels, verify against approved label copy specifications

23. Examples of Validation Activities Activity Performed by Description Deliverable Requirements Reviews Business Analysts, Developers, Users Study & discuss system requirements to ensure they meet business needs Reviewed statement of requirements, ready to be translated into design Design Reviews Developers Study & discuss system design to ensure it will support requirements Design, ready to be translated into software, hardware, documents, etc. Code Inspections Developers Informal analysis of code to find defects Software ready to be inspected or tested Code Walk-throughs Developers Formal analysis of code to find defects, problem areas, coding techniques, design flaws Software ready to be tested

24. Examples of Validation Activities Activity Performed by Description Deliverable Unit Testing Developers Test single units of code. Validates that the software performs as designed. Software unit ready for testing with other components Integration Testing Independent Test Team Test related units, programs, modules. Validates that multiple parts of the system interact according to design. Portions of the system ready for testing with other portions System Testing Conducted at end of Integration Testing. Test entire system. Validates system requirements. Tested system, based on specification Acceptance Testing Users Test system to make sure it works in the environment. Validates business needs. Tested system, based on user needs

25. Detailed Validation Matrices Requirement Test Case 1 2 3 4 5 6 Validate sign-on x Invalid term ident x Validate password x Password/ Signon error x Validate Main Menu x Re-enter O, X, M, R x Validate Customer # x Customer # Required x Field entry too long x Customer ___ not found x

26. Strategy Evaluate Design Execute Execution • Organize Validation Tools • Train Team Members • Execute Validation Plan • Track Validation Progress • Perform Regression Testing • Document Results

27. Strategy Evaluate Design Execute Evaluation • Evaluate Compliance of System Development • Evaluate Defects • Develop Findings and Recommendations • Formalize Report

28. Who Is Responsible for Validation? Developer Business Unit (User) Validation Process Quality Assurance Independent Tester

29. Business Unit Responsibility • Define system requirements • Define data used in the system • Enforce Change Control • Plan & Perform Acceptance Testing • Ensure adequate backup & recovery of data is available • Work with developers to coordinate schedules • Investigate Defects • Develop User Documentation • Develop User Procedures • Enforce security • Provide audit trail • Train personnel

30. Developer Responsibility • Follow development process • Work with QA to improve the process • Manage changes to system components • Provide adequate backup & recovery • Maintain configuration management • Plan & Perform Unit Testing • Provide Technical Documentation • Maintain communication with QA , Testers & Business

31. QA Responsibility • Define Development & Validation Process • Verify personnel have adequate training & resources for their job • Review documentation - adherence to standards • Periodically review process activities • Evaluate vendor capabilities & processes • Verify enhancements & maintenance are documented • Conduct Risk Assessments • Verify business unit has established procedures • Ensure system is adequately secure

32. Independent Tester Responsibility • Follow Validation Process • Working with Users &/or BA’s, perform Requirements Validation • Work with QA to improve the process • Plan & Perform Integration & System Testing • Conduct Risk Assessments • If required, Plan & Perform Stress & Performance Testing

33. Testing Levels & Validation Business Needs Acceptance Testing Requirements System Testing Design Integration Testing Code Unit Testing Production

34. Plan for Maintenance • Prepare for Maintenance during Development • Keep Deliverables and Documentation up-to-date • Requirements • Design • Test Cases, Test Data, Test Results • Maintain all related deliverables during error correction • Requirement specifications • Program specifications • Design specifications • Defect reports

35. Maintenance Deliverables • Risk Assessment • Software updates • Hardware upgrades • System documentation • Manuals, procedures • Revised/New Test Cases • Updated Validation Plan

36. Validation Issues • Responsibility lies with the user • If outsourced, the responsibility is shared between the outsourcing organization and the user • Develop deliverables as system is maintained • Group maintenance changes together • Schedule maintenance in releases • Follow defined methodology

37. Purchased System Responsibilities Validation Activity Vendor Customer System Staff Users Feasibility Review x x Requirements Review x x x Design Review x x Code Review x Unit Testing x Integration Testing x System Testing x x Acceptance Testing x Conversion Testing x x Maintenance Testing x x

38. Cost of Quality The Cost of Correction $ $

39. Savings Failure Failure Appraisal Appraisal Prevention Prevention Without Quality Process With Quality Process Cost of Quality Real Benefits

40. Validation Costs and Benefits Incremental Benefits Benefits Net Benefits Net Costs Time Incremental Costs Costs Development Maintenance

41. Self-Assessment Y N ? Are risk assessments being performed on projects? Is there a methodology in place for system development and validation? Are verification techniques, such as walk-throughs, being used? Have all personnel been trained in validation techniques? Are standards in place for system development and validation? Are the users active participants in the validation process? Are the results of the validation used to improve the development and validation processes? Is the culture ready to accept new processes for validation? Is there someone responsible for validation of systems?

42. Validation: Did we build the right system? Testing: Did we build the system right?

43. System Validation • Validates BOTH Business & Technical Requirements • Reduces Development Costs by identifying Errors early in Development Cycle • Reduces Risk of Operating Failures after Implementation • Reduces Post Implementation Development Costs by Fixing Defects during Development • Improves Overall Quality of System • Greatly improves User Satisfaction

44. Summary Projects Delivered With Validation Projects Delivered Without Validation The Road to Success!!

45. Questions / Discussion