Fusion Testing

1. Fusion Testing Maximizing Test Execution By: James Tischart

2. About Me • Director - Product Delivery SaaS at McAfee • 15+ years experience in testing and engineering • Multiple certifications in Agile and Testing • Passionate about Testing as an eng discipline, a science and an art • Continue to challenge the status quo, test new approaches and always strive to improve • Support the legitimacy of the testing professional in the broader engineering world

3. What is Fusion Testing? • An occurrence that involves the production of a union • Organization of structured testing • Freedom of exploratory testing • Rigors of automated testing • Combined into one test methodology • To achieve • Maximize code execution • Increased test coverage • Reduced test artifact and documentation • Higher quality for users • Improved data for organization

4. Fusion Focus – start your day with 15 minutes of thought Usage – how will your users work with the system Scope – decide on the scope of everything Initiate – just go and explore Organize – create a plan & be ready to deviate from it Note – keep track of your exploration to retrace steps

5. Fusion Testing Guidance Maximize test execution with Fusion by: • Identify Goals for Testing • Choose the right mix of methodologies • Utilize Tests Lists to guide exploration • Automate the Right Things • Document at the Right Detail

6. Fusion Implementation Guide your team through the change Create a Change Message Mentor your team members Train the team in testing and engineering Create Metrics to measure success & failure

7. Fusion & Planning • Release Test Plan • Set the goals for the overall release • Plan the test environment • Organize the metrics and performance • Plan a budget, resources and a time frame • Plan your automation and tool set • Plan at a high level • Iterative Test Plan • Very effective in Agile implementations • Break up the testing into iterations • Low level details for the time frame • Keep to two weeks or less • Detail the testing to accomplish • Detail the performance tests • List what will not be tested • Identify risks to the plan • Plan what Regression Testing • Identify what test-lists need to be written • Daily Plan • Individually created each morning • Provides guidance and goals for the day • Details the execution plan • Test Lists • A grouping of test ideas • No specifics in the test list • Priority, variations & guide for exploration

8. The Power of Many • Utilize different users of your system to bring fresh perspectives • - Support personnel - Sales staff • - Developers - Training • - User groups - Product Management • Help guide the testing by providing checklists, environments and goals • Don’t give exact details or specific steps this minimizes innovation • Make the event fun and you will have many people continuing to help • Have testing experts available to help the volunteers for problems

9. Identify Defect Traits • Next analyze Sev 1 internally found and fixed issues and determine the traits • Inform the developers & testers of the traits • Create test lists or test ideas to cover these traits • Repeat the process with each severity level you use • For new products use historical traits from similar projects • Analyze Severity 1 Customer reported incidents • Determine the trends to those defects in coding & testing • Constrain the review to a calendar year of defects

10. Fusion & Automation • Determine your layers for automation: • Server • Databases, server processes • Machine interaction points • Process interaction with other layers • Middleware/API layers • User Interface • Optimize your test artifacts • Don’t duplicate test-cases • Use automation for traceability.

11. Test Results & Metrics Since you can’t test everything, here are some ideas of results to report: • # of Test Ideas Executed based on Priority • # of Test Cases passed/failed versus total coverage that the tests represent • # of Automated Tests Passed/Failed by # of executed times • Performance/Benchmark comparisons by build/iteration/release • Team Quality Satisfaction Rating – get the gut feeling of the team • Open defects to highlight potential issues that your customers may find • Go to fusiontesting.blogspot.com for more details on these metrics There will always be interest in the test data no matter the test structure. Q: How can you accurately assess quality when the testing combinations exceed the particles of the universe? A: You can’t! We need to present what was tested, what has not been tested and support the assessment used to make this prioritization.

12. Traceability Matrices

13. Implementation Challenges • There are challenges to implementation centralized in 5 areas: • Management • Product Management • Engineering Teams • Regulated Projects • Outsourcing

14. Challenge: Management • Identify the Challenges: • Reliance on historical metrics • Understand the current processes and practices • Decision-making timelines • Need traceability to feel confident of data • Respond to the Challenges: • Provide better metrics • Show how new process improves quality • Prove how decision can be made faster • Review traceability needs and support them

15. Challenge: PMO • Identify the Challenges: • Reliance on historical metrics • Struggle with the details of testing • Need predictability for costs and schedules • Require improved time to market • Respond to the Challenges: • Debunk misleading metrics • Show how more testing provides better data • Prove reduced costs with efficiency gains • Track lost opportunity cost on test artifacts

16. Challenge: Engineering • Identify the Challenges: • Takes time away from coding • Testing can be tedious • Not their specialization • Rely on a serial approach to testing • Respond to the Challenges: • Test first design increases new coding time • Automated & exploratory tests are less tedious • Understanding testing improves code writing • Less defects will be logged with up-front tests

17. Challenge: Regulation • Identify the Challenges: • Documentation Requirements • Formal or Standards Approval • Full traceability • Rigorous Automation & detailed test results • Respond to the Challenges: • Lean towards more structured testing • Use exploratory but document results • Plan for and work in shorter iterations • Automate more, document more

18. Challenge: Outsourcing • Identify the Challenges: • Require more instruction and detail • Lack expertise in the product and markets • Multi-cultural communication • Formal process & document to support work • Respond to the Challenges: • Phase in Fusion and train on exploratory • Provide more details in documentation • Implement multiple communication paths • Support and train new processes

19. Ensuring Success Implement Monitor Adjust Implement in stages Measure the impact to customers (before & after) Collect metrics for a project for test execution time before implementation Introduce Fusion to cover about 25% of testing Increase Fusion usage to 50% to further improve execution time Continue to phase in until 75 to 90% of testing is Fusion Monitor Increases or decreases in execution time Review documentation size closely reviewing the test artifacts Measure the amount of direct automation Review with the team the amount of time spent on maintenance Continuously try to improve exploration time and increase efficiency Adjust based on the needs of the project One project may require more precise documentation Another project may require less structure Identify these requirements at the start of projects and adjust Reduce the amount of time spent on artifact management Inform people of the time impacts of data collection

20. Five Keys to Fusion 5 5 5 • Detail – the right level of detail at the right time • Planning – consistently plan, execute and adjust • Automate - spend time on specifics in automation • Report – measure what truly represents the team and customer • POM – get more people involved to expand the breadth of test 5 5

21. Conclusion Rick Craig James Bach Thank You My family, friends, colleagues who have supported me! QA&Test Hung Nyugen Cem Kaner