Quality Assurance

1. Quality Assurance April 2012 • Chip Morgan & Ameer Noorani • UT Austin – MIS 374 Guest Lecture

2. Who is Chip? Chip Morgan UT BBA MIS, 1991 18 years experience delivering software projects for a wide range of clients in various states of peril Employer History Neiman Marcus – developer and QA/methodology BSG – project consulting and staff training BSI – product development Context Integration – curriculum development, project consulting GreenGrape Technologies – technical SME, architect, developer SunGard Consulting Services – project consulting and sales USAA – technical architect, wealth management and financial advice Hobbies Beach volleyball Disc golf Jabber/XMPP NAESB Smart Grid Initiative

3. Who is Ameer? Ameer Noorani • UT BBA MIS, 2008 (who taught my MIS 374 class?) • UTSA MBA, 2014 • 4 years experience delivering software projects Employer History • University of Texas at Austin – student technician for media console, hardware, and helpdesk support • USAA Summer Internships – tester, tested applications and created automation scripts for investment projects • USAA – software developer on wealth management, investments, and financial advice projects Hobbies • Football • Swimming • Softball • Video Games

4. EVERYONE IS RESPONSIBLE FOR QUALITY! Project Manager Technical Lead Product Manager Technical Architect Project Delivery Manager Functional Analyst Quality Assurance Engineer UI Producer Project Executive Sponsor Software Developer Quality Assurance Sr/Lead Business Manager Business SME Change & Release Coordinator UI Designer

5. Agenda • What is Quality Assurance? • Waterfall vs. Agile • Automation Testing • Test Environments • Other Types of Testing QA Challenges & Opportunities

6. Quality Assurance – What is it? “A Software Quality Assurance Engineer is involved in the entire software development process to ensure the quality of the final product. This can include processes such as requirements gathering and documentation, source code control, code review, change management, configuration management, release management and the actual testing of the software.”

7. Quality Assurance - Importance Brand • Apple – iPhone 4 (Antenna reception issue) • Toyota – Accelerator Adoption of New Tools and Functionalities User Experience Live Money Applications USAA Members/Channel of Choice • Deployed members have limited access to Internet (mobile.usaa.com) • Applications must work!

8. Quality Assurance – Profile of an Ideal Tester Passionate About Testing Detail Oriented Business and Technical Knowledge Understand Impacts Good Communication Skills Analytical Thinking Issue Resolution Key Traits for a Tester Key Traits for Anyone in IT

9. Quality Assurance – Cost of Fixing Defects

10. TYPICAL QUESTION TYPES THROUGHOUT THE PROJECT open-ended close-ended Design Requirements Construction Implementation • What is the current process? • What are your current pain points? • What are your priorities? • Would you like to have a “Select All” option on the results page? • When user selects cancel, do you want a “Are you sure?” prompt?

11. QA at USAA - 2012 Production Changes 18,818 Production Changes (RFCs) • System Testing: • 15,194 Defects Found • 242,079 Test Cases Executed • Release Regression Testing: • 163 Defects Found • 15,022 Scenarios Executed • Automated Production Validation: • 196 Defects Found • 94,340 Scripts Executed • Production Monitoring: • 3,025 Defects Found • 275 High/2,750 Medium & Low 83.7% of Defects Found in Testing FOR INTERNAL USE ONLY: May be shared with

12. Waterfall vs. Agile

13. Waterfall • Structured approach • Familiar to most developers world wide • Some organizational requirements dictate process over dates • NASA for instance Requirements Design Development Testing Deployment

14. Waterfall • Typically some overlap Requirements Design Development Testing Deployment

15. Waterfall • Date driven can yield this: Requirements Design Development Testing Deployment

16. What’s Agile?

17. Agile - Scrum Team Iteration Daily Standup Code Design User Story Test User Story User Stories Tasks Analysis Release Iteration Demo Iteration Retrospective Iteration Product Backlog Iteration Backlog End of Release

18. Agile - Iteration Retrospective < What should we do LESS? > What should we do MORE? = What should we TRY?

19. Example: Actual Project (Waterfall vs. Agile)

20. Waterfall Requirement vs. Lean Agile Story Waterfall Requirement: The system shall notify status on Salesforce.com when an Investment Review has been generated. Lean Agile Story: As an advisor I want to be notified when an Investment Review is generated so that I can schedule time to review it. Validation Criteria’s: • Validate that a task is created in Salesforce for the advisor when an IR is generated • Validate that an email notification is sent to the advisor when an IR is generated

21. Waterfall Timeline – Project Execution Plan Project Starts in September Implements Next Year as One Big Release to Production

22. Agile Timeline – Project Execution Plan • Changed Project Execution Strategy from Waterfall to Agile • Entire Project Team (including Business Partners) Went Through Agile Training • Divided Project Scope into Four (Epic) Business Deliverables From One Big Release

23. Agile Timeline – What Actually Happened • Epic 2 Was Delivered 90 Days Earlier Than Original Plan! • From Demos/Lab Prod Feedback, Business Realized Epic 4 Was Not Needed!

24. Agile - Continuous Integration Who Broke the Code Build!?!! Developer’s Code is Merged Frequently Into Test Environment No Waiting Till the End for Integration Testing Better Chances to Catch Missed Requirements Earlier Better Test Coverage • Better Unit Testing & Iteration Functional Testing

25. Test Environments

26. Test Environments @ USAA • Unit Testing Environment • Developer’s Workstation • Local Environment • Integration Testing Environment • Developers Sandbox for Agile or Waterfall • DEV Environment • Enterprise System & Release Testing Environment • Merge & System Test with Entire Enterprise Efforts Targeting a Release Date (Waterfall or Agile) • Runway/Release Test Environment • System Testing/Iteration Functional Testing Environment • Project Test Environment (Waterfall) • Iteration Functional Testing (Agile) • Demo Environment (Agile) • Production QA Environment • Sneak Preview/Pilot of New Functionality to Live Users • Validate New Functionality • Lab Production*(Agile) • A Slice of Prod • Feedback from Users in Agile Projects • Projects Have Full Control • Production • Validate New Functionality is Present

27. Lab Prod Lab Prod Lab Prod Large Lab Prod Large Lab Prod Small Lab Prod Small Lab Prod Lab Prod Lab Prod Flow

28. Automation Testing

29. Automation Testing • What is Automation? “Use of special software to control the execution of tests and compare actual outcomes to predicted outcomes. Test Automation can automate repetitive testing scenarios and/or help execute additional test scenarios which are difficult to perform manually.” • Actual Lessons Learned by a Project: • Essential for Building in Quality on Efforts • Allows for Fewer Testing Resources • Allows for More Testing Coverage in a Iteration

30. Automation Testing Refactoring • Re-organizing part of the code • Used for maintenance, performance, and new functionality addition • Increases code quality & testing coverage For example, if a programmer wants to add new functionality to a program, he may decide to refactor the program first to simplify the addition of new functionality in order to prevent difficulty with maintenance.

31. Because of Refactoring… When testing in Iteration 1…. Test Iteration 1 with every code change In Iteration 2….. Test Iteration 2 AND Test Iteration 1 What about Iteration 14?

32. Automation Testing Benefits • Reduce manual testing • Ability to Quickly Regression Test Applications • Allows for on-demand execution • Driven by code changes • Ensures completed stories are still working properly • Finds defects for closed stories • Dedicated Lab Automation Lead • Remove impediments • More accurate estimates • Participates in Ceremonies

33. Automation Testing Challenges • Maturation • Current State: Manual Testers and Automation Testers • Future State: Manual and Automation Tester • Environment Stability • Test Data

34. Testing Challenges & Other Types of Testing

35. Testing Opportunities & Challenges Mobile Testing Browser Testing Across Various Browsers Test Data • Perfect Test Data – Miss key scenarios • Bad Data – Rabbit hole Test Environments Stability • Multiple Efforts Making Changes Testing in Agile - Maturity

36. Other Types of Testing Performance Testing • Testing for response times on applications • Testing for volume of users on applications • Tools: Dyna Trace, Load Runner Security Testing • Testing for OWASP Top 10 security issues (ex: Cross-Site Scripting, SQL Injection, etc.) • Ensure the security of member data in usaa.com and mobile devices • Tools: Automated and manual (e.g. IBM Security Appscan, Webscarab, and Paros) Accessibility Testing • Functional testing for users with disabilities (ex: vision, color-blindness, etc) • Ensure accessibility of applications across channels (ex: USAA .com, mobile, etc.) • Tools: JAWS / NVDA / WAV toolbar / Compliance Sheriff

37. Performance Testing – Why Should You Do It? User Experience (# of Seconds for Backend Service Calls) • Response & Wait Time On Application • Asynchronous vs. Synchronous System Design System Capacity (# of Users Accessing System) • Projected Business Growth • Average Peak Volume vs. Highest Peak Volume • Marketing Campaigns

38. Security Testing – Six Basic Concepts Confidentiality – Need to know basis Integrity – Information is correct Authentication – Identify person/message Authorization – Access control Availability – System will work Non-Repudiation – Message delivery

39. Accessibility Testing “Laws and public opinion frown upon discriminating against people with disabilities” • Perceivability • Images across alternative media types, color contrast, text size • Operability • Flashing content and risks to users (epileptics) • Understandability • Comprehension of content, text legibility • Robustness • Application of technology, javascript, etc.

40. Recap • Quality Assurance – Responsibility & Importance • Waterfall vs. Agile • Automation Testing – Reduce Manual Testing & More Test Coverage • QA Challenges & Opportunities – Environment Stability, Test Data, Browsers • Non-Functional Testing – Performance, Security, and Accessibility Testing Various Test Environments & Phases – Prod QA, Lab Prod

41. Questions

43. Scrum Process 43

44. Waterfall Timeline – Project Execution (Slide 1 of 3)

45. Waterfall Timeline – Project Execution (Slide 2 of 3)

46. Waterfall Timeline – Project Execution (Slide 3 of 3)