A Day in the Life of a Tester

1. A Day in the Life of a Tester Irinel Crivat SDET Technical Lead ASP.NET http://www.asp.net

2. Overview • The purpose of testing a program is to find problems in it • What a tester does? • Design test case (automated & manual) • Find problems is the core of a tester work • Write problem reports

3. What are we testing? Writing dynamic, high-performance Web applications has never been easier ASP.NET helps you deliver real world Web applications in record time. ASP.NET combines developer productivity with performance, reliability, and deployment.

4. What are we testing? Improved Performance and Scalability • Compiled execution • Rich output caching • Web-Farm Session State Enhanced Reliability • Memory Leak, DeadLock and Crash Protection Easy Deployment • "No touch" application deployment • Dynamic update of running application • Easy Migration Path Developer Productivity • Easy Programming Model • Flexible Language Options • Great Tool Support • Rich Class Framework New Application Models • XML Web Services • Mobile Web Device Support

5. Feature Crews • Feature Crew “A small self-organizing, cross-discipline group that works together to completely deliver a discrete Feature”. • Crew has flexibility to decide how to work, how to meet quality bar • Feature: Independently testable functionality taking 1-6 weeks of elapsed time to complete • Crew: 1PM, 1-5 Dev, 1-5 Test, other disciplines as appropriate • No remaining work is left to do on a feature when it gets checked in • interim builds of the product are higher quality • Tasks are done earlier in the development cycle

6. Feature Crew Tenets • Small, agile work units – the smaller the better • Dedicated resources to reduce context switching • Autonomy and decision making pushed to lowest levels – you decide how to work, who does what • All members “in it together” – begin and end together • Feature Crew continues until all work is “done” - feature complete, no bugs, tests in place, etc.

7. Key Concepts • Quality Gates A set of quality metrics and deliverables that must accompany a feature when it is first submitted to the product. • Feature A discrete piece of independently testable functionality that integrates at the same time into the PU Branch. • Feature Branch A light-weight source branch that is dedicated to the development of a single feature. Goal is to allow Feature Crew to fully stabilize in isolation before integrating. • SCRUMA project management methodology that involves regular check-in meetings to map progress.

8. What do Feature Crews offer? • Increased Efficiency • More parallel involvement of all disciplines (better Dev and Test integration) • Less context switching in the core phase of building a product • Design issues and feature bugs are found earlier when they are cheaper to fix • More effective and regular communication with cross discipline counterparts • Fewer intra-version breaking changes • Enhanced Agility and Predictability • More honest schedules • More consistent definition of “done” with quality gates • More confidence in release dates • Spend more time building and less time stabilizing • Improved Quality • Higher quality new features • More stable builds to allow for better customer feedback

9. Feature Crew Life Cycle

10. Discipline Activities Dev Test PM UE Design Phase Feature design, functional spec Feature design, dev doc, prototyping Feature design, test plans, Feature design, content plans Checkpoint 1 Design Review Implementation Phase Coding Coding, bug fixing Test libraries, automation Threat model, bug triage, Feature DCRs, Bug triage, writing samples, quickstarts Writing content Checkpoint 2 Status Review Testing Unit tests, bug fixing Testing IntegrationPhase Finalizing quality gates, testing, bug fixing Finalizing quality gates, testing Finalizing quality gates, testing Finalizing quality gates Feature complete, ready to integrate RI

11. Responsibility Areas Feature Crew Members

12. SCRUM • Agile programming methodology • Small, dedicated groups • Daily sync up meetings among crew members • Anyone can attend, but only crew members can talk • Led by a Scrum Master who tracks progress and drives problem solving • Ideally – short ~10 minute meeting where crew members answer only three questions: • What did you do since we last met? • What are you doing next? • Are you blocked? Yes/No • Problem solving and discussion happens after the SCRUM meeting

13. Phase-by-Phase Breakdown • FC Kickoff • Design Phase • Checkpoint #1 – Design Review • Implementation Phase • Checkpoint #2 – Progress Review • Integration Phase • Feature Complete/Integration Verification

14. FC Design Phase • Activities – agreeing what to do, how to do it • Create functional specs, dev design plans, test plans, UE plans • All disciplines document plans from their perspective in parallel • Enumerate Quality Gates, Dependencies, Fundamentals • Build into FC plans, decide when/who/how • Identify risks, issues with any • Schedule Quality Gate requirements • Updated costing of features, updates to schedule if needed • Define Checkpoint process for FC (depends on feature complexity) • Deliverables • functional spec, dev design, test plans, UE plan • FC Schedule solidified, Work items identified, Feature list updated with appropriate info • Feature Branch created, builds setup • Tools in place • How it works • FC works together to define designs, all participate in all aspects of design, iterative brainstorming meetings • Interactive, iterative discussions between PM, Dev, Test, UE • Design phase completion triggers Checkpoint #1 • FC decides when/how to schedule checkpoint with mgmt team

15. Testing - Design Phase • Characteristics of a good test • It has a reasonable probability of catching an error • It is not redundant • It’s the best of its breed • It’s neither too simple nor too complex • It makes program failures obvious

16. Testing - Design Phase • Techniques to come up with powerful test cases • Equivalence class analysis • Boundary analysis • Testing state transitions • Testing race conditions and other time dependencies • Doing error guessing

17. Test case design Equivalence class analysis • If you expect the same result from two tests, you consider them equivalent • A group of tests forms an equivalence class when • They involve the same input variables • They result in similar operations in the program • They affect the same output variables • None force the program to do error handling or all of them do

18. Test case design Boundary analysis • The boundary values are the biggest, smallest, soonest, shortest, loudest, fastest, ugliest members of the class, i.e. the most extreme values • Many testers include a mid-range value in their boundary tests. This is a good practice.

19. Test case design Testing state transitions • Every interactive program moves from one visible state to another. If you do something that changes the range of available choices or makes the program display something different on the screen, you’ve changed the program state • Advice: • Test all paths that you think people are particularly likely to follow • If you suspect that choices at one menu level or data entry screen can affect the presentation of choices elsewhere, tests the effects of those choices or entries • Try a few random paths through the program.

20. FC Implementation Phase • Activities – getting it done • Coding, creating unit tests, • Test automation • Testing and bug fixing • Quality work: API Reviews, Threat models, etc. • Fundamentals (perf, stress, etc.) • Deliverables • Daily SCRUM meetings (minimally quick hallway/email sync-ups among FC) • Feature code and unit tests complete • Automation tests in place • Documentation, including tech reviews of content, samples, app building • All FC bugs addressed • How it works • Dev and Test work in tandem - iterative, parallel progress made throughout • PM facilitates, tracks/communicates status, drives bug triage • All FC members help out as appropriate for load balancing – no discipline boundaries • Checkpoint #2 scheduled by FC during Implementation phase when main coding is complete

21. Testing – Implementation Phase • Activities – getting it done • Find bugs • File problem report, • Test automation • Run automated tests and perform analysis on the failures • Quality work: API Reviews, Threat models, etc. • Fundamentals (perf, stress, etc.) • Deliverables • 70% automation tests in place • 100% passing rate on the automated suite • All problem report (bugs) are verified/closed

22. Fundamentals Details • Perf and Scalability • Stress and Capacity • Globalization/Localization • Security • Compatibility • Side-by-Side • Accessibility • User Experience • Hosting • 64-bit • Servicing

23. Problem report written A good report is You need to track the problem so you must describe it in writing. Otherwise, some details (or the whole problem) will be forgotten. You also need a report for testing the fix later

24. Problem report numbered A good report is • Track Problem Reports numerically. • Assign a unique number to each report. • If you use a computerized database, the report number will serve as a key field. • This is the one piece of information that always distinguishes one report from all the rest.

25. Problem report simple A good report is simple = not compound Describe only one problem on one report Why multiple bugs on a single report are a problem? • The programer can fix only some of them and pass the report back as fixed, even though some bugs have not been fixed. This wastes time and leads to bad feelings. Remaining problems often stay unfixed long time…

26. Problem report understandable A good report is You must describe the program’s problematic behavior clearly. Keep all unnecessary steps out of the steps required to reproduce the problem.

27. Problem report legible A good report is Think of the person reading it. Unless you are reporting a disaster, the programmer will toss an illegible report onto his/her pile of things to look at next year

28. Problem report non-judgmental A good report is Nobody likes being told that what they did was wrong. As a tester, that’s what you tell people every day. Do not express personal judgments in your reports The programmer and the tester are a team that makes the product better

29. Problem report reproducible A good report is How can I make the bug reproducible? • You can describe how to get the program into a known state, and from there describe an exact series of steps • Find the simplest, shortest and most general conditions that will trigger the bug • Find related problems • Find alternate paths to the same problem • Find the most serious consequences • Look for the critical step

30. Test Automation • Automation framework (in house) • Building Maintainable Testsuites • Automation Results Reporting • Automation auto-analysis tools • Code coverage • Keeping test suites reliable • Gauntlet and Code Reviews

31. FC Integration Phase • Activities – making sure it works • integrate from PU branch • Run all tests, verify feature works in integrated world • Several functional runs • Stress • Performance • Code Coverage Run • Fix and verify bugs (includes Product bugs) • Quality Gates final verification • integrate into PU branch, final verification in PU branch • Deliverables • Feature List data updated • Quality Gates complete, verified • Final integration testing complete • How it works • FC tests feature with latest integrated code from PU Branch • All FC members verify feature meets requirements and works as expected • Team members move onto next Feature Crew

32. Resources: • Books • How to Break Software: A Practical Guide to Testing, by Whittaker, James A. (Author) • Lessons Learned in Software Testing, by Cem Kaner (Author) • Effective Software Testing: 50 Specific Ways to Improve Your Testingby Elfriede Dustin(Author)

33. Comments, Q & A