The Full Testing Experience

1. The Full Testing Experience Meeting new Business Challenges with Dynamic Quality Assurance • Bill Maurer • Developer Technology Specialist • Microsoft • Bill.Maurer@microsoft.com

2. Who is Bill Maurer?? Visual Studio Product Specialist “Dev Tools & Process” more than “Dev Technologies” Based in Minneapolis, MN Serves enterprise customers in: Minnesota, Iowa, Nebraska, Kansas, Missouri, North Dakota and South Dakota. 31 years in the software industry, 15 at Microsoft 22 years as a developer and development team lead 4 years introducing Microsoft .NET to developers 5 years in current role

3. Agenda • Test Planning • Manual Testing • Automated Testing • Lab Management • Wrap-up • (Remaining) Questions

4. Have you heard any of these? • “my testers are spending too long testing the same thing” “tooling is expensive (time, licenses, people)” “the developers say the defects are useless” “when is my software ready to ship?” “developers and testers work in silos and don’t communicate/speak the same language” “we have a changed requirement, what do I need to test?”

5. Project Manager

6. 68%

7. What if you could… reduce the time it takes to determine the root cause of a bug enable users to easily run acceptance tests and track their results reduce the time it takes to configure and manage test environments reduce the time it takes to verify the status of a reported bug fix repair a bug without having to reproduce it

8. Welcome to the revolution

9. What testers doAn Agile example • inception • construction • release • set test strategy • define done, done • establish environments • review configurations • create plans • generate data • add stories to plan • define acceptance • select regression tests • author tests • run tests • file bugs • verify fixes • automate tests • update master plan • identify product debt • identify test debt • select tests for automation • regression testing • release doneness testing • release sign off • iteration execution • iteration retrospective • release iteration • release planning • iteration planning multiple iterations

10. Where does testing happen? 70% of testing happens here Majority of test tools target here Black Box Testing White Box Testing API Testing

11. Visual Studio 2010 Test Capabilities generalist specialist coded ui tests manual tests web performance tests unit tests load tests test case management lab management data collectors (video, action log, event log etc) team foundation server with reporting (bugs, test cases, requirements, user stories, source control, build)

12. Let’s drill down!

13. What testers do • inception • construction • release

14. Are we there yet? • Test Plans • Test Suites & Requirements • Test Cases & Shared Steps • Test Configurations • Tracking Test Results

15. Anatomy of a Test Plan & how it all relates Team Project Test Plan(s) Test Suite(s) Work Items Test Case(s) Steps Start & End Dates Requirement State Source Code Automation Status Test Configuration (Manual & Automated) Stand-alone suite Test Data Automated Builds Parameters

16. Visual Studio 2010 Test Capabilities generalist specialist coded ui tests manual tests web performance tests unit tests load tests test case management lab management data collectors (video, action log, event log etc) team foundation server with reporting (bugs, test cases, requirements, user stories, source control, build)

17. Let’s take a look:Planning your Tests

18. What testers do • inception • construction • release

19. How do I get there from here? • Run Tests • Verify Bugs • Analyze Test Runs

20. Visual Studio 2010 Test Capabilities generalist specialist coded ui tests manual tests web performance tests unit tests load tests test case management lab management data collectors (video, action log, event log etc) team foundation server with reporting (bugs, test cases, requirements, user stories, source control, build)

21. Let’s take a look:Manual Testing

22. Reporting on Project Progress • Randy to add the stories report here • Following dashboard screenshots included in case you want them, please don’t remove from master deck. How many bugs have been found? Are the tests passing or failing? Are we making progress on requirements?

23. Reporting on Development Progress Are we completing work? Are we progressing as expected? Are we resolving roadblocks? Are we completing requirements?

24. Reporting on Testing Progress Are we making progress on running test plans? How are our builds doing over time? Are we fixing bugs? What’s the quality of our bug fixes?

25. What testers do • inception • construction • release

26. Automation Strategy iteration 1 iteration 2 release iteration feature a feature a feature b feature c feature d unit testing manual testing regression testing performance testing

27. Shifting into Overdrive. • Coded UI Tests • Automating Regression Tests • Performance & Stress Tests

28. Visual Studio 2010 Test Capabilities generalist specialist coded ui tests manual tesst web performance tests unit tests load tests test case management lab management data collectors (video, action log, event log etc) team foundation server with reporting (bugs, test cases, requirements, user stories, source control, build)

29. Let’s take a look:Evolving Manual Tests into Automated TestsandWeb Performance & Load Tests

30. What testers do • inception • construction • release

31. Visual Studio Lab Management 2010 Leveraging virtualization for ALM Library Share 1 Stored VM 3 Stored VM 2 Stored VM 1 TFS 2010 Hyper-V Host 1 Virtual Machine 3 • Lab Management • Build Management • Test Case Management • Project Management • Source Control • Work Item Tracking • Reporting Virtual Machine 2 Virtual Machine 1 Physical Environment / ESX Machine 3 Lab Agent Machine 2 Test Agent Machine 1 Build Agent Test Agent Build Agent Build Controller Microsoft Test Manager Visual Studio 2010 Test Controller

32. Visual Studio 2010 Test Capabilities generalist specialist coded ui test manual test web performance test unit test load test test case management lab management data collectors (video, action log, event log etc) team foundation server with reporting (bugs, requirements, user stories, source control, build)

33. Let’s take a look:Managing Test Environment Complexity

34. Team Build Full Workflow (2) Compile projects (1) Get source Team Foundation Server Team Build (8) Publish results to Team Foundation Server (3) Restore running environment to snapshot Test Controller (7) Send test results (4) Run deployment scripts for each machine Virtual Environments (6) Execute automated tests (5) Create post-deploy environment snapshot

35. Reporting on Builds Build progress overtime is tracked Recent build status is reported to the team

36. Reporting on Builds How many lines of code are we churning? Are we getting enough coverage?

37. Amount of code covered by tests Amount of code modified Failing tests are flagged for review

38. Wrap-Up

39. Your Creativity Unleashed • Save time, focus on high value tasks • Fast Forward for Manual Testing • High quality bugs with single click • Rich Modern Experience • Elegant Task focused UI • Combined manual & fast forward for manual testing • Plan, Test, Track flow • Embrace Change • Rapid Setup/Teardown of Environments • Test prioritization to assess impact of code check ins • Traceability and Visibility

40. Simplicity through Integration • TFS is THE collaboration hub • Break silos between roles • Share Virtual environments between team members • Quality Insight • Cross subject area reporting • Real time feedback • Go Live Readiness • Performance Testing Early and Often • Highly scalable (also 64 bit support) • Standard languages for scripts

41. Quality code ensured • No more “no repro” • Integrated, Extensible Diagnostics • Rich Bugs, fix first time, every time • IntelliTrace • Eliminate wasted cycles • Test in production-like environments • Efficient, predictable build hand off • Automation to reduce Regression • Coded UI Test • Familiar language & toolset support

42. Making it Real Company: Flextronics Size: 200,000 employees Problems: • Tools Chaos • Poor Collaboration • Long QA Cycles • Poor Project Tracking

43. Good News! • Visual Studio 2010 Test Professional is priced similar to Visual Studio Professional! • Visual Studio 2010 Test Professional is included in Visual Studio 2010 Ultimate so you may already own it! • Team Foundation Server is now included with MSDN! • Team Foundation Server Lab Manager is now included with MSDN!

44. Call to Action • Download Visual Studio and Team Foundation Server RTM virtual machine for Windows Virtual PC and try Test Professional today! • http://bit.ly/VSTFSVPC • Check out the Online MSDN Documentation – Testing • http://bit.ly/TestApp • Learn more about the ALM Assessment & ALM Catalyst • http://bit.ly/ALMLive • Let us help you build a roadmap to success

45. Questions and Answers

46. Visit Team System Café for… • Freshly brewed news • Upcoming events • Download links • Practical tips to go • Download Visual Studio 2010 Trials • http://tinyurl.com/ycfbfyo • Get Started with Lab Management • http://tinyurl.com/394nyfe • Slides for event are located under the Resources tab • Titled:“Full Testing Experience – Bill Maurer” teamsystemcafe.net

47. © 2009 Microsoft Corporation. All rights reserved. Microsoft, Visual Studio, the Visual Studio logo, and [list other trademarks referenced] are trademarks of the Microsoft group of companies. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation.  Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation.  MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED, OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.