1 / 23

Curtis Anderson Sivaprasad Padisetty

Curtis Anderson Sivaprasad Padisetty. May 29 th , 2003. Overview of Presentation. Part I – Software Development and Automation The Project Cycle Benefits of Automation Effective Automation Part II – Test Distribution and Development Test Distribution Systems Test Development Summary.

larya
Download Presentation

Curtis Anderson Sivaprasad Padisetty

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Curtis Anderson Sivaprasad Padisetty May 29th, 2003

  2. Overview of Presentation Part I – Software Development and Automation • The Project Cycle • Benefits of Automation • Effective Automation Part II – Test Distribution and Development • Test Distribution Systems • Test Development • Summary

  3. Software Development and Automation Curtis Anderson

  4. The Project CycleTest Investment in Automation • The investment in test automation starts early • Focus on adding testability to spec/design phase • Investment usually peaks as Development slows feature coding Testing Phase Major Milestone Phase M1,2… Testing Investment Release Phase Planning (M0) Time

  5. The Project Cycle Automation Payback Testing Phase Major Milestone Phase M1,2… Automation Payback Planning (M0) Release Phase Time • Return on Investment is near zero early • Largest gains at project end game and sustaining engineering

  6. Benefits of AutomationWhat does Automation Bring to the Project? • Consistent Test Execution • Stability Guarantees • Reliability Metrics • Reproduce-able Quality Assessment

  7. Advantages of AutomationWhy Automate ? • Machines instead of People • Run existing tests automatically on a new build or version • Exact same test happens every time • Do More Testing • Simplistic, repetitive (BORING) tests are run for free. People spend time looking only at results • People are free to ad hoc and search for bugs

  8. Advantages (continued) • Automation can perform costly, hard, or impossible to execute tests • Perf, Stress, Scalability, Config, OM/API, Matrix • Consistency and Reproducibility • Performance, Regression • Reusability • How many File Open drivers are there? • Logging, etc. • Ensure a level of stability before a human spends time • Keeps the ‘patient healthy’ • Easy to hand off (external teams & sustaining engineering)

  9. Risks and Problems • Over-reliance on automation results • Its easy to generate lots of results that creates too much noise • Maintenance • Exact same test happens every time • False failures and false positives • Skill set to produce • Re-inventing the wheel Time Management is Key!

  10. Unit Test Interoperability Localizability Interoperability Performance Reliability Scalability Security Globalization Checkin Test Build Verification Acceptance Test Pass/Comprehensive Regression DogFood/Deployment System Integration Code Coverage Test Categorization Categorizing tests helps target testing and keep things focused. What does it do vs. When does it run

  11. Make Your Automation Effective • Time Management is key • Cost of authoring • Re-use everything • Write solid automation • Document well • Base all automation on test cases • Be willing to punt if it’s not working • Run the right automation at the right time • Roll up the right metrics to your project

  12. Test Distribution System Overview Why do we need a system? • Standard way to execute tests • Automate the execution • Standard reporting and history What does it do? • Repository for managing automation • Test distribution • Report Results • Track & manage failures

  13. Test Distribution and Development Sivaprasad Padisetty

  14. Test Distribution SystemComponents • User Interface • Middle Tier • Database • Machine Management/Controller • Client Agent • Test Harness • Machine Imaging and other common tasks • Results Viewer • Email

  15. Test Distribution SystemFeatures/Requirements of a good system • User Friendly, Modular, Flexible • Integrated Automated/Manual test support • Distributes execution – lab management • Multi stage, multi machine • Offline execution and Private network support • Test Version Control • Object Model • Queuing and Resource management • Hardware and Software tracking • Configuration management & Matrix support • Deployment/Image management • Results reporting

  16. Test Development • Abstract from code as much as possible – the less code that changes to add a test case the better • Separate infrastructure code from test code • Re-use as much code as possible • Modeling and Data Driven tests help scale

  17. Test DevelopmentGood Authoring is: • Focused • Consistent • Accurate & Reproducible • Portable, Easily Leveraged • Impervious to Feature Changes • Feature churn • Language, even application, user, etc. Cost / Benefit is the ONLY measurement (Don’t forget sustain engineering)

  18. Test Development Writing Automation is Software Development • Log-Log-Log-Log-Log everything • Plan & Design • Develop coding standards • Write maintainable code • Document you code and your test cases • Peer-reviews before check in • Use a version control system (Source Depot or VSS) • Test your test. • Log-Log-Log-Log-Log everything This code will last as long as the product code!

  19. Test DevelopmentDevelop Configuration Agnostic tests No hard coded anything! • links like files in the definition • No dependency on a specific machine name • No dependency on a specific lab • Should be package-able. • Configuration changes & dependencies should be clearly defined.

  20. Reporting and Metrics • Good logging saves time • Individual Log files from Test Execution • Roll up summary reports for all tests • Available through system UI and email • Project roll up • Metrics help with project status

  21. Summary • Test Automation is a competitive advantage for MS • Lack of Automation hurts innovation in the product • Test Development is Software Development • Many resources and experts exist across the company

  22. Test Automation Resources Automated Testing Handbook – Linda G. Hayes Software Test Automation, Efffective Use of Test Execution Tools, Mark Fewster and Dorothy Graham Internal • Windows (WTT & WCTI) • http://wttweb, http://team/sites/wtt, http://winwebtest/wcti and http://wttforum • Test Categoization doc link: http://teams/sites/wcti/stc/default.aspx • Join Partner DL wttpart, to participate in Shared Development community. • Office • http://office and http://oasysnet • Automation Strategy Doc - http://officenet/Teams/DnR/DnR%20Test/Specs/Office%20Automation%20Strategy.htm • Office automation contacts – mailto:offauto • DevDiv • http://devdiv and http://maddog/ • MadDog forums - http://maddogforum/forums/ • MadDog General Docs - http://maddog/MadDog%20General%20Docs/Forms/AllItems.htm

  23. Questions?

More Related