1 / 34

The Testing Pyramid

The Testing Pyramid. Nancy Chacko. About Me. Working in software testing for over 10 years Director of Software Engineering in Test at Opower Founding member of DCAST (DC Agile Software Testing group). http:// meetup.com /d- cast. Objective.

yanka
Download Presentation

The Testing Pyramid

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The Testing Pyramid Nancy Chacko

  2. About Me • Working in software testing for over 10 years • Director of Software Engineering in Test at Opower • Founding member of DCAST (DC Agile Software Testing group) http://meetup.com/d-cast

  3. Objective We will explore how to get testers and developers to think about testing in a holistic manner. By working together to create the set of tests and determine where that testing belongs, we can increase efficiency, reliability and reduce debugging time for failuresin the future.

  4. Overview • Testing and test engineers in an agile environment • Layers of testing and the pyramid • Uniting dev and test • Pushing tests to the lowest layers

  5. Analysis Design Build Test Maintain

  6. Agile Testing • Emphasis on rapid development forces a focus on testing • TDD is often practiced by developers • Automation is key to running fast • Testers transitioning to agile can often feel lost • If developers are testing, what is my role? • Teams are not used to working together – the divide of waterfall can continue

  7. Layers of Tests • Unit tests • Tests at the method level, often use mock objects • Integration/service/component tests • Tests at interfaces, often use real objects and DBs • End-to-end automated tests • GUI tests that need to be run for regression testing • Manual/exploratory/ad hoc/GUI tests • Used to fully exercise UI and explore unknowns

  8. Typical Divide • Development typically owns tests at the code level • Unit tests • Integration tests • Testers often own the tests at the UI level • Manual tests • Teams sometimes work together to automate the UI tests • End-to-end automated tests

  9. Typical Divide Test Dev Dev & Test Manual Tests Manual Tests Manual Tests End-to-End Automated Tests* End-To-End Automated Tests* End-To-End Automated Tests* Integration Tests* Integration Tests* Integration Tests* Unit Tests* Unit Tests* Unit Tests*

  10. http://watirmelon.com/2011/06/10/yet-another-software-testing-pyramid/http://watirmelon.com/2011/06/10/yet-another-software-testing-pyramid/ http://blog.mountaingoatsoftware.com/the-forgotten-layer-of-the-test-automation-pyramid http://benhutchison.wordpress.com/2008/03/20/automated-testing-2-the-test-pyramid/ http://www.informit.com/articles/article.aspx?p=1433906&seqNum=3 http://blogs.agilefaqs.com/2011/02/01/inverting-the-testing-pyramid/

  11. Test Pyramid Closer to Code Closer to Customer Manual Tests End-to-End Automated Tests* Integration Tests* Unit Tests*

  12. Uniting Dev and Test • Lack of communication often leads to trouble • Duplication of tests • Tests performed at an inefficient layer • Bugs are found later than they could be • Need to break teams out of old mindsets and get them to see the other side of the picture

  13. What a Tester Sees • Usually sees a functional spec • Simple login form that shows informative errors • Test cases • Email address not associated/a password • Does the page work in IE6? From facebook.com

  14. Test Cases

  15. Michael Hunter’s “You Are Not Done Yet” From http://www.thebraidytester.com/downloads/YouAreNotDoneYet.pdf

  16. What a Front End Developer Sees <html> <jsonSubmit validate email and password are not null> <form onSubmit login> <input email> <input password> </form> </html>

  17. What a Back End Developer Sees def login if email is nil redirect to login_form, set error_message to “Please enter email” if password is nil redirect to login_form, set error_message to “Please enter a password” hash_password = hash(password) db_password = query_db(email) if db_password is nil redirect to login_form, set error_message to “Invalid email or password” if hash_password != db_password redirect to login_form, set error_message to “Invalid email or password” set cookie, redirect to logged_in_page end

  18. Test Cases

  19. Uniting Dev and Test • Simply consolidating lists can help reduce testing while maintaining quality levels • May actually be able to increase quality – new perspectives spark new tests

  20. Test Pyramid Test Dev Dev & Test Manual Tests Manual Tests Manual Tests End-to-End Automated Tests* End-To-End Automated Tests* End-To-End Automated Tests* Integration Tests* Integration Tests* Integration Tests* Unit Tests* Unit Tests* Unit Tests*

  21. What Layer for What Test • Once a united set of tests is created, next step is to decide where to put the tests • Based on what you want to verify, try and find the layer that allows you to do this with minimal interference

  22. Push Testing As Low As It Can Go!

  23. Benefits of Lower Level Testing • More efficient: Tend to run in less time – no cost of starting a browser or user wait times • More reliable: GUI tests fail inconsistently more often than unit/integration tests • Easier debugging: less components are involved so root cause is faster to fine

  24. Multiple Layers • Some tests will belong in multiple places • Time to start thinking about what to verify! • Often the same setup will need to be verified at multiple layers for different reasons • Sometimes a little redundancy is not a bad thing

  25. Email address not associated/no password Unit/Integration GUI/Manual Verify that JS error message is shown and no request is sent to the server Verify that with JS disabled, the server error is well formatted on screen • Verify that the “Please enter a password” error message is returned

  26. How to pick a layer • Think about what is actually being tested • Find the layers of code that are actually involved • Requires a good understanding of the architecture • Best done as a collaboration between dev and test • Push the testing as low as you can • The higher the layer, the harder you should think about how to group tests together

  27. Test Matrix

  28. Test Matrix • Becomes the basis for testing • Can easily divide work to be done now without overlaps • Need to remember the verification may differ at the different layers • Document may only live for the life of a few iterations • A little redundancy is ok

  29. Manual/Exploratory Testing • Often undervalued in an agile environment • Very useful for helping to answer the unknowns: “Does the page work in IE6?” • May lead to new tests to automate • Automation is for tests you want to continually run • Exploration helps ensure the gaps between those tests work

  30. Group Higher Level Tests • Think about the true goal of the tests • Often, they will be layout related as opposed to core functions • Don’t need to test 10 error messages, just the ones that push the boundary cases • Longest message, message with unusual characters …

  31. Test Pyramid Manual Tests End-to-End Automated Tests* Integration Tests* Unit Tests*

  32. In Practice • Need a good understanding of the underlying architecture • Get you dev and testers working together • Bullet point the tests that you think are valuable • Determine the layer for each test • Aggressively group the higher level tests together • Don’t underestimate the benefits of exploratory testing • Divide and conquer the work!

  33. Thank you!

More Related