1 / 31

Do It In Production

Do It In Production. Testing Where It Counts. Seth Eliot. Senior Knowledge Engineer, Test Excellence. About Seth. Digital Media Services. A/B Testing of Services. Petabytes Processed. Services and Cloud. The Future of Software Testing. Part 1 Nov 2011 Testing in Production. Part 2

judah
Download Presentation

Do It In Production

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Do It In Production Testing Where It Counts Seth Eliot Senior Knowledge Engineer, Test Excellence

  2. About Seth Digital Media Services A/B Testing of Services Petabytes Processed Services and Cloud The Future of Software Testing Part 1 Nov 2011 Testing in Production Part 2 March 2012 TestOps Part 3 July 2012 The Cloud

  3. Testing Measurement A quantitatively expressed reduction of uncertainty …about quality of a system under test… …based on one or more observations

  4. Testing in Production Real Users TiP Data Driven Validation Data Driven Decision Making Production Environments

  5. Real World Performance Monitoring JSI JavaScript Instrumentation 500 Million measurements per month

  6. What is Big Data? Velocity Volume MB  GB  TB Variety PB  EB  ZB Value

  7. Xbox Kinect Twitter

  8. What Does Data Have To Do With TiP? Use Data for Quality Assessment Biggest and Best Data is in Production Testing in Production (TiP) Production is Truth

  9. Why Testing in Production (TiP)? “Walking Directions” Find this with a unit test…. This route may be missing sidewalks

  10. 5 Million Metrics Historical trending and analysis Hadoop Grid Report: CPU, Network… Operational Data Store (ODS) System… Application… Business… …Metrics

  11. Engineers stay with code every step of the way Constant dogfooding with reporting tools Metrics find… Problems at Scale Metrics find… Feature broken for significant percent of users “This process works for Facebook partly because Facebook does not, by and large, need to produce particularly high-quality software” Really?

  12. Just Throw it in Production? Unit Testing Functional Testing Integration Testing Testing in Production Functional Testing Integration Testing …maybe less of this TDD Instrument Everything Metrics and Optics Performance and Scalability

  13. Two Types of Data-Driven Validation Passive Validation Active Validation Real … Users … Applications … Infrastructure Operational Intelligence Business Intelligence Synthetic Transactions Availability Reliability Performance

  14. Performance Availability From Azure

  15. Fault Injection Latency Monkey Chaos Monkey Chaos Gorilla Game Day

  16. Synthetics Have Risks Monkey Who's Got the Monkey? Who's Got the Monkey Now?

  17. Synthetics Have Risks

  18. You’ve learned a lot already… P A P P A

  19. Experimentation “To have a great idea, have a lot of them” -Thomas Edison A/B Testing Dogfood and Beta Exposure Control

  20. Experimentation at Google 1/3 2/3 “…dice and slice in any way you can possibly fathom” 1% launches Shadow launches

  21. Experimentation at Netflix 1B API requests per day Canary Deployment

  22. Science! A brief look at data science Even small data is useful data Watch out for sample bias Beware of averages

  23. How many years have you worked in software?

  24. Averages are Your Enemy

  25. Tools of Big Data: Hadoop Hadoop HDFS Map-Reduce ADD FCA ECC 1xA 1xC 1xF 2xA 0xB 3xC 2xD 1xE 1xF 1xA 2xD ADD FCA ADDFCAECC ADD FCA ADD FCA  1xE 2xC ECC ECC ECC

  26. Tools of Big Data: Cosmos Stores hundreds of petabyte On tens of thousands of computers Processing 2PB per day

  27. Hadoop on Azure …Hundreds of thousands of requests per second “We know we can't anticipate the 101 things that will go wrong, The only thing we can control is ensuring our team responds appropriately to those situations.“ – Jerry Hook, Executive Producer Halo HDInsight Xbox Console Halo Servers HDInsight Sharepoint Action Hack using modified Xbox “But even if you’re following the law, you can do things where people get queasy.” Target Power Pivot

  28. Availability (y) over time (x) Data Cosmos Machine Learning Predict 75% of dips 24 hours ahead of time c c c

  29. A set of observations to reduce uncertainty about quality of a system under test Testing The best observations are often in production Passive Active

  30. Thank you! Do It In Production Testing Where it Counts seth.eliot@microsoft.com ? blog: http://bit.ly/seth_qa me…. @setheliot

More Related