Test Automation at the Scale of Google

1. Test Automation at the Scale of Google Baran Ozgul Sr. Test Engineer - Google AdSense Olivier Gaillard Sr. Software Engineer in Test - Google AdSense

2. Guessing Game How many test cases are executed per year at Google? 100 x 10^9 ~14 000 years of test execution per year

3. Agenda • Case study: AdSense Frontend • Google's Innovation Culture • The Google Scale • Developer Engagement • Testing Infrastructure • Web Testing at Google • Key learnings • Q&A

4. Google's Innovation Culture

5. "The only true failure is not attempting the audacious." Google's Innovation Culture "Launch early, launch often" • Take risks • Launch and iterate

6. Search: Idea-to-user in less than 48 hours Chrome: A new version every 6 weeks Launch and iterate Google+: 100 new features in first 90 days

7. The Google Scale

8. The "Google Scale" • 60 offices, 30+ countries • 1 giant code base everything built at latest check-in • 11K people in R&D • 10s of thousands of changed code lines / day • 10s of products, 100s of feature teams • Weekly to sub-daily release cycles How is this possible?

9. Paradigm Shifts! • Developer owns code • Test engineering = quality assurance • Reliance on people to scale • Test to prevent bugs • Spread thinly • Developer owns code + test • Test engineering = Productivity enablers • Reliance on common test infrastructure to scale • Capacity to fix bugs fast • Staff high impact projects

10. Developer Engagement

11. Developer engagement

12. Developer engagement

13. Developer engagement

14. Test Engineering "Engineering productivity" • Test leadership • Process automation • Code health signals • Test evangelism

15. Test Leadership • Make existing code testable • Implement the first N tests, lead by example • Use data to define testing priorities • Remove testing bottlenecks • Make it faster to write and run the tests

16. Process automation • Continuous integration • Automated releases • Release tracking • Production monitoring • Automated code quality checks

17. Code health signals Code coverage Latency (ms) If you don't measure it, you can't control it.

18. Metrics: Incremental Code Coverage

19. Codelabs & Techtalks ("yt: code health talks") Testing on the Toilet (TotT) Internal & external test automation events (GTAC) Testing intergrouplet Test certification Test Evangelism Make testing part of the development culture

20. Test Certified Programme • An indication test automation maturity • Show value every stage • “Test Certified” Levels 1: Setup tools – get familiar with testing 2: Write tests for all new code 3: Get all code under test 4: Improve test coverage 5: Invest in testable code

21. Case Study AdSense Frontend

22. Case Study: AdSense Frontend Before • Time until next release: months • Build broken for weeks • 5+ release candidates • Reliance on manual testing for release confidence • Near absent test automation coverage • One man to release them all

23. Case Study: AdSense Frontend Now • Developers write tests for their features • Affected tests run at every code change • Manual testing: only exploratory • Build gets fixed in hours • weekly releases • <2 release candidates per release • 6 auto-staging releases per day • Push-button releases

24. Test Infrastructure

25. Life of a code change (AdSense Frontend) code review Push to production minutes hours days

26. Life of a code change (AdSense Frontend) code review Submit code through presubmit queue tests fail? Push to production code ready for submission minutes hours days

27. Life of a code change (AdSense Frontend) code review Submit code through presubmit queue builcops/build breakers notified tests fail? Push to production code submitted impacted tests run code ready for submission minutes hours days

28. Life of a code change (AdSense Frontend) push to production dry run code review Submit code through presubmit queue builcops/build breakers notified tests fail? Push to production code submitted impacted tests run code ready for submission minutes hours days

29. Test enablers • test engineers worked on optimizing this process • makes it easier to create/maintain tests • developers are more likely to create more tests • lots of these steps are using a common infrastructure • search for "continuous integration google"

30. Web Testing

31. Browser Automation Selenium 2.0/WebDriver • Core browser automation platform (others exist) • A collection of language specific bindings to control a browser • Emulate true user actions • Support for major browser manufacturers

32. Browser Automation WebDriver driver = new FirefoxDriver(); driver.get("http://www.google.com"); WebElement element = driver.findElement(By.name("q")); element.sendKeys("Cheese!"); element.submit(); assertTrue(driver.getTitle().contains("Cheese")); How does this scale?

33. Browser automation Shard 1 Browser 1 Test Browser Shard N Browser1 Test Browser Test Runner Output Aggregator Shard 1 Browser 2 Test Browser Shard N Browser M Test Browser

34. Browser automation (AdSense Frontend) Shard 1 Chrome Server Test(s) Browser(s) Database ... Shard 1 Firefox Server Test Runner Test(s) Browser(s) Database ... Shard 1 Internet Explorer Server Test(s) Browser(s) Database ...

35. Browser Automation: Test enablers • Test engineers worked on making it easy to write/maintain browser tests • developers are more likely to more tests • Browser tests run at every affecting code change some costly tests are throttled

36. Key Learnings

37. Key Learnings • Push testing up-stream • Early developer testing brings biggest wins • Position ourselves as productivity enablers • Make/Hire developers passionate about testing • Invest in a great testing infrastructure • Reduce the cost of writing tests to zero • Automate as much as possible

38. Thanks "Google Testing Blog" "Google Test Automation Conference (GTAC)" "Google Engineering Tools" "WebDriver" or "Selenium" "GTAC" "Clean code talks" "Web Testing at Google GTAC" "Build Software at Google Scale GTAC"