Fast, Reliable, and Cheap: Quality Assurance Director of Harvard Business School IT

1. An ATF story Fast, reliable, and cheap Director of Quality Assurance,Harvard Business School IT Sukeshi Shah Poornima Sanivada Jeffrey Manville Senior Quality Assurance,Harvard Business School IT ServiceNow Systems Administrator,Harvard Business School IT

2. This presentation may contain forward-looking statements. These statements are intended to be covered by the safe harbor provisions contained in the U.S. Private Securities Litigation Reform Act of 1995. These statements reflect the current beliefs of ServiceNow and are based on current information available to the company as of the date hereof. ServiceNow does not assume any obligation to update the forward‑looking statements provided to reflect events that occur or circumstances that exist after the date on which they were made. The forward‑looking statements in this presentation are subject to various risks and uncertainties that could cause actual outcomes and results to differ materially and adversely from those expressed in such forward‑looking statements. Further information on factors that could affect the company’s financial and other results is included in the filings we make with the Securities and Exchange Commission from time to time, including those discussed in our most recent Annual Report on Form 10-K. The information in this presentation on new offerings, features, or functionality is intended to outline our general product direction and should not be relied upon in making a purchasing decision. Safe harbor notice for forward-looking statements

3. Speaker introduction • Sukeshi Shah • Poornima Sanivada • Jeffrey Manville Director of Quality Assurance Harvard Business School IT sushah@hbs.edu Senior Quality Assurance Harvard Business School IT psanivada@hbs.edu ServiceNow Systems Administrator Harvard Business School IT jmanville@hbs.edu

4. Agenda Team dilemma  6 1 2 3 4 5 Tester’s ATF strategy Developer’s ATF experience Impact to ServiceNow SDLC Closing remarks Q&A

5. Team dilemma Poornima Sanivada

6. QUALITY vs. QUANTITY HBS ServiceNow – 2 week Sprint Cycle DEV Instance TEST Instance STAGE Instance PROD Instance Story Grooming • 5 stories accepted Code & validate 7 business days Deploy & validate 1 business day Skip Deploy Go Live Few stories accepted Slow defect detection Targeted testing (partialregression suite) Skip staging the deployment steps

7. Manual regression test effort per iteration

8. QUALITY vs. SPEED HBS ServiceNow – Upgrade Cycle DEV Instance TEST Instance STAGE Instance PROD Instance Upgrade Code remediation Coding/Testing iterations 20 business days Regression & UAT 15 business days Deploy & Smoke Test 1 day Deploy Go Live 7.5 days regression test effort per iteration Very slow defect detection More repetitive iterations cause test fatigue

9. Tester’s ATF strategy Sukeshi Shah

10. Recommended prerequisites for implementing ATF • Testcase has repeatable execution steps and results • Testcase is specific and order independent • Create naming convention (Test IDs) • Test suite is a meaningful grouping testcases

11. ATF Testcase design optimization DO NOT create an ATF testcase by coding every step executed in a manual test workflow. • Example 1 • More steps slow execution time • More steps create greater instability • Creates unnecessary redundancies with the test suite DO create ATF testcases by leveraging inserted records at targeted points. • Example 2 • Optimizes execution time • Achieves greater stability • Isolates functional behavior to be tested DO execute and monitor behavior.

12. Example 1 – Excess coding steps

13. Example 2 – Leverages Record Insert

14. Relationship between Test names and Reporting • Concise test description • Identifies test coverage • Identifies version • Test ID • Identifies test component • Identifies Test # • Maps to Update Set name

15. Testcase Management and traceability DEV Instance Date: 2019.MAR Test ID: CHG_403 Version: v4 TEST Instance Date: 2019.JAN Test ID: CHG_403 Version: v3 PROD Instance Date: 2019.JAN Test ID: CHG_403 Version: v3

16. ATF Suite Health and Maintenance • Keep current and relevant • Refactor testcases when patches or functionality changes require • Delete testcases that are not relevant or become redundant • Add new cases selectively • Keep runtime < 30 minutes to encourage frequent usage • Monitor results daily to ensure accuracy and stability • Eliminate false positives (< 1%) • Eliminate false negatives (< 1%) • When a testcase misbehaves, then diagnose and correct within 24 to 48 hours

17. Developer’s ATF experience Jeffrey Manville

18. Testing with ATF Types of Tests from Developers Unit Tests Model Tests • Write fast, descriptive unit tests using the Jasmine Test Framework • Quickly run and rerun tests as development progresses • Maintain larger, more complex scripts • Provide guarantees that an application continues to function as expected • Responding to failing tests BEFORE hand-off to QA Support small iterative changes introduced during development Guarantees individual features that make up an application Source: Information goes here

19. Development Feedback Cycle Manual Testing Vs ATF • Independent of QA resources • Waiting minutes for failure signal from ATF ManualTesting ATF • Heavily dependent on QA resources to test • Waiting for days for QA Signal

20. Validating Notifications Custom Step Configuration: testing notification triggers

21. Validating Notifications Custom Step Configuration: Step Execution Script

22. Custom Step Configurations: adding the test step

23. Custom Configurations Helpful configurations to your ATF application Source: Information goes here

24. Impact to ServiceNow SDLC Poornima Sanivada

25. Test execution effort—ATF vs. manual Minutes vs. days

26. ATF impact to 2-week Sprint Cycle Provides higher QUALITY with higher QUANTITY DEV Instance TEST Instance STAGE Instance PROD Instance Story Grooming Stories accepted Code and validate Deploy and validate • Deploy and validate Go Live 5 Storiesaccepted 7 Business days 1 Business day Skipped Go Live BEFORE ATF 12 Storiesaccepted 7 Business days < 60 minutes <1 Business day Go Live AFTER ATF 140% increase in throughput Rapid defect detection (94 tests validate 2x daily) 87% reduction in testing time Fast Go/No-Go Decision Stage deployment steps and validate in < 30 minutes

27. ATF impact to upgrade cycle Provides higher QUALITY with faster SPEED DEV Instance TEST Instance STAGE Instance PROD Instance Upgrade Code remediation Coding/testing iterations Regression and UAT • Deploy and smoke test Go Live 20 Business days 15 Business days 1 Business day Go Live BEFORE ATF ATF Tests detect areas for remediation 5 Business days < 4 Business days <1 Business day Go Live AFTER ATF 75% reduction in time to release, while ensuring qualitytest coverage

28. Closing remarks Jeffrey Manville Poornima Sanivada

29. Looking forward… Lessons learned Wish list Investment in initial strategy will reduce amount of time spent refactoring. Ability to configure a timeout session. 2 1 Carefully evaluate the impact of SN patches. At times, patches have adversely impacted ATF functionality. Interactive ATF Dashboard. 3 2 Continuous ATF test maintenance is critical for ensuring reliable results. 4 Mature Test Driven Development practice already existed within the team. 1

30. Questions Sukeshi Shah: sushah@hbs.edu Poornima Sanivada: psanivada@hbs.edu Jeffrey Manville: jmanville@hbs.edu