An overview of DevOps from The Phoenix Project

1. An overview of DevOps from The Phoenix Project Dean Capps April 2015

2. The Phoenix Project • The book is a fictional account as experienced by Bill Palmer. The Company, Parts Unlimited, is facing challenges, both external and internal. On a Tuesday morning, Bill is called into the CEOs office and informed that the executive hierarchy of the IT department have decided to “pursue other opportunities” and that he is the new VP of IT. • Parts Unlimited is counting on the Phoenix Project to rescue the company. Commitments have been made by the CEO internally and externally. • Bill has a few months to overcome all the challenges, stabilize the IT systems, deliver Project Phoenix and effectively rescue the company. • The book contains 35 chapters that read like a geeky thriller with many characters that anyone working in a large organization will readily identify and identify with. • This presentation is based on “The Phoenix Project Resources” at the end of the book which present the subject matter in a more scholarly format. • I have tried not to interject any personal comments or biases in this deck. All the information is directly from the book summarized to the best of my ability • Special thanks to my friend and colleague Ron McEwan for lending me this book and encouraging me to read it. Created April 2015 - Dean Capps

3. Origin of DevOps • The manufacturing revolution of our age • Instead of optimizing the transformation of raw materials to finished goods we are converting business needs into capabilities and services that provide value for our customers • In the 1980s the goals were • Protect sales commitments • Have inventory on hand to satisfy demand • Control manufacturing costs • Reduce inventory to reduce costs • Above leads to conflict which was resolved by adopting Lean principles • Reducing batch sizes • Reducing work in progress • Shortening and amplifying feedback loops Created April 2015 - Dean Capps

4. Origin of DevOps • Apply the Lean principles to IT • DevOps coined by Patrick Debois and Andrew Shafer in 2008 • Common usage in 2009 at the Velocity Conference • Presentation by John Allspaw & Paul Hammond – “10+ Deploys Per Day: Dev and Ops Cooperation at Flickr” • In the Phoenix Project, the term DevOps is • The outcome of applying Lean principles to IT • Principles based on more than a century of sound management practices • Instead of being applied to the transformation of physical goods, used here to accelerate flow of work through Product Management, Development, Test, Operations and Security • How does agile fit in? • DevOps benefitted from Agile • Small teams operating with high trust • Small batch sizes • Smaller more frequent releases Created April 2015 - Dean Capps

5. The chronic conflict • A core chronic conflict between Development and Operations preordains failure for the entire IT organization, as well as the organization it serves. • Left unchecked, the conflict • increases time to market • creates longer and more problematic deployments • increases the number of Sev 1 outages • operations becomes increasingly buried with unplanned work Created April 2015 - Dean Capps

6. Break the cycle with DevOps • Already adopted by high-performing organizations such as Amazon, Google, Twitter, Etsy, Netflix etc. • Adopt a set of techniques called DevOps • routinely deploy 100s or 1000s of changes per day • while preserving world-class reliability, stability and security • code deployment lead time measured in minutes or hours • allows them to innovate and out-experiment their competitors • all of this with higher quality and better customer outcomes Created April 2015 - Dean Capps

7. Why do DevOps? • Competitive advantage • Faster feature time to market • Increased customer satisfaction • Increased market share • Increased employee productivity and happiness • Allows the organization to win in the market • Technology has become the dominant value creation process and an increasingly important means of customer acquisition Created April 2015 - Dean Capps

8. Why do DevOps? • Hallmarks of high-performers • Always “accelerate from the rest of the herd” • In 2009, ten deploys per day was fast • In 2012, Amazon average 23,000 per day • 2012 Puppet Labs “State of DevOps Report” • High performing companies • 30x more frequent code deploys • 8000x faster code deployment lead time • 2x the change success rate • 12x faster MTTR • High performers had lead times measured in minutes or hours; lower performers measured in weeks, months or quarters Created April 2015 - Dean Capps

9. Why do DevOps? • By breaking the chronic conflict cycle high performers are • Deploying features more quickly • Delivering these high levels of reliability actually requires that changes be made more frequently • Better IT and organizational performance • 2x more likely to exceed the below goals: • Profitability • Market share • Productivity Created April 2015 - Dean Capps

10. What it feels like to live in a DevOps world • Product owners, Development, QA, Operations and InfoSec work together relentlessly to help each other and the overall organization win • Enable the fast flow of planned work into production while preserving stability and security • Development spends 20% of time ensuring that • Work flows smoothly • Speeding up automated testing • Improving deployment infrastructure • All applications create useful telemetry Created April 2015 - Dean Capps

11. What it feels like to live in a DevOps world • Everyone needs fast feedback loops to • prevent problematic code from going into production • problems are quickly detected and corrected Created April 2015 - Dean Capps

12. DevOps culture • Everyone shares a culture that • Values each other’s time and contribution • Enables organizational learning and improvement • Dedicates time to putting those lessons into practice • Values nonfunctional requirements (e.g. quality, scalability, manageability, security, operability) as much as features because they are just as important to achieving business objectives • High trust, collaborative culture where everyone is responsible for their work. • Approval and compliance processes are the hallmarks of low-trust command-and-control management culture • High trust relies on peer review Created April 2015 - Dean Capps

13. DevOps culture • Hypothesis driven culture • Requires everyone to be a scientist • Take no assumptions for granted • Measure everything because our time is valuable • Build only things customers want • Paradoxically, deploying code becomes boring and routine • Instead of deploying at night/weekends with stress and chaos; deploy during the business day without most people noticing • Staff involved in deployments work regular hours rather than nights/weekends Created April 2015 - Dean Capps

14. Code deployment must become routine • Low stake feature rollout • Developers are consistently getting fast feed back • Automated unit, acceptance and integration testing being run in a production like environment • Continual reassurance that the code and environment will operate as designed • Always in a deployable state • After deployment, production metrics demonstrate that the • Code is working • Customer is getting value Created April 2015 - Dean Capps

15. Code deployment must become routine • High-stakes feature releases • Code delivering the functionality is already in production invisible to the customer, but enabling the features for internal testing • New features made slowly visible to small segments of customers • Automatically rolled back if errors occur • When we have confidence that the feature(s) is working; expose to the next segment of customers • Rollout is • controlled • predictable • reversible • low stress Created April 2015 - Dean Capps

16. Code deployment must become routine • High-stakes feature releases • Reduced deployment risks • Because we can deploy quickly we can experiment in production • Test out business hypothesis for every feature we build • Iteratively test and refine our features in production using feedback from customers Out-experiment the competition to win in the market place! “Features are always a gamble. If you are lucky, 10% will get the desired benefits. So the faster you can get those features to market and test them, the better off you’ll be.” – Erik Created April 2015 - Dean Capps

17. The three ways explained • The First Way • Left to right flow of operations from Development to Operations to the Customer • To maximize the flow we need • small batch sizes and intervals of work • never passing defects to downstream work centers • optimize for global goals • Practices include • continuous build, integration and deployment • creating environments on demand • limiting work in process • building safe systems and organizations that are safe to change A note on “left to right” At multiple moments in the book, our protagonist (Bill) meets with a Yoda like character (Erik) on a catwalk above the manufacturing plant. From this vantage point, Bill observes that raw materials enter the plant on the left and finished goods exit on the right. Created April 2015 - Dean Capps

18. The three ways explained • The Second Way • Constant flow of fast feedback from right to left at all stages of the value stream, amplifying to ensure that problems do not repeat and to enable faster detection and recovery • To maximize the flow we need • Create quality at the source • Create or embed knowledge where we need it • Practices include • Stopping the line when our builds and tests fail • Constantly elevating the improvement of daily work • Creating fast automated test suites to ensure that code is always in a deployable state • Shared goals and pain between Development and Operations • Pervasive telemetry so that everyone can see whether code and environments are operations as designed and that customer goals are met Created April 2015 - Dean Capps

19. The three ways explained • The Third Way • A culture that fosters two things • Continual experimentation • Requires taking risks and • Learning from success and failure • Practices include • Creating a culture of innovation and risk taking as opposed to fear or mindless order taking • High trust as opposed to low trust, command-and-control • Allocating 20% of Development and Operations towards nonfunctional requirements • Constant reinforcement that improvements are encouraged and celebrated The third way is all about ensuring that we are continually putting tension into the system, so that we are continually reinforcing habits and improving something. Created April 2015 - Dean Capps

20. Top DevOps Myths • DevOps replaces Agile • Compatible • Logical continuation of Agile • Agile is not a prerequisite for adopting DevOps • DevOps replaces ITIL or ITSM • IT Infrastructure Library or IT Service Management • In order to accommodate the faster lead times and higher deployment frequencies many areas of ITIL require automation (CM, configuration and release processes) • DevOps means NoOps • DevOps puts more responsibility on Development • Requires many Operations tasks to become self-service • Automate rather than be ticket based (e.g. get a production like environment) • DevOps is only for open source software • Principles are universal • Independent of the underlying technology Created April 2015 - Dean Capps

21. Top DevOps Myths • DevOps is just “infrastructure as code” or automation • More than just automation • Requires shared goals and pain • DevOps is only for startups and unicorns • For any organization that must increase flow of planned work • While maintaining quality, reliability and security for the Customer • Many of the DevOps unicorns were once horses • Twitter struggled to scale • LinkedIn struggled with problematic deployments after their IPO • Facebook in 2009 was at the breaking point for infrastructure operations Created April 2015 - Dean Capps

22. The Four Types of Work • Because work can be assigned to people in many ways (email, phone, text etc.) we want to make visible our existing commitments. There are 4 types of work that IT does: • Business Projects • Development projects • Typically run by PMO • Internal IT Projects • Infrastructure wither external or internal • Not centrally tracked • Reside in the DBA Manager, Storage Manager etc. • Creates a problem because there is no way to identify committed capacity • Changes • Generated from the previous two types of work • Tracked in a ticketing system • Unplanned work or recovery work (Firefighting or anti-work) • Operational incidents or problems caused by the previous types of work • “Like matter and antimatter, in the presence of unplanned work, all planned work ignites with incandescent fury, incinerating everything around it” • The IT capacity death spiral – Not paying down technical debt Created April 2015 - Dean Capps

23. The wait time/resource busy graph In the book, Bill rapidly realizes that Brent, one of the senior engineers is overcommitted and a number of people and projects are depending on him to be available. • On the x-axis is the % busy for a given resource • On the y-axis is the approximate wait time (queue length) • As resource utilization goes past 80%, wait time goes through the roof • Assume that all resources are 90% busy, the graph shows a wait time of 9 hours. If there are 7 handoffs (work centers or resources), the total wait time is 63 hours! • The author feels that this is the most effective way of communicating the catastrophic consequences of overloaded IT workers and the fallacies of using typical project management strategies for IT operations. Created April 2015 - Dean Capps