Transforming IT Operations: A Survey of Effective Practices

1. Transforming IT Operations: A Survey of Effective Practices Shawn Winnington-Ball Information Systems And Technology 03 December 2013

2. Introduction • There are some fundamental problems in IT that people are working to solve • Let’s examine some of the ideas and approaches • Knowledge gleaned from various sources: books, blogs, articles • A selection of what I find compelling

3. IT/business • IT is a critical function in the achievement of business goals • Business goals have become IT goals • Past the point of no return where we can fallback to manual processes • IT risk is therefore business risk

4. IT/business • In our digital society, there is tremendous value in using IT to create novel ways of enhancing our experiences • Digitalization (Gartner) • Business success is tied to IT success, and how creatively and capably the IT hammer can be wielded

5. IT risks • What are some of the risks that might prevent us from achieving our IT goals? • There’s too much work to do already • Fixed culture: ‘we’ve always done it this way’ • Sufficiently resilient/secure IT infrastructure • Silo mentality: the right people aren’t talking about the right things • Insufficient understanding of true priorities

6. The approaches • From here on out, IT operations context • The Phoenix Project: IT is in the toilet, and the miraculous recovery • The DevOps movement: bury the hatchet • IT process improvement efforts, culture change

7. IT is a mess • The situation: too much to do, everything chaotic, messy, unordered • Where do you begin when overwhelmed? • Tough to build a house with a jumbled pile of bricks, lumber, screws and shingles • The right work isn’t getting done: inefficient practices and processes

8. Unclogging the pipes • Analyze active work, see the big picture • Who spends the time on this work? • Which of the work is repeatable? • Which of it requires specialized knowledge? • What are the organization’s true priorities and how does the work fit with them? Is there a disconnect?

9. Unclogging the pipes • Collect the work, categorize it • Projects, Infrastructure, Changes, Unplanned • Infrastructure development/maintenance work is internal project work: call as it much • 20000’ view: what are All The Things currently underway? • This is our Work in Progress, active tasks

10. Unclogging the pipes • Clear the backlog: what is preventing the work from getting done? • Constraints and bottlenecks • Systematically clear them • Low-hanging fruit: cease unplanned work • Underlying causes: why does IT break?

11. Unclogging the pipes • Steady ongoing changes, make them less prone to causing unplanned work • Technical debt, taking shortcuts now will cause pain later • Control the release of work into IT • Demand outstrips capacity: don’t auto-accept new commitments

12. Unclogging the pipes • Determine total IT capacity. What commitments can we reasonably take on? • Isolate key projects and freeze ongoing efforts for everything else • Identify the work that only one person does and standardize it, document the process • Elevate preventative work: if it breaks often it gets the most attention

13. Unclogging the pipes • ‘Setting the tempo by our constraints’ • Say NO now but say YES later once the backlog is clear • It’s easy to be honest about your capabilities when you have a clear picture

14. Free and clear • What can these ideas bring about? • Reduction in chaos • Ordered approach to work, priorities-based • No more uncontrolled change • Honest assessment of true capabilities

15. DevOps overview • What is DevOps? A collaborative approach to how IT development and operations relate • Tension between creating and maintaining • Development: fast, agile, creative • Production: stable, predictable, resilient • Reconciling different perspectives

16. DevOps overview • Borne from the Agile development movement: fast code release, quick sprints • Speed is of the essence: companies need to keep up with competition, provide value quicker and more often, more reliably • The DevOps philosophy is summed up in three guiding principles…

17. DevOps – First Way 1. Systems Thinking • Performance of the entire system • Fast flow of work: continuous integration, deployment: small legos not big bricks • Understand that value is generated in IT from left to right: development to production, always moving forward • ‘”Reduce friction, increase velocity”’ (Farr)

18. DevOps – Second Way 2. Amplify feedback loops • Bring developers closer to their live code: if sysadmin is on-call, why not the developer • Improve the duration between learning of and correcting failures • When the system is broken, fix it before completing the work itself

19. DevOps – Third Way 3. Culture of continual improvement and learning • Take risks, fail quickly, move on • Prevent failures from reaching production • The basis of improvement is practice and repetition: make it habitual and widespread • Test your supposed resilience: break things on purpose to see what happens

20. DevOps: the toys • Infrastructure as code: heavy use of configuration management • Versioned environments, automated deployments • Graph anything and everything • DevOps isn’t tools but they are invaluable to establishing the culture

21. The Visible Ops • Prescriptive guide based on ITIL • ITIL doesn’t tell you where to begin; daunting effort • Authors provide 4 distinct phases of process improvement • Case study based: what do the shining stars have in common?

22. The Visible Ops • “80% of outages caused by operator and application errors” • Cultural problems • Change management is made too tough • “Cowboy culture”; misplaced sense of agility • Reactive, always firefighting, never planning • Constantly chasing audit requirements

23. The Visible Ops • Characteristics of high-performing orgs • High availability as measured by MTBF and MTTR • High throughput of successful changes • Investment early in IT lifecycle: release mgmt • Visible audit controls • IT ops and security working closely, mentor/mentee • Low amounts of unplanned work • Server to admin ratio > 100:1

24. The Visible Ops • “Stabilize the patient” • Identify most problematic infrastructure • Publish change policy: Thou Shalt Not Touch • Create designated change windows • Use Tripwire to verify compliance • Create Change Advisory Board body comprising stakeholders, use change request tracking system • Initiate change management meetings (to authz changes) and daily change briefings (to announce)

25. The Visible Ops • “Catch and Release” & “Find Fragile Artifacts” • Interrogate all systems, ask many questions of them • Find the systems that are unique, scary, important, and historically problematic • Determine how many unique configurations you actually have • Document systems and services and interdependencies in a CMDB

26. The Visible Ops • Create a Repeatable Build Library • Infrastructure as fuses; replace, don’t fix • Engineer builds for fragile infrastructure • Reduce unique configurations in production • Create ‘Golden Builds’: system images • Identify lowest common denominators across the environment

27. The Visible Ops • Continual Improvement • Metrics: can’t manage what you can’t measure • Fact-, not belief-based management • MTTR and MTBF are key, affected by release stage planning efforts • Closed loop between phases 1-3 • Release, controls, resolution

28. LISA 2011 • SREs at Google: Tom Limoncelli • Disconnect between dev and prod, competition brings them closer out of necessity • Faster feature release, pent-up waterfall methods no longer suffice • Dev teams run their own services for 6+ months • SREs provide self-service to devs: systems, storage, bandwidth, monitoring, docs: videos, wikis, SLA metrics

29. LISA 2011 • Deployinator at Etsy: Erik Kastner, John Goulah • Speed and agility valued: 30+ code deploys/day • “Be wrong as fast as possible” • Graph everything that can be measured • The entire company is on IRC, up to CEO • Code push announcements are published via IRC bot

30. LISA 2011 • Puppet: Luke Kanies • A pep talk for an obstinate, slow-moving sector • Competition drives innovation: do it better and faster than the next person • Zynga was adding 1000 servers per week (!) • Cloud computing is independence and self-service, not doing it all yourself, relying on sub-contractors

31. LISA 2011 • Game Day: Jesse Robbins, Opscode • Things happen, adjust your response to them • Determine the MTTR on your own terms • Rules: • Preparation: goals: mitigate impact, reduce MTTR, MTBF • Participation: all hands on deck, everyone suffers together • Exercises: ‘trigger and expose latent’ defects, start small • Work up to full data centre outage! • Essentially positive outlook, can-do attitude

32. IT culture • Tools, tools, tools is the typical mantra • Discuss the ideas, habits and beliefs that underpin our approach to our jobs and IT • Technology is rapid, people aren’t • “Give People priority. If a few more projects spent a third or more of their time, effort and money on People aspects (consultation, collaboration, walkthroughs, training, pilots, training, coaching, training, support, feedback...) instead of Technology and ITIL consultants, we might have some more successful ITSM implementations.” (Rob England, itskeptic.org)

33. IT culture • How do you compel people to change their views and habits? • Address ‘how is this time any different?’ • Address ‘how does this affect me?’ and ‘what do I stand to gain from it?’ • Courage to tell it like it is: be honest and don’t avoid conflict out of fear • Be vulnerable, share your personal story

34. Conclusions • Many great ideas on how to advance IT operations to meet business goals • Perhaps we just need ideas to flourish in small pockets? • Can’t ordain cultural change: find places where it will grow and support the good ideas • Organize more places to connect like-minded people

35. Sources, inspiration • The Visible Ops, The Phoenix Project – Behr, Kim, Spafford • http://itskeptic.org (ITSM consultant, kinda grouchy, great critical perspective) • http://blogs.pinkelephant.com/troy (ITSM consultant, several years’ of blog material) • “SRE@Google Limoncelli” • “Opscode Gameday LISA 2011” • http://agile.dzone.com/articles/agile-its-second-decade-0 • http://itrevolution.com/learn-more-about-concepts-in-phoenix-project/ • http://itrevolution.com/nick-galbreath-on-integrating-information-security-into-devops/ • http://itrevolution.com/one-of-the-best-devops-talks-on-it-transformation-continuously-deploying-culture-by-rembetsy-and-mcdonnell-velocity-london-2012/ • http://itrevolution.com/the-three-ways-principles-underpinning-devops/ • http://itrevolution.com/video-of-my-2012-puppetconf-keynote/ • http://noelbruton.wordpress.com/2013/11/23/the-phoenix-project-exposes-itils-anti-management-backwardness • https://speakerdeck.com/atalanta/how-not-to-do-devops • http://venturebeat.com/2013/09/30/an-idiots-guide-to-devops

36. Questions swball@uwaterloo.ca