Getting to Push Button Deploys

1. Getting to Push Button Deploys Moovweb January 19, 2012

2. Things Everyone Wants

3. Reality … is it done yet??? … is it done yet??? … is it done yet???

4. Gotta Have It! PROS • Increase velocity of change/responsiveness • Get Operations out of the loop • Buzzword compliance – aka Shiny!!!

5. PROS Increase velocity of change/responsiveness

6. PROS – Ops Gets Out of Loop • Less work for Operations • Developers feel more in control

7. Gotta Have It! CONS • Increase velocity of change • Shared responsibility • Not always easy

8. Make It So...

9. Components

10. Components

11. Components

12. Components

13. Components - MoovStack

14. CI • Jenkins ( maiden name is Hudson ) • Checks out a deploy branch from git • Runs a few sanity checks • Packages software • Uploads to software repo • Tags build in git

15. Jenkins http://jenkins-ci.org/

16. Jenkins

17. Jenkins

18. Rake

19. Packaging • But it's sooooo hard save yourself first.....

20. Using FPM https://github.com/jordansissel/fpm

21. Building a Gem

22. Right Tool Packages • Something that does not change often • Don't need to roll back quickly • Don't need more than 1 version at a time • You cache external repos or limit to just yours

23. Right Tool Gems • Fast roll back – gem cache • Easy setup of gem server • Need more than 1 version at a time • Bundler for dependencies • Bundle exec for peace of mind

24. Configuration Management • Puppet everything from bare metal/base image • Seperate data from manifests with an ENC • Max troubleshooting time is 20 minutes • Amazon free ”chaos monkey” service

25. Puppet – Start with a sane base

26. ENC – Use Groups! http://docs.puppetlabs.com/guides/installing_dashboard.html

27. ENC – Node Entries Inherit

28. ENC – Scripting Additions

29. ENC - Output

30. Manifests Contain ENC Variables

31. Templates Contain ENC Variables

32. Puppet - Tips • Execs hurt – try to avoid them • Recursive directories – use vcsrepo instead • Write your own types – just restart in between • Run mongrel or passenger • Use behind a load balancer for fun • Use reporting, but take the DB off of your master • Write your own facts and save lots of time

33. Puppet - Scaling

34. Mcollective – Do It NOW!! • Orchestration framework • Crazy Parallel • Write more ruby code... • Clients and Agents

35. Mcollective – Crazy Parallel 6380 nrpe checks across 370 hosts in 3.8 seconds https://github.com/maguec/devops/tree/master/mcollective/nrpe

36. Use Facts and Classes to Filter

37. Mcollective – Custom Agent

38. Mcollective – Custom Client

39. Mcollective - Tips • Remeber you have a machine gun pointed at your foot • Wrap command in IO.popen and log everything • Log to syslog • Use ApacheMQ • For EC2 tune your keep alives (registerinterval) • Remember you have a machine gun pointed at your foot

40. Mcollective - Scaling

41. Sinatra • Web app in 5 minutes • Much more minimal than rails • Written in Ruby so I can interface with Mcollective easily http://www.sinatrarb.com/

42. Sinatra – Addons • sinatra-respond_to - useful as API • Pony – send email from Sinatra • Twitter Bootstrap – make things pretty

43. Public Example https://github.com/maguec/mco_dashboard_example

44. Public Example

45. Really Simple

46. Sinatra - Tips • Run a few behind Nginx • Use helpers whenever possible • Log every push • Use it to integrate all your data

47. Actual Push Procedure

48. Dashboard Functionality • Push software • Audit software • Real-time monitoring check • Performance data integration • Log data integration • Troubleshooting tools for developers • Utilities for non-technical staff • Open trouble tickets • Cost information (in progress) • Userful links to other data sources

49. Oppy – an example

50. Oppy – an example