Continuous Integration in the Cloud with Hudson

1. Continuous Integrationin the Cloudwith Hudson Kohsuke Kawaguchi Jesse Glick Sun Microsystems, Inc. Hudson committers

2. Rise of Continuous Integration • Offload from people, push to computers $ computers us time

3. Spend more CPU power to help you • … even if it only helps a little • First on your laptops and workstations • IDEs are at the forefront • And then to the servers • a.k.a. “Continuous Integration” • More frequent build/test executions • Static code analysis tools • And more to come

4. Hudson https://hudson.dev.java.net/ • Open-source CI server at java.net • Emphasis on ease of installation and use • “java -jar hudson.war” execution • Configure everything from browsers • Extensibility • 140+ community-developed public plugins • By 150+ contributors • Estimated 13,000 installations

5. It basically does builds and tests • Check out the source code • Subversion, Perforce, Git, Mercurial, CVS, … • Do builds and/or tests • Ant, Maven, MSBuild, shell script, … • Record results • Binary, test results, code coverage, static analysis • Notify people • E-mail, IM, RSS, tray apps, IDEs

6. Localized to 8 languages

7. And hopefully more to come…

8. Adoption in all kinds of businesses

9. Doing Distributed Builds

10. Why Distributed Builds? • You need to use multiple computers because… • You need different environments • You need isolation • There’s only so much you can do with 1 computer

11. Before we talk about clouds… • Going virtual doesn’t solve… • Software installation problem • Node failure problem • Remote maintenance problem • … • What does Hudson do to help you with these?

12. Installing new slaves • For first 20 or so slaves, we did it manually • Insert CD, click, type, click, type, click, … • But that doesn’t scale • Then we automated • Available as “Hudson PXE Plugin”

13. Automated System Installations • Hudson + PXE plugin • ISO images of OS • Slaves • Power on, hit F12 • PC boots from network (PXE)

14. Automated System Installations • Hudson + PXE plugin • ISO images of OS • Slaves • Power on, hit F12 • PC boots from network (PXE) • Choose OS from menu • Installs non-interactively Your corporate IT guy & his DHCP server

15. Automated System Installations • Supports OpenSolaris, Ubuntu, CentOS, Fedora • Trivial with most Linux • Cooperate with Windows, too • Quite useful outside Hudson, too • No more broken CD drives • No more CD-Rs

16. Distributed builds with Hudson • Master • Serves HTTP requests • Stores all important info • Slaves • 170KB single JAR • Assumed to be unreliable • Scale to at least 100 • Link • Single bi-di byte stream • No other requirements

17. How master and slaves start talking • For Unix slaves, via SSH • Only need SSH and JRE on slaves • We just need a host name

18. How master and slaves start talking • For Windows, DCOM • We just need admin user name and password • No manual intervention • Works even from Unix masters

19. How master and slaves start talking • Via Java Web Start • When master cannot see slaves • A separate socket connection is made

20. Automating JNLP launch • Once started, can be installed as Windows service

21. Automating JNLP launch • Emulate the JNLP client headless $ java -jar slave.jar -jnlpUrl URL

22. Automated Tool Installation - JDK • JDK from http://java.sun.com/ • Hudson automatically chooses the right bundle • Always up to date with new releases

23. Automated Tool Installation - Apache • Ant and Maven from Apache

24. Automated Tool Installation - Custom • Download arbitrary archive and unpack

25. Automated Tool Installation - Custom • Run arbitrary shell commands • Can have variants by OS

26. Automated Tool Installation - Extensible • Write your own • Simple Hudson extension • Just write Java code to create tool on slave • In progress: • SCMs – Mercurial, … • Install from Subversion

27. Heterogeneous Cluster Challenge • Your builds/tests need to run in specific environment • Dependency on individual nodes hurts utilization jobs slaves WombatWindows test Windows #1 GlassFishWindows test Windows #2 Hudson Windows test Solaris#1 Hudson Solaris test

28. Labels to rescue • Label is a group of slaves • Tie jobs to labels jobs slaves WombatWindows test Windows #1 Windows GlassFishWindows test Windows #2 Hudson Windows test Solaris#1 Solaris Hudson Solaris test Windows #3

29. Forecasting failures • Hudson monitors key health metrics of slaves • Low disk space, insufficient swap • Clock out of synch • Extensible • Slaves go offline automatically • Catch problems before they break builds

30. Clean up mess after builds • Kill runaway processes • Daemons, background processes left by your build • Works on Windows, Linux, Mac, and Solaris

31. Load Statistics Monitoring

32. When it’s time to add more slaves

33. Hudson made this extensible • Hudson detects excessive workload • Hudson notifies plugins • Plugins can provision more slaves • … assuming that you have that infrastructure

34. And that brings us to clouds…

35. Amazon EC2: The Good • Pay as you go (10¢/h or so) • Loads on Hudson tend to be spiky • Programmable API • Instances launch at machine-speed • EC2 instances are forgetful

36. Amazon EC2: The Bad • Your data is still inside your firewall • Takes time to check out code • … or to archive build artifacts • Some data just can’t be moved • EC2 instances are forgetful • Can your tests run in parallel?

37. Hudson EC2 plugin • Built on top of typica* • What does it do? • Automatically provisions slaves on EC2 on demand • Picks the right AMI depending on demand • Starts slave agent • Shuts down unused instances * http://code.google.com/p/typica/

38. Putting it all together capacity # of executors queue length usage time

39. Hudson “Appliance” on EC2 • Run the master in the cloud too, if you like • Hudson on stock OpenSolaris AMI • Data stored persistently in Elastic Block Storage • Dynamically expandable thanks to ZFS • Online, too • Packaged as a wizard

40. Demo

41. Overloading your cluster

42. Hudson Hadoop plugin • Just a few mouse clicks to install • Turn every Hudson slave into a Hadoop node • Distributed file system • Automatic data replication (fault tolerant) • Nice for storing old artifacts, logs, test records, … • Map/reduce framework • Large scale test results analysis / datamining • More interesting work to be done in the future

43. Hudson Selenium Plugin • Selenium • Tests webapps by scripting browsers • Selenium Grid • Runs Selenium over a grid of computers • Allow Hudson labels to specify where to start browsers • Hudson & Selenium both need heterogeneous cluster

44. Selenium Grid Hudson slaves Hudson master (selenium hub)

45. Conclusion • CI is here to stay • We’ll continue to push more workload to servers • Hudson makes this easy for you • Reap the benefit of a cluster in multiple ways

46. Resources • http://hudson.dev.java.net/ • BOF-5105 “Hudson Community Meet up” • Today 7:45pm same room • Hudson booth inside Sun Pavilion • Support Subscription • hudson@sun.com

47. Kohsuke Kawaguchi Jesse Glick kohsuke.kawaguchi@sun.com jesse.glick@sun.com http://hudson.dev.java.net/

48. Agenda • Quick Hudson introduction • Doing distributed builds • How Hudson make distributed builds easier • Taking infrastructure to cloud

49. Setting up slaves • Keeping slaves consistent is a good thing • Particularly hard on heterogeneous environment • General system administration tasks • Network configuration • Package installations for native tools • Tools like Puppet or cfEngine are supposed to help • Install build tools in the cluster • Prepare tools on one file system • rsync to everywhere • This part of Hudson needs improvements

50. Hudson EC2 plugin usage • Tell Hudson your AWS account information