110 likes | 289 Views
Red Hat on Jenkins: By the numbers. Nam Duong Red Hat, Inc. https://openshift.redhat.com. Agenda. Products currently leveraging Jenkins Deployment size: # of Masters/Slaves/Jobs in use across Jboss Middleware QE Top 10 list of interesting Plugins Code contributions
E N D
Red Hat on Jenkins: By the numbers Nam Duong Red Hat, Inc. https://openshift.redhat.com
Agenda • Products currently leveraging Jenkins • Deployment size: # of Masters/Slaves/Jobs in use across Jboss Middleware QE • Top 10 list of interesting Plugins • Code contributions • Experiences/Pitfalls/Lessons learned • Best Practices • Demo: • Production/Staging server (sheer numbers) • OpenShift on Jenkins • EC2 and GIT plugin demo
Red Hat products • Products currently leveraging Jenkins • JBoss offerings: • Enterprise Application Platform • JBoss Developer Studio • SOA • JBoss Portal • JBoss OSGI • Many, many others • OpenShift – OpenShift is our Platform as a Service (PaaS) offering. Our Jenkins instances are running in the cloud, hosted on OpenShift Flex and used to build and test OpenShift
Deployment Size • JBoss Middleware Team: • 1 Master • IBM System x3550 M3, 2x E5640 CPU, 24GB RAM, 2x SAS 73GB 15K-RPM in RAID1 • Slaves connected 113 (various OS: linux, Win, Solaris, HP, AIX) • ~4700 Jobs configured (~1000 running/month) • ~95% of Middleware test automation is called/executed via Jenkins • OpenShift Flex Team: • 1 Master • 7.5 GB, 4CPU, 10GB EBS (RHEL 5.6) • 10 Slaves Building EC2 Images and 5 slaves running tests • 20 jobs (RHEL 6.1) • ~90% of our automation is executed via Jenkins
Interesting plug-ins used • Management (Groovy*, Disk Usage*, Build Publisher*, Workspace Cleanup*) • Cloud Control (EC2 plugin, DeltaCloud*) • Enterprise focused plugins (SmartFrog) • Code Coverage (cobertura, emma) • Source control (GIT/SVN/CVS) *Represents code contributions
Code Contributions • Memory leaks: Involved in diagnosis, resolution • In the Hudson timeframe, we contributed to stability through reporting of bugs/instability (large deployment) • First ones to run into these types of problems (pure volume) • LTS (long term support - Jenkins stable branch) • Development and maintenance of plugins (see previous) • reporting bug + bug fixes
Experiences/Pitfalls/Lessons learned • 1 master may not handle all jobs at once - potentially provide failover functionality on Jenkins • Jobs getting stuck due to polling console output through ssh. Need to limit jobs from writing to console. • Disk Utilization issues - Overcome through quotas set on jobs • Logging was too verbose, Jenkins instance stalled due to low/no disk space • OOM issues when trying to view the giant logs - fix in progress
Best Practices • Set limits on memory usage (quota) of slave • Set limits (quota) on Disk usage for jobs • Tools on shared NFS drive – for maintenance and replication/recovery • Memory – lots of memory to buffer incremental growth of Master • Jenkins master and all slaves under CFEngine control (config management/recovery) • Like all reliable services, use Dev/Stg/Prod increments. • Can scale vertically, but has anyone tried scaling horizontally?
Demo • Production instance within corporate firewall: • Production: <Screenshots available upon request> • Staging our plugins • Staging: <Screenshots available upon request> • Jenkins running on OpenShift Flex; testing OpenShift Flex • <Screenshots available upon request> EC2 and GIT plugin demo • Used to create a client testbed • Creates an EC2 instance, prints the IP address, pulls down all code bits
Links/Contact • Email: nduong@redhat.com • https://openshift.redhat.com • We appreciate your feedback of your user experience
Thank You To Our Sponsors • Coming Soon: The CloudBees Newsletter for Jenkins • Please complete the Jenkins survey to help us better serve the community • (bonus: a chance to win an Apple TV!)