StickyMinds.com and Better Software magazine presents…

1. StickyMinds.com and Better Software magazine presents… Four Tips for Managing Software Production With Geographically Distributed Teams Sponsored by Electric Cloud Non-streaming participants should call 1-866-761-8643 International Non-streaming participants should call 1-904-596-2362

2. Four Tips for Managing Software Production With Geographically Distributed Teams Martin Van Ryswyk

3. Once Upon a Time… • Developers worked in small groups in the same office • They integrated their software with each other in an ad hoc manner • When the software had to be integrated someone typed ‘make’ on The Build Machine and went home for the night…

4. And Then… • First: distributed teams • Mergers, acquisitions, etc. result in teams scattered across multiple locations • Then: contractors and outsourcing • Contract with service provider that assumes responsibility for portions of development lifecycle • On- or offshore • Now: offshore development centers • Many organizations maximizing benefits of offshore by hiring internal teams in less costly locations

5. Generated New Development Challenges • Communication challenges • Time-zone differences • Cultural differences • Workflow challenges • Lack of documented process • Handoff problems • Management challenges • Lack of visibility into remote teams’ work • Unable to track work done by contractors Translated to rework, less-than-promised cost savings and often poor quality

6. The First Wave of Solutions Design, Define, Code Build Test Deploy Software Creation Software Production • Requirements management • Multi-site SCM • Wikis • Project & portfolio management tools • Process & workflow tools • Collaborative development environments

7. Production Systems Are Now the Bottleneck • Development projects larger, more complex • Little visibility into status • No more “overnight” for builds • Broken build or failed test may cost a day to track and fix • The build grows to be a full-time responsibility: the build manager is born • Often expected to supply round-the-clock support for remote teams

8. Who is Impacted? • Developers • Developers want to be able to build and test the software themselves on all required platforms • Check-in and get instant feedback on integration problems • Would prefer pre-flight builds • QA • Needs fully-tested builds for their work • They need regular builds (perhaps once daily) that have undergone developer testing • Build & Release Team • Juggling many builds a day • Need real management tools • Everyone wants live build feedback, Perl + Excel isn’t enough

9. A True Story… • Engineering organization for $500M/year enterprise software product • 200+ engineers distributed across 12 locations • Including: CA, TX, MA, Ireland, India • Result of integrating many acquired companies • Core team in MA managed hardware and software for SCM, Build, Test

10. A True Story, con’t. • Remote teams had minimal hardware • Could do local builds, but had to queue for complete build/test cycles across all platforms • Company was spending $$$ to provision remote sites with duplicate hardware and administrators • MA team ran nightly build/test queue • Due to complexity, full build/test matrix could barely complete overnight • Remote teams had to manually request hardware and time for specific tests • India/Ireland teams typically a day late or a day ahead due to time differences • All communication done via email & conference calls • Only MA had real-time visibility

11. A True Story: Implications • Key frustrations: • Lack of access to a current, clean build for all teams • Lost productivity and schedule delays for remote teams • Integration nightmares • Didn’t find issues until MA team did full integrations • Remote teams need access, either local or remote, to all target platforms and test configurations • Wasting $$ on redundant hardware and remote admins • Manual system required constant tending by core team and provided no visibility into status or results • Hampered communication and eliminated collaboration Time and money wasted actually reduced cost savings expected from offshore development

12. What Can we Learn? • Distributed teams place extreme demands on manual systems • Front-end tools may not pay off if back-end is still bottleneck • With complex build/test matrices, provisioning distributed teams can be expensive • When there is no night, the traditional nightly build routine will not suffice

13. Getting There on Your Own • Faster turnaround • Make pre-flight builds accessible to developers • Buy lots of SMP hardware and try out GNU Make parallelism or manually parallelize the build • Visibility • Build an intranet page, integrate it yourself with the current build and source code management system • Resource management • Virtual environments for remote build/test to maximize existing hardware • Automation • Build ad-hoc script attached to SCM system to initiate builds upon check-in • Use cron, rsh for job scheduling

14. Tip #1 Things To Try: Faster Turnaround • Pre-flight builds • Set up an internal web page that allows a developer to fire off a build • The developer should be able to select: • Which branch to build… • … or the location of his personal source tree • Whether to run unit tests or not • The build runs and the engineer gets an email with the result • Start a new rule: developers run on demand builds before they commit

15. Things to Try: Faster Turnaround • Parallel builds • Look into tools that can speed up your build • Large SMP Machines (gmake –j) • Distributed builds (distcc) • Set an under-60 minute goal for a full build and smoke test • Caution: dependencies should be explicit or well-understood before attempting parallelism

16. Tip #2Things to Try: Visibility • Set up a system the feedback to the entire team whether the full build succeeded • an intranet or wiki page showing live build status • shown on a flat panel screen in large engineering locations • Scheduled builds start to be the heartbeat of engineering

17. Tip #3Things to Try: Resource Management • Virtual machines to provide access multiple configurations/platforms • Minimize additional hardware investment • If concerned that developers might “mess up” production environment: • Dedicate machines for pre-flight build/test • Provide remote access where possible

18. Tip #4Things to Try: Automation • Integrate your fast builds with your source code management system • Write a script - whenever the source code changes, run a build and test it • Generic job schedulers exist • Will need additional scripting, customization, and maintenance • Or, write your own scheduler • cron, rsh • Now the build is really the heartbeat of engineering

19. Before You Build Your Own… • …think about: • How much time do you have to write all that code? • What else could you be doing instead of building your own? • Who is going to maintain the system? • How long will this take to roll out? • How you are going to manage many builds per day? • How are you going to get the build from 4 hrs to 15 min? • How are you going to manage hardware failures? • What happens when you add another project/product?

20. Build or Buy: An Example

21. Requirements for a Distributed System • Supports frequent turnaround • Short builds and test cycles • Provides real-time visibility • Preferably via Web interface • Enables effective resource management • For hardware and other resources • As automated as possible • Reduce 24x7 reliance on individual administrators or build team Production system must be available anytime, from anywhere…

22. Introducing Electric Cloud

23. Software Production Management CODE BUILD TEST DEPLOY Software Creation ? SCM $2 B Software Production Dev ToolsMulti-Billion Test Creation$1.5 B

24. SPM Problems • Slower time to market • Lower quality • Lack of compliance • Roadblock to modern development techniques • Agile • Global teams • Virtual environments Resource intensive to maintain Slow Not Repeatable Not accessible to engineers Brittle Lost Productivity Not Standard

25. SPM Solution Electric Commander Electric Accelerator Electric Insight • Accelerate • Automate • Analyze

26. ElectricCommander ElectricCommander is a Web-based platform for managing distributed processes Easy to implement and evolve processes Faster throughput, better resource utilization Better visibility across the organization Facilitates centralization and reuse

27. Tie it all Together SW DEVELOPERS SW DEVELOPERS ENGINEERING MGR BUILD TEAM OUTSOURCEDQA Build Tools Unit Tests Deployment Tools Test Servers Build Servers Production Servers Virtual Servers SCM Tools

28. Builds Grow 15 min 6 min 10 min BUILD Cpp UNIT Deploy Coverage Package Doxygen Deploy C-Depend Static Analysis Lint

29. Builds Grow Cpp UNIT Deploy Coverage Package Doxygen Deploy C-Depend Static Analysis Lint 3 Hours 15 min 6 min 10 min BUILD

30. The Problem: Dependencies 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 BUILD Cpp UNIT Deploy Coverage Package Doxygen Deploy Dependencies (Hidden and/or Expressed) C-Depend Static Analysis Electric Accelerator Solves this Problem Lint 3 Hours 15 min 6 min 10 min

31. The Problem: Dependencies Cpp UNIT Deploy 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 Coverage Package Doxygen Deploy C-Depend Static Analysis Lint 15 min 6 min 10 min BUILD Dependencies (Hidden and/or Expressed) 3 Hours

32. Accurate, Fast Parallel Builds BUILD BUILD Cpp UNIT Cpp UNIT Deploy Deploy 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 010101010101010101 Coverage Coverage Package Package Doxygen Doxygen Deploy Deploy C-Depend C-Depend Static Analysis Static Analysis Lint Lint 15 min 3 Hours 15 min 6 min 10 min Builds up to 20X Faster Electric Cloud Exclusive

33. Real World Results -X- Build Time (hours) Build Time (hours) 10X 20X Before After Before After Build Time (hours) Build Time (hours) 8X 12X Before After Before After

34. ElectricInsight BUILD Understand the details of your build Enable further optimization

35. Customer Examples Networking Semiconductor Cellular ISV Other

36. Electric Cloud Vision • Solve all your Software Production needs: • Automation • Acceleration • Visibility • Remove Software Production as the bottleneck to faster time to market and higher software quality • Make Software Production: • Self-service: Evolve from scripts managed by experts to buttons pushed by developers • Reusable: stop creating from scratch again and again • Consistent: enable IT to offer software production as a service to developers Confidential - Please Do Not Distribute

37. Thank You • For More Information • Electric Cloud • 408.419.4300 • info@electric-cloud.com • www.electric-cloud.com Confidential - Please Do Not Distribute

38. Have a question for the speakers? Ask now. Q & A