1 / 48

Build & Continuous Integration for C/C++ in Accelerator Controls

Jérémy Nguyen Xuan 27.02.2013 on behalf of BE/CO. Build & Continuous Integration for C/C++ in Accelerator Controls. BE-ABP/PH-SFT/ BE-CO workshop. Agenda. Introduction – Motivation and Goals Maven NAR – Build system for C/C++ Bamboo – CI server, automatic builds

pascha
Download Presentation

Build & Continuous Integration for C/C++ in Accelerator Controls

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Jérémy Nguyen Xuan 27.02.2013 on behalf of BE/CO Build & Continuous Integration for C/C++ in Accelerator Controls BE-ABP/PH-SFT/BE-COworkshop

  2. Agenda • Introduction – Motivation and Goals • Maven NAR – Build system for C/C++ • Bamboo – CI server, automatic builds • Deployment into Operations • Integration and System Testing

  3. Overall goal • Minimize Operations downtime due to software defects • Provide High quality software • Well tested • Maintainable • Unified modern development process for CO • CO Java development environment already mature • Common software development process • Good set of tools

  4. SW development process

  5. C/C++ Teams and Projects in CO • 5 Development Teams (FESA, CMW, TIMING, Drivers, Diamon) • ~30 developers at CERN • +20 projects on SVN • Many clients • At CERN • Outside CERN (GSI) • Lines of Code • > 250’000 • Supported platforms • LynxOS ,SLC5, SLC6, Windows • Java • ~200 developers • ~1000 projects on SVN

  6. DEP CMW DRIVER TIMING example-get tgm serializer util directory-client tim tgv drvrutil proxy FESA stomp UnitTestAllTypes_prj log log-stomp rbac rda core-framework cern-framework 3rd Party libxml2 boost (headers ppc4) curl (ppc4) omniThread beecrypt omniORB IceUtil 6

  7. Agenda • Introduction – Motivation and Goals • Maven NAR – Build system for C/C++ • Bamboo – CI server, automatic builds • Deployment into Operations • Integration and System Testing

  8. Generic Makefile • For Java, we already have • Guidelines / best practices • Recommended set of tools • Common build tool • C/C++ teams asked for the same recommendations

  9. SIP4C++ • Working group with various teams from BE/CO • Agreed on common standards and tools • Directory structure and naming • Google test/mock, Valgrind, Doxygen • Common implementation of Makefiles • Common set of compilation flags • Manifest implementation • Tracing message (who, when, where) • On-going work • Quality assurance (Coverity) • Post-crash analysis (Google BreakPad)

  10. Project Makefile.dep Makefile • Dependencies information • FESAFWK_VERSION = 0.0.1-alpha1 • FESAFWK_HOME = (INSTALL_LOCATION)/$(CPU)/fesa/fesa-fwk/$(FESAFWK_VERSION) • DEPENDENT_COMPILER_OPTIONS += -I$(FESAFWK_HOME)/include • DEPENDENT_LINKER_OPTIONS += -L$(FESAFWK_HOME)/lib -lfesa-core -lfesa-core-cern -lcmw-rda -lxml2 • Project information • PROJECT = fesaclass • PRODUCT = FesaTestEnvironment • LIB_NAME = FesaTestEnvironment • VERSION = 0.0.1 NFS Make.fesa Make.generic • Installation location • SVN url • Specific compilation flags • Compiler/Linker rules • Google test/mock support • Doxygengeneration • Valgrindintegration • Tag SVN • Install on NFS Make.common • cross-compilersdefinition

  11. Motivation for Maven • C/C++ build tool • Started ~1 year ago • Evaluating Maven as CmmnBuild successor • Opportunity to unify the build tools • Common software development process • Built a prototype with Maven NAR • Reuse same philosophy as Java

  12. What is ? • Buildtool • Build management, makingiteasy • Provideuniformbuildprocess • Encouraging best practices • Dependency manager • Versioning • Resolve and downloadartifacts • Deal withconflicts • Plugin based • Has numerous plugins • You candevelopyourown

  13. Maven - How doesitwork? Source Code Build Process Artifact

  14. process-sources Maven - Buildlifecycle validate Source Code generate-sources process-resources compile test-compile Artifact test package integration-test verify install

  15. Maven - Convention over configuration <project> <groupId>cern.testbed</groupId> <artifactId>testbed</artifactId> <version>1.0.0</version> <description>The TestBed focuses on testing Core Software Components.</description> <url>http://wikis/display/ABCO/Control+System+Test+Bed+Facility</url> <dependencies> <dependency> <groupId>cern.japc</groupId> <artifactId>japc-ext-cmwrda</artifactId> <version>[2.0.0,3.0.0)</version> </dependency> <dependency> <groupId>cern.japc</groupId> <artifactId>japc</artifactId> <version>2.11.0</version> </dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.8.2</version> <scope>test</scope> </dependency> </dependencies> </project>

  16. Maven - Convention over configuration <project> <groupId>cern.testbed</groupId> <artifactId>testbed</artifactId> <version>1.0.0</version> <description>The TestBed focuses on testing Core Software Components.</description> <url>http://wikis/display/ABCO/Control+System+Test+Bed+Facility</url> <dependencies> <dependency> <groupId>cern.japc</groupId> <artifactId>japc-ext-cmwrda</artifactId> <version>[2.0.0,3.0.0)</version> </dependency> <dependency> <groupId>cern.japc</groupId> <artifactId>japc</artifactId> <version>2.11.0</version> </dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.8.2</version> <scope>test</scope> </dependency> </dependencies> </project>

  17. Maven - Convention over configuration <project> <groupId>cern.testbed</groupId> <artifactId>testbed</artifactId> <version>1.0.0</version> <description>The TestBed focuses on testing Core Software Components.</description> <url>http://wikis/display/ABCO/Control+System+Test+Bed+Facility</url> <dependencies> <dependency> <groupId>cern.japc</groupId> <artifactId>japc-ext-cmwrda</artifactId> <version>[2.0.0,3.0.0)</version> </dependency> <dependency> <groupId>cern.japc</groupId> <artifactId>japc</artifactId> <version>2.11.0</version> </dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.8.2</version> <scope>test</scope> </dependency> </dependencies> </project>

  18. Maven - Repositories CERN Network Internet query query download download Maven Central repo.maven.org Local repository Nexus artifacts.cern.ch

  19. Maven NAR - Basic idea • Maven NAR • Maven plugin • Compiles native code for various architectures and linkers • Made some extensions • The idea is to separate the build tasks between Makefiles and Maven  Compilation handled by Makefiles  Dependencies & Versioning by Maven

  20. Maven NAR - Example Pom.xml <project> <groupId>cern.cmw.cpp</groupId> <artifactId>cmw-rbac</artifactId> <packaging>nar</packaging> <version>3.6.2</version> <dependencies> <dependency> <groupId>cern.cmw.cpp</groupId> <artifactId>cmw-serializer</artifactId> <version>1.3.0</version> <type>nar</type> </dependency> <dependency> <groupId>cern.cmw.cpp</groupId> <artifactId>cmw-util</artifactId> <version>1.2.1</version> <type>nar</type> </dependency> </dependencies> </project> mvnnar:makedep

  21. Maven NAR - Example R = /nfs/cs-ccr-nfsdev/vol1/u1/jnguyenx/demo_phsft/cmw-rbac-cpp/target/nar/ DEPENDENT_COMPILER_OPTIONS += -I$(R)cmw-serializer-1.3.0-noarch/include -I$(R)cmw-util-1.2.1-noarch/include DEPENDENT_LINKER_OPTIONS += -L$(R)cmw-serializer-1.3.0-i386-SLC5-gpp-static/lib/i386-SLC5-gpp/static -L$(R)cmw-util-1.2.1-i386-SLC5-gpp-static/lib/i386-SLC5-gpp/static -lcmw-serializer-lcmw-util Makefile.dep.L865

  22. Workflow Compilation phase Packaging phase Installation phase Makefiles Maven NAR Maven NAR Source code Dependencies Binaries NAR package Binary repository Resolvedependencies Maven NAR Maven NAR Dependency resolution phase

  23. Makefiles & Maven - Benefits • Dependencies and Versioning are automatically managed • Developers keep their habits with Makefiles • Enforced standards • directory structure • directory names • file names • Unification • development/release/deployment process • Google test/mock • Compilation flags • Traceability • Manifest • versioning • Sharing projects for organizations outside CERN becomes easy

  24. Agenda • Introduction – Motivation and Goals • Maven NAR – Build system for C/C++ • Bamboo – CI server, automatic builds • Deployment into Operations • Integration and System Testing

  25. Bamboo - CI Server • We use Atlassian tools: JIRA, Confluence, Crowd, Crucible, Clover and Bamboo • Good reports and notifications • Cascade builds • Several build agents are configured to build the code on different platforms

  26. 1.

  27. 2. 1.

  28. Agenda • Introduction – Motivation and Goals • Maven NAR – Build system for C/C++ • Bamboo – CI server, automatic builds • Deployment into Operations • Integration and System Testing

  29. Deploymentinto Operations • Deploy a binary on a target machine • Deploymenttool - inhouse • Common for all CO projects (Java & C/C++) • Instance descriptor in XML • Facilitates configuration & logging • Reproducibility • Backups and rollback • Installation notifications

  30. Agenda • Introduction – Motivation and Goals • Maven NAR – Build system for C/C++ • Bamboo – CI server, automatic builds • Deployment into Operations • Integration and System Testing

  31. Integration and System Testing • Approach • To fulfill our goals, a mini-accelerator lab was built with the following aspects: • Completely separated from the operational environment • Validate products before going in operations Note: Stagingdevelopment Development Testing Production Release Candidate Release

  32. Hardware First step to build our TestBed: Set up an environment where to install BE/CO software. 3 layers to simulate: • Client side • Middletier services • Back-end

  33. CO Testbed Hardware in place TIMING FEC03 SERVER06 SERVER07 FEC01 FEC02 FEC05 FEC04 Niall TC may 2009

  34. Software Selected only essential components for Controls to: • Limit the complexity of the system • Give more flexibility to tweak components • Detect defects more efficiently

  35. CO TestBed architecture

  36. Deployment The products were manually deployed • Time consuming Automatic deployment into the TestBed • Speed up development cycle • Faster feedback to the developers

  37. Our idea • Use SVN Branches to identify release candidates • Automatically build a new release candidate and its dependees • Store the binaries in a separate repository from the release one • Somehow deploy these binaries into the TestBed

  38. 1. SVN Branch 3. Nexus – Binary repository 2. Bamboo - CI server 4. Deployment

  39. Workflow

  40. Type of tests In our TestBed, we exercise • Functional testing • System integration testing • Backward compatibility testing

  41. Conclusions • Severalyears of experience in building Java SW • Developingcommon solution for Java and C/C++ • Standardize as much as possible • Functionaltestingwith the TestBed

  42. Nexus - Release Management • Nexus stores several versions of a product • development • releasecandidates • final release • Maven uses GAV (groupId, artifactId, version) to identify an artifact • Features • Search • Full integration with Eclipse IDE • RSS feeds • Etc…

More Related