1 / 58

Wojciech Sliwinski Wojciech.Sliwinski@cern.ch Beams Department, Controls Group CERN

Continuous Integration and Quality Assurance for the Accelerator Controls codebase at CERN JINR/CERN/MEPHI Computing school DUBNA , 24 th October 2011 . Wojciech Sliwinski Wojciech.Sliwinski@cern.ch Beams Department, Controls Group CERN. Outline. Defining Software Quality

zeno
Download Presentation

Wojciech Sliwinski Wojciech.Sliwinski@cern.ch Beams Department, Controls Group CERN

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Continuous Integrationand Quality Assurancefor the Accelerator Controlscodebaseat CERNJINR/CERN/MEPHI ComputingschoolDUBNA, 24thOctober 2011 • Wojciech Sliwinski • Wojciech.Sliwinski@cern.ch • Beams Department, Controls Group • CERN

  2. Outline • Defining Software Quality • ContinuousImprovement • Context – Accelerator Controls Codebase • Applying QA & CI for the Controls Codebase • Conclusions Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  3. Outline • Defining Software Quality • ContinuousImprovement • Context – Accelerator Controls Codebase • Applying QA & CI for the Controls Codebase • Conclusions Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  4. Software Quality (SWQ)… • We know it’s “a good thing” ™ • People think… • “I know it when I see it” • “I know the value of it” • But what are the actual characteristics of SWQ? • Do we have a common understanding of SWQ? • The lack of it normally appears as consequences… Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  5. 1985: Therac-25 – Software issues… • Radiation Deaths linked to SoftwareErrors • Killed 2 people • Injured dozens of others • http://www.ccnr.org/fatal_dose.html Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  6. 1996: Ariane 5 –floating point conversion error Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  7. 2007: Airbus A340 – Software issues … Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  8. 80s – The Kano Model (Marketing) Satisfied Delight, Surprise, Innovate Needswell fulfilled Needs not fulfilled Linear Performance Customer Satisfaction Basic Needs Met Dissatisfied Product Features Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  9. Early and Other Quality Definitions • Zen and the art of motorcycle maintenance, Robert Pirsig, 1974: • “If quality exists in an object, then you must explain why scientific instruments are unable to detect it” • “On the other hand, if quality is subjective, [existing onlyin the eye of the observer] then this Quality is just a fancy name for whatever you'd like.” • “Quality is not objective. It doesn't reside in the material world [...] Quality is not subjective. It doesn't reside merely in the mind.” • McCall et al, 1977 and Boehm et al, 1978 First elaborate studies on the notion of 'software quality’ • Quality factors – only indirectly measurable like “reliability” • Quality criteria – objective aspects that support the factors, like “uptime”. Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  10. IEEE SWQ Standards • IEEE Glossary of Software Engineering Terminology defines it as • “the degree to which a system, component, or process meets customer or user needs or expectations” • IEEE Std 1061-1992 IEEE Standard for a Software Quality Metrics Methodology • Does not actually prescribe any metrics! • Defines method to evaluate metrics that could be applicable to a particular case • Plenty more – eg. IEEE Std 730 and friends Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  11. ISO Quality Standards • ISO 9000 series of standards for quality management systems: • Process based • “the degree to which a set of inherent characteristics fulfilsthe requirements” • ISO 9126 Software Quality Characteristics and Metrics, 2001 • Comprehensive but … large & complex • 6 keyqualityattributes: • Functionality, Reliability, Usability • Efficiency, Maintainability, Portability Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  12. SWQ – Huge Set of Characteristics Functionality Suitability Accuracy Interoperability Security Functionality compliance Reliability Maturity Fault tolerance Recoverability Reliability compliance Usability Understandability Learnability Operability Attractiveness Usability compliance Efficiency Time behavior Resource utilization Efficiency compliance Maintainability Analyzability Changeability Stability Testability Maintainability compliance Portability Adaptability Installability Co-existence Replaceability Portability compliance Availability Maintainability Efficiency Portability Flexibility Reusability Integrity Testability Interoperability Reliability Robustness Usability Effectiveness Safety Productivity Satisfaction • Only a subset applies to all projects • Choose those that suit the project goals Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  13. McCalls’ categories of SWQ factors Source: J.A. McCall, P.K. Richards and G.F. Walters, Factors in Software Quality, RADC-TR-77-369, 1977, US Dep. of Commerce. Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  14. Internal & External Quality Attributes Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  15. The SW Quality “Iceberg” Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  16. The Maintainability Index (MI) MI = 171 – 5.2 x ln(aveV) - 0.23 x aveV(g’) - 16.2 x ln(aveLOC) + 50 x sin √2.4PerCM Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  17. MI applied to FreeBSD codebase KernelCodebase UserPrograms Moduleswith MI < 40 arerejected ! Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  18. Applying Quality • Computers don’t care about quality! • Metrics are tests for quality characteristics • Quality “clusters” • Testing is ensuring a quality of conformance • Bugs occur in clusters too! • Like testing, if quality is low, builds should fail • Quality characteristics must be part of the whole development lifecycle • Quality products only come from quality requirements, quality design, quality code, … Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  19. Applying Quality (Assurance) Software Quality consists of: • Engineering Methods (proper analysis, design, …) • Standards and Procedures (common to all) • Technical Reviews (eg. code reviews) • Configuration management (repositories, versioning) • Measurement (code, metrics) • Testing (unit, system, integration, acceptance) • Improvement (of the items above) • Quality is a way of life • If you think Quality is expensive, try an accident… Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  20. CapabilityMaturity Model (CMM) Levels* • Initial (chaotic, ad hoc, individual heroics) • the starting point for use of a new process. • Repeatable • some processes are repeatable, possibly with consistent results • Defined • the process is defined/confirmed as a standard business process, and decomposed to levels 1 and 2 • Managed (quantatively) • using process metrics, management can identify ways to adjust and adapt the process to particular projects • Optimized • process management includes deliberate process optimization/improvement. * Capability Maturity Model from SE Institute. Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  21. Outline • Defining Software Quality • ContinuousImprovement • Context – Accelerator Controls Codebase • Applying QA & CI for the Controls Codebase • Conclusions Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  22. ContinuousImprovement (CI) – Origins • Deming’s Plan-Do-Check-Act (PDCA) • The Toyota Production System • “Stop the line” quality control • About eliminating “waste” (obstacles) • A learning, non-judgmental whole team approach • Adopted by the Agile movement for SWD • Caveat: Development is an “empirical process” • Introduced when an organization recognizes it has arrived at some impasse Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  23. CI - How it works • “People (at all levels) freely discuss the information, issues, ideas, evaluate them, choose, plan and execute improvements” • Focuses on how things get done: • Standardized (and ing) common tasks • Only fixing possible issues • e.g.: repeated mistakes, time sinks, quality holes • Very suitable where tools, technology, external outcomes and requirements change often Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  24. CI - What it relies on • Objective information: • To evaluate current situation and processes • Assess applied improvements • Judge outcome as worthwhile, or retry • Demonstrate added value of CI against overhead • Improvement culture: • “Free speech” – everyone’s ideas are welcome • A real desire to try it and improve • Accepting there will not be total agreement • Replacing skeptical inaction with real experimentation  Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  25. CI – It can fail! • No one “owns” responsibility for implementation • No charter to make changes • No time allocated for improvement • No follow up by team or sponsors Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  26. Improvement Practices • Change Agents • AKA: Coaches, Monitors, Consultants, Champions • One or more floating people searching for improvements • Kaizen time aka “renovation days” • Team plans some improvements • All actions executed by all at same time • Reflection workshops • Regular meetings to decide on improvements • Apply one or more ideas • Keep what works Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  27. CI - Summary • Team driven incremental changes • Team evolves and improves together • Drives increase in quality, efficiency and satisfaction • Not a tool or technique – it is an attitude • Not leader oriented, instead peer based consensus • Keeps us ahead in best practice and professionalism Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  28. Bibliography • Software Requirements • K. Wiegers, Microsoft Press, 2003 • Code Complete: A Practical Handbook of Software Construction • S. McConnell, Microsoft Press, 2004 • Software Engineering: Principles and Practice • H. van Vliet, John Wiley & Sons, 2008 • Wikipedia page on SWQ ISO standard 9126. Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  29. Bibliography • The Toyota Way, J. K. Liker, 2004 • Agile and Iterative Development, C. Larman, 2004 • Sustainable Software Development, Kevin Tate, 2006 • Implementing Lean Software Development, M. & T. Poppendick, 2007 • Rapid Development, S. McConnell, 1996 • The Enterprise and Scrum, K. Schwaber, 2007 • Collaboration Explained, J. Tabaka, 2006 • Peopleware Papers: The Human side of Software, L. Constantine, 2001 Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  30. Outline • Defining Software Quality • ContinuousImprovement • Context – Accelerator Controls Codebase • Applying QA & CI for the Controls Codebase • Conclusions Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  31. Context - CERN Accelerator Complex Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  32. Context - The CERN Control Centre Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  33. CTRL CTRL T Controls Software Infrastructure PUBLIC ETHERNET NETWORK OPERATOR CONSOLES FIXED DISPLAYS OPERATOR CONSOLES A 3-tier architecture Presentation Layer • Presentation Layer • Graphical interactive applications • Fixed Displays • Business Layer • General purpose and specific Application servers • Device Layer • Front End Device servers • Communication to the equipment goes through Controls MiddleWareCMW TCP/IP communication services Client tier FILE SERVERS APPLICATION SERVERS SCADA SERVERS DB Business Layer CERN GIGABIT ETHERNET TECHNICAL NETWORK Server tier TCP/IP communication services TIMING GENERATION RT Lynx/OS VME Front Ends WORLDFIP Front Ends PLC T T T T CMW TCP/IP communication services Device Layer Resource tier T WorldFIP SEGMENT (1, 2.5 MBits/sec) T T OPTICAL FIBERS FIP/IO T PROFIBUS T BEAM POSITION MONITORS, BEAM LOSS MONITORS, BEAM INTERLOCKS, RF SYSTEMS, ETC… QUENCH PROTECTION AGENTS, POWER CONVERTERS FUNCTIONS GENERATORS, … ACTUATORS AND SENSORS CRYOGENICS, VACUUM, ETC… LHC MACHINE Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  34. Controls Software Codebase - Situation • ComplexAcceleratordomain • ~6 million LOC • Java/J2EE • ~200 developers (CERN + otherlabs) • ~1000 projects/modules • C/C++ • ~35 developers (CERN + otherlabs) • +20 major projects • Oracle database • Eclipse IDE + plugins • SVN + Maven + CommonBuild( • Atlassian Suite (JIRA, Fisheye, Crucible, etc.) • NowadaysencourageScrum • Focused on Software Quality Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  35. Controls Software Codebase - Situation • Lots of Code • ~6 million lines, 20’000+ classes. • … and stillgrowing Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  36. Situation – Self Assessment and Outlook • Unknown quality • Quality unchecked • No mentoring or guidance? • Limited standards • HighStaff turnover • Students, Project Associates, Fellows … • More projectscoming (PS renovation, …) • More service provision (looking after running services) • Will this cause future problems like maintenance? Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  37. Outline • Defining Software Quality • ContinuousImprovement • Context – Accelerator Controls Codebase • Applying QA & CI for the Controls Codebase • Conclusions Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  38. Objectives – improve Software Quality • Introduce quality improvement as an integral part of the everyday development work • Leverage tools to automate the process as much as possible • Establish guidelines and metrics to measure progress Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  39. Quality in the DevelopmentCycle • For each stage • Agreed mandatory activities and deliverables • Tools identified and integrated with IDE (Eclipse), giving immediate feedback Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  40. Design Reviews • Design meetings with external members • To verify the soundness of design in an early stage • To identify overlapping functionality • Promote a set of common components and libraries At the level of sub-components, main classes and design patterns UML: class and sequence diagrams Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  41. Code Reviews & CodeAnalysis • Goal: • Identify defects • Ensure maintainable code • Verify conventions are followed • Static code analysis tools toidentifycommon mistakes and bug patterns: • FindBugs • Checkstyle • Eclipse warnings • Person-to-person time consuming • Only for critical code • Mentoring of junior developers • Light-weight person-to-person code reviews with FishEye + Crucible Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  42. Author and reviewers Comments inline Files being reviewed Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  43. Reviewing Code (Crucible)

  44. The ‘bug’ line indicated A list of ‘bugs’ ‘Bug’ explained Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  45. Testing & ContinuousIntegration • General agreement on benefits oftesting and CI • To increase level of testing, unit tests mandatory deliverable of eachproject • >30% test coverage for non-trivial classes, measured with Clover • To discover inter-project issues early, ContinuousBuilds with Bamboo: • CI serverfromAtlassian • Compilesand runs unit tests • Builds fail on: • Low test coverage • Basic PMD rules Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  46. Continuous Integration and Test (Bamboo) Code metrics Test coverage for project Classes with highest risk

  47. Red = not covered Green= covered Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  48. Top/flop list generated (Work in progress) Triggered by changes in a dependency Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  49. Integration and System Testingwith CO Testbed • CO Testbed - mini-acceleratorlab • Hardware and Software • Core components from the AcceleratorsControlsduplicatedintothis Test Environment • System and Integration tests are executed by Bamboo CI server • Automaticexecution of testswith“real”system • Automaticdeployment of ReleaseCandidates Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

  50. CO Testbed Hardware in place TIMING FEC03 SERVER06 SERVER07 FEC01 FEC02 FEC05 FEC04 Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

More Related