EMI Quality Assurance Processes (PS06-4-499)

1. EMI Quality Assurance Processes (PS06-4-499) Alberto Aimar (CERN)CERN IT-GT-SL Section Leader EMI SA2 QA Activity Leader

2. Outline • EMI context and QA objectives • Guidelines and Metrics • Tools and Testbeds • Reports and Reviews • Conclusions EMI QA Activities - A.Aimar (CERN)

3. EMI Mission Statement The European Middleware Initiative (EMI) project represents a close collaboration of the major European middleware providers - ARC, gLite, UNICORE and dCache - to establish a sustainable model to support, harmonise and evolve the grid middleware for deployment in EGI, PRACE and other distributed e-Infrastructures EMI QA Activities - A.Aimar (CERN)

4. EMI Project Structure NA1 - Administrative and Technical Management NA2 – Outreach and Collaborations SA1 - Maintenance and Support SA2 - Quality Assurance JRA1 - Development, Integration and Evolution EMI QA Activities - A.Aimar (CERN)

5. The Product Teams DRAFT - EMI QA Activities - A.Aimar (CERN) - DRAFT

6. The Challenge • Four large MW projects developing several products • Many development teams > 25 Product Teams • Request from the grid infrastructures and the EU funding • Have more uniform releases and interoperable middleware distributions • Provide a common degree of verification & validation • Have metrics and objective criteria to compare product quality and evolution over time • No resources to have independent QA efforts • Dev teams focus on development (JRA1) and maintenance (SA1) • No time to set up report on metrics, install tools, maintain testbeds • Provide homogeneous packaging, reporting and reviewing of the products • Have limited impact on the way development teams work • Let them still use their build systems, dev tools, bug trackers, etc EMI QA Activities - A.Aimar (CERN)

7. EMI QA (SA2) Tasks and Objectives • Quality Assurance Process Definition and Monitoring • Standards-compliant software engineering process. • Continual activity of monitoring its application • Metrics Definition and Reporting • Definition, collection and reporting of software quality metrics. • Reports information on the status of the software to take corrective actions • QA Implementation Review and Support • Review activities of the QA, test and certification implementations. • Sample review of test plans, compliance, porting guidelines, documentation, etc • Supporting the Product Teams in implementation of tests and use of testing tools • Tools and Repositories, Maintenance and Integration • Definition and maintenance of tools required to support QA process • Supporting activity to software providers to integrate external tools • Repositories for the EMI software packages, tests, build and reports • Testbeds Setup, Maintenance and Coordination • Setup and maintenance of distributed testbeds for continuous integration testing • Coordination and provision of larger-scale testbeds from collaborating providers EMI QA Activities - A.Aimar (CERN)

8. Software Quality Plan DRAFT - EMI QA Activities - A.Aimar (CERN) - DRAFT

9. Software Quality Plan • Software Quality Assurance Plan (SQAP) • Document that specifies the tasks needed to define and measure quality, responsibilities for quality monitoring, documentation required and procedures • Plan that will be followed to manage the QA process • SQAP specifies the manner in which the EMI project is going to achieve its software quality goals. • Metrics and measurements are associated in order to evaluate the quality of the EMI software and lifecycle • SQA tasks, roles and responsibilities • EMI technical activities (SA1, SA2 and JRA1) • EMI technical bodies (PTB and EMT) • Even of specific individuals/roles in EMI • The SQAP covers documentation, reporting and reviewing tasks • Describes the metrics that will be used for the QA reporting and reviews • Will be updated regularly, based on experience and needs • It is complemented by a set of guidelines (on dev and doc) EMI QA Activities - A.Aimar (CERN)

10. Development Guidelines • A set of very practical Guidelines for QA and Software Development is available: • Configuration and Integration • Packaging and Releasing • Change Management (bugs, patches, etc) • Certification and Testing • Metrics Generation • Move towards a uniform setup and common definitions and conventions in the project • Releases, patches, versions • Packaging, repositories, distributions • User support, documentation • Advantages for e-infrastructures are obvious but it requires some work and little changes by the dev teams in EMI • The guidelines were defined taking the best practices from the 4 middlewares and are mostly agreed by all dev teams EMI QA Activities - A.Aimar (CERN)

11. Development Guidelines DRAFT - EMI QA Activities - A.Aimar (CERN) - DRAFT

12. Documentation Guidelines • Definition of the Minimum Required Documentation for each software component. • Installation Guide and Troubleshooting guide • User Documentation • Software Requirements Specifications • Software Design Description • Software Verification and Validation Plan and Report • General Project-wide Required Documentation • Technical Development Plan • Software Release Plan and Schedule • Software Maintenance and Support Plan • QA Tools Documentation • Continuous Integration and Certification Test beds Documentation • Security Assessments EMI QA Activities - A.Aimar (CERN)

13. Quality Metrics • Metrics are needed to quantify and qualify the status of the software components • Use as much as possible numerical metrics • All automated and extracted in the same exact way , by the same tool(s) • Starting with a selection of useful and simple metrics • Tools available give a common environment to extract metrics and test • Same metrics for all components, in order to have fair reports • Types of SQA metrics • Metrics on code, process, support, documentation • Internal and external metrics (code and process) • Language dependent (Java, C++, Python) • Examples of metrics • Reaction time to critical bugs, delays in releases • Complexity, bug density, test coverage • We refer to QA standards, but use what is realistically applicable • ISO /IEC 9126-1,-2,-3,-4 and ISO/IEC 25000 EMI QA Activities - A.Aimar (CERN)

14. Metrics DRAFT - EMI QA Activities - A.Aimar (CERN) - DRAFT

15. Tools • Metric, Testing and Packaging Tools • Compliance and interoperability of the software products • Integration builds of all middleware components • Same identical platforms for all builds, use standard packages on platforms • Automatic deployment and distributed testing of software products • Integration of data coming from different sources and tools • Common report of metrics from different static and dynamic software QA tools • Collection of data from several req and bug trackers used by dev teams • Using same tool for packaging and testing tools • Use of an exchange format for different sources • Support for repositories and distribution • Common software repository for all EMI middleware • Use the standard RHEL/SL and Debian repositories and formats • Generation of QA reports • Metrics extraction, storage and archival • Graphs and reports at all levels of detail • Comparison tables and plot trends EMI QA Activities - A.Aimar (CERN)

16. Tools DRAFT - EMI QA Activities - A.Aimar (CERN) - DRAFT

17. Testbeds • EMI SA2 provides a distributed testbed • Real hw resources (+ obviously using virtualization) • For integration testing in EMI project • For the product teams testing • Distributed over the sites of the middleware partners • Testbed available to Dev teams for testing their software • The teams can easily test their components with what is in production or will soon be (RC) • Production and RC available for all components, servers, etc • Product Team can configure the services needed for its specific tests • Provide support for the typical scenarios • Integration testing within a minor release • Integration testing for a major release • Cross middlewares tests across the network • Large-scale tests, real usage scenarios EMI QA Activities - A.Aimar (CERN)

18. Testbeds DRAFT - EMI QA Activities - A.Aimar (CERN) - DRAFT

19. QA Reports and Reviews • QA reports objectively describe the quality of the product • Only clear metrics are included, e.g. number of bugs/SLOC, reaction time, trends over time, successful/failed releases, etc • Reports both on the product but also on how the team works • Not the goal of the QA activity, a mean to see strengths and weaknesses • The same type report for all components • Allows comparisons among components • Trends over time of the same components • Fully automated. Dev teams can have their report weekly • See the results and execute corrective actions • SA2 reviews of the QA reports and supports the teams • Provides the general reviewing every Month and formally every Quarter • Reports to the EMI mgmt in case of issues • Supports the dev team that need help with metrics, tools, testbeds EMI QA Activities - A.Aimar (CERN)

20. QA Reports DRAFT - EMI QA Activities - A.Aimar (CERN) - DRAFT

21. Conclusions • Dedicated QA is a useful activity in large projects, but guidelines, procedures should not overload the developers • Challenge of implementing QA in a distributed and heterogeneous environment • Different kind of sw products, different tools, distributed teams, etc • Collected experience from the developers in order to define realistic and shared solutions • Never seen a top down or theoretical software approach working... ... so we try a (slightly) different one. • Define metrics and automate report generation • Provide also practical services, not just procedures • to developers: supported and automated tools, testbeds, packaging • to e-infrastructures: verified and homogeneous sw, doc, repositories https://twiki.cern.ch/twiki/bin/view/EMI/SA2 DRAFT - EMI QA Activities - A.Aimar (CERN) - DRAFT

22. Thank you EMI is partially funded by the European Commission under Grant Agreement INFSO-RI-261611 EMI QA Activities - A.Aimar (CERN)