Grid and Distributed Software Certification and Quality Assurance

1. Grid and Distributed Software Certification and Quality Assurance GGF17 Certification BoF Marc-Elian Bégin CERN

2. Agenda • Introduction • ETICS overview • OMII (Steven Newhouse) • Discussion • Conclusion Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo

3. ETICS: E-infrastructure for Testing, Integration and Configuration of Software Marc-Elian Bégin Dissemination, certification and web tools Work-package Leader CERN

4. Vision and Mission • Vision: A dependable, reliable, stable grid infrastructure requires high-quality, thoroughly tested, interoperable software middleware and applications • Mission: Provide a generic service that other projects can use to efficiently and easily build and test their grid and distributed software.Set up the foundations for a certification process to help increasing the quality and interoperability of such software Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo

5. Short Term Use Cases • Automatically build software on different platforms • Automatically collect and store metrics during build process • Quality Assurance trend analysis on collected metrics • Automatic handling and management of dependencies • Automatic execution of testsuites, including build, installation and configuration of dependencies • “Shop” dependencies from quality measured artifacts (components, services and plug-in) via managed repository • Offload deployment of complex middleware services • Combinations of the above Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo

6. ETICS in a Nutshell • ETICS stands fore-Infrastructure for Testing, Integration and Configuration of Software • It’s an SSA • It has been granted a contribution of 1.4 M€ • It has a duration of two years • The project has started on January 1st, 2006 Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo

7. The ETICS Partners Software configuration, service infrastructure, dissemination Build system,software configuration, service infrastructure, dissemination, EGEE, gLite, project coord. The Condor batch system, distributed testing tools, service infrastructure, NMI Web portals and tools, quality process, dissemination, DILIGENT Test methods and metrics, unit testing tools, EBIT Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo

8. ETICS Objectives • Objective 1 (technical) • Provide a comprehensive build and test service especially designed for grid software • Support multi-platform, distributed operations to build software and run complex test cases (functional, regression, performance, stress, benchmarks, interoperability, etc) • Objective 2 (coordination, policies) • Establish the foundations for a certification process • Contribute to interoperability of grid middleware and applications by promoting consistent build and test procedures and by easying the verification of compliance to standards • Promote sound QA principles adapted to grid environment through the participation to conferences, workshops, computing training events (CSC, ICEAGE, EGEE) Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo

9. Web Application NMI Scheduler Web Service Service Overview Via browser Build/Test Artefacts Report DB Project DB Via command- Line tools NMI Client Wrapper WNs ETICS Infrastructure Clients Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo

10. User roles • Guest: can execute local builds and test of modules • Developers: can create configurations and execute local and remote build and test of modules • Integrators: can register build artefacts in repository • Testers: can register and perform trend analysis of test results • Release Managers: can publish and release artefacts, create release notes and other documentation • Module Administrator: create modules and assign permissions with a given project • Administration: can create and manage projects, assign permissions, set policies, etc Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo

11. QA Certification Process • It’s the other major goal of the project • ETICS must deliver a feasibility study for the implementation of a QA certification process for grid and distributed software • Not just a document, but a full assessment of what procedures, tools and rules maximize the chances of project of delivering reliable, interoperable software based on the real data collected in two years of activity • We will propose it as the starting point of a coordinated QA Certification activity in the context of a more permanent grid infrastructure initiative • Will be done in close collaboration with the ETICS Experts Board to make sure it is practical and realistic Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo

12. Standardisation effort • e-Infrastructure Reflection Group (e-IRG) • e-IRG mandate: “coordinates on a high European level the introduction of a (grid based) infrastructure for e-Science” • Working on raising awareness in • Software Quality Assurance and automation • Standard software Certification Process • Co-chaired a working session during last e-IRG meeting in Linz on “Middleware Interoperability and Repository” • Global Grid Forum (GGF) • GGF mandate: “international community leading the pervasive adoption of grid computing in research and industry” • Chairing a working session in the next GGF meeting in Tokyo on “Grid and Distributed Software Certification and Quality Assurance” • Raise awareness on the need for Standard Certification Processes, including automated tests for interoperability Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo

13. Typical ETICS working scenario OMIIEU/UK HEP \ BIO \INDUSTRY Repositories Other(projectspecific) ETICS OMIIEU/UK Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo

14. Repositories • All artifacts produced when using the ETICS services are made available to the community at large • Source and binary packages • Test reports • Quality assurance reports • Benchmarks • Interoperability evaluations, etc • ETICS is involved in digital libraries and repositories on two fronts: • ETICS has among its partners DILIGENT, a Grid powered Digital Library • ETICS is collaborating with the EU-OMII project to set up a reference repository of quality assured software packages Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo

15. Virtual Test Lab • The Grid must embrace diversity in terms of platforms, development languages, deployment methods, etc • However, testing all resulting combinations is very difficult and time consuming, not to mention the manpower required • ETICS is proposing a high-efficiency automation process, but still providing and especially maintaining the required hardware and software resources is not trivial • A possible solution is the use of virtualisation Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo

16. ETICS Engine Virtual Test Lab Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo

17. ETICS Engine Microsoft Virtual Server The Virtual Test Lab Images can contain preinstalled OSs in fixed, reproducible configurations Security risks are minimized, the image is destroyed when the test is over. But it can also be archived for later offline analysis of the tests Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo

18. Example of Web Application Project structure Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo

19. Example of Web Application Project configuration Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo

20. Example of Build Report Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo

21. Example of Test Report Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo

22. Example of Job Submission Report Preliminary integration of the web application with NMI Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo

23. Example of Client Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo

24. Status – ETICS Development • Client • Available for local and remote building and testing software (read only) • Editing feature under development • Server • Available for building and testing software (read only) • Editing feature under development • Secure access under development • Reports engine under development • Web Application • Available for browsing ETICS metadata • Editing feature under development • Editing stopgap solution • Tool for registering new projects Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo

25. Status – Usage • Build system • Large element of gLite 3.0 building • On track to build entire gLite 3.1 stack • In the process of building CERN/FIO Quattor software • In the process of building DILIGENT middleware/software • Building ETICS software • Test system • Porting gLite Certification Testsuites • Executing ETICS stress testsuite • Deployment • UoW: ~70 machines • CERN: ~20 machines • INFN: ~3 machines Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo

26. Potential new users • Automatic execution of build/test • MPI testsuites execution and trend analysis: requested by Cal Loomis (EGEE/NA4) • Build and test of OMII components (UK+Europe) • CERN Administrative Information Services • Build and test of J2EE applications • Others Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo

27. Long Term Future and Sustainability • We envision ETICS to become a permanent service after its initial two-year phase • As projects start using and relying on it for managing the software development cycle, the ETICS infrastructure should get enough “critical mass” to be supported by research and industrial organisation as other “commodity” services • In addition, we want to propose ETICS as one of the cornerstones of a more permanent international collaboration to establish a European and world-wide grid infrastructure Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo

28. Thanks http://www.eu-etics.org Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo

29. OMII Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo

30. Questions/Issues for discussion • How do you see certification and QA in your context? • Do you need it? • Are you doing it? • If you don’t, why? • Would a “standardised certification process” be useful? • Building • Do you do multi-platform build and test? • If so, how? • If not, who does it for you, or why not? • Where do you put your build artefacts? • Where do you find your dependencies and how to you manage them? • How do you manage your releases? • Testing • Do you have testsuites? Are they automated? • Are you using tools to implement and execute your tests? • How do you execute them and how do you handle test outputs? • How do you analyse the results • Scope for standardisation • Process • Test execution and output formats • Metrics and their collection Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo