1 / 36

Toward a Quality Certification Process for Grid Research Projects: an ETICS feasibility Study

Toward a Quality Certification Process for Grid Research Projects: an ETICS feasibility Study. Andrea Manieri Adriano Rippa - Engineering Ingegneria Informatica s.p.a. On behalf of ETICS Project. Why bother?. R&D project results are often interesting, but:

coye
Download Presentation

Toward a Quality Certification Process for Grid Research Projects: an ETICS feasibility Study

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Toward a Quality Certification Process for Grid Research Projects: an ETICS feasibility Study Andrea Manieri Adriano Rippa - Engineering Ingegneria Informatica s.p.a. On behalf of ETICS Project

  2. Why bother? • R&D project results are often interesting, but: • They aim only at proving concepts • They suffer of deviation from plans due to their investigation nature • Developers (sometimes) don’t pay enough time to testing, documentation, etc • Developers are not so skilled in software engineering • After the end of a projects: • None (out of the consortium) is willing to use such sw • The software itself is difficult to maintain and improve • So, many times, effort and brilliant results are lost! Open Grid Forum 20 - Manchester 07-11 May 2007

  3. That are only dreams? • Software components fully tested and documented • Maintenance of the know-how for build, deploy and run the software • Software components that build on different platform • Software bundle with all the right dependencies and the right libraries to be used by other developers Open Grid Forum 20 - Manchester 07-11 May 2007

  4. Quality assurance… • QA is a matter of: • In which way sw has been created (procedural) • How sw is written (structural) • How sw perform (behavioural) • Currently we have: • QA standards (I.e. CMMi, ISO, ITIL, …) • SW Testing • So-called Quality of Service (at least partially) Open Grid Forum 20 - Manchester 07-11 May 2007

  5. What’s missing, then? • A systematic approach to • Build software with all the correct dependencies • Extensively Test each components • That allows gathering some measures • Related to the build performance • Reporting tests results and • Other characteristics of the code • With which derive the overall quality of the software produced! • DISCLAIMER: this study will focus, initially, on Research Projects Open Grid Forum 20 - Manchester 07-11 May 2007

  6. Yet another Quality Assurance model? • Current models are useful but: • are process oriented • provide theoretical guidelines on the organization • Theproblem: • Guidelines can be applied but it’s hard to systematically verify goodness of results Open Grid Forum 20 - Manchester 07-11 May 2007

  7. ETICS feasibility study • To study and propose agrid software Certification model that can • Be automated and • Be not subjective • Used within Research Projects • ETICS will analyse the feasibility of a Certification for grid software: • To support the safe adoption of software generated by research projects • To help researchers in produce good grid applications Open Grid Forum 20 - Manchester 07-11 May 2007

  8. The timeline Now OGF 20/EGEE UF 7-11 May OGF 21 15-19 Oct FIRST BETA VERSION FINAL BETA VERSION FIRST OFFICIAL VERSION FEBRUARY MAY OCTOBER DECEMBER ECHOGRID/EUChinagrid Conf. Beijing 24-25 April QUALIPSO Conference Dec 2007 Belief Conference RIO – BRASIL 25-28 June Open Grid Forum 20 - Manchester 07-11 May 2007

  9. ETICS: the project • Started in January 2006 • Duration: 2 years • Funded by the European Commission: ~1.4 M€ • 5 partners • 2 early adopter projects • EGEE/gLite • DILIGENT/gCube • Spin-off of the integration and test activities of EGEE-I for gLite middleware • ETICS Service: • Improve the way Grid software is built and tested • Increase sharing in the Grid communities Open Grid Forum 20 - Manchester 07-11 May 2007

  10. The ETICS Partners Build system, software configuration, service infrastructure, dissemination, EGEE, gLite, project coord. Software configuration, service infrastructure, dissemination The Condor batch system, distributed testing tools, service infrastructure, NMI Web portals and tools, quality process, dissemination, DILIGENT Test methods and metrics, unit testing tools, EBIT Open Grid Forum 20 - Manchester 07-11 May 2007

  11. ETICS is a framework… • based on a grid infrastructure where it’s possible to: • Run any scripts over the code (e.g ant ) • Run the unit or component testing • Run a script to deploy one or more services • Run a test to verify the conformity of some API or guidelines (e.g IPV6 compliance) • … • Then collect all these data to • perform (trend) analysis • gather measures and calculate metrics… Open Grid Forum 20 - Manchester 07-11 May 2007

  12. Where ETICS tools apply DESIGN and CODEWRITING CODE BUILD and INTEGRATION ETICS context TESTINGincludingCOMPLIANCE EXECUTION Open Grid Forum 20 - Manchester 07-11 May 2007

  13. A Land of nobody? • WRONG! • Many projects on-going (i.e. QUALIPSO for the quality in the open source environment..) • Plenty of build tools (GUMP, Maven, *-forge, Savanna, etc) • Often open source… • Several testing tools (Pulse, Visual Build, etc) • Often commercial… Open Grid Forum 20 - Manchester 07-11 May 2007

  14. Terminology • Measure: a value (number or category) assigned to an attribute of an entity (e.g. software component) • Measurement: The act or process of assigning a number or category to an entity to describe an attribute of that entity. • Metric: A function whose inputs are software measures and whose output is a single numerical value that can be interpreted as the degree to which software possesses a given attribute that affects its quality. • Perspective: A set of homogenous software attributes that lead to some quality assumptions Open Grid Forum 20 - Manchester 07-11 May 2007

  15. Grid Quality Assurance Certification Model (GQACM ): a 3D QA Model • Let’s suppose A, B, C to be different perspectives • Common QA Standards are Waterfall-like • GQACM uses 3 independent perspectives to provide client-friendly results • Users can be interested in some attributes and less in others.. • So-made perspectives can have sublevels… A B C A C B Open Grid Forum 20 - Manchester 07-11 May 2007

  16. GQACM: Preliminary remarks • GQACM is independent from the ETICS tools • However, any GQACM implementation must use automation – the most useful characteristic of the model is that it can be automatically applied! • GQACM has three perspectives (I.e. independent points of view) to derive the quality degree of a software : • Code • Platforms • Standards C Open Grid Forum 20 - Manchester 07-11 May 2007

  17. The GQACM: Perspectives • Code Perspective • Usual metrics depending on programming paradigm • … to qualify the degree of goodness of the code (static analysis) • Platform Perspective • The type and quantity of different platform (I.e. family of platforms) where software built correctly and can be automatic deployed • Standard Perspective • Software tested for compliance to applicable standards(e.g. BES, JSDL, SRM, WS-I, IPV6) • These perspectives areindependent each other Open Grid Forum 20 - Manchester 07-11 May 2007

  18. Perspectives composition • Code Perspective • Coding Conventions • Code Commenting • Other QA metrics related to the corresponding programming language • Platform perspective • The set of platforms (declared, most used, all available?) • Standard perspective • The set of (available) standards against the compliance is proven. Open Grid Forum 20 - Manchester 07-11 May 2007

  19. Test execution metrics • C1 (method level): • if C is the cyclomatic complexity (of the method) and T is the number of executed tests then RM1 = C/T • C2 (class level): • if CK RFC is the the number of methods which can be invoked from an instance of the class target to answer a message ) and T is the number of executed tests on it then RM1 = RFC/T. • C3 (class level): • if E is the number of events which interact with the class and T is the number of executed tests then RM4 = E/T • C4 (class level): • if CK NOC is the Number Of Children and T is the number of executed tests on the class then RM8 = NOC/T Open Grid Forum 20 - Manchester 07-11 May 2007

  20. Code comment metrics • C5: • if CLOC are lines of comments, then RM5 = CLOC/LOC • C6: • if F is the number of functions and C is the number of comment paragraphs then RM6 = F/C • C7: • Automatic documentation programs use: e.g. doxygen Open Grid Forum 20 - Manchester 07-11 May 2007

  21. Coding convention metrics 1/2 • Only one class definition per header file • Filename = Classname • Filename in lowercase • No cycles in “include” operations • No C instruction (in C++ programming) • Coherent Programming Convention • Curly Brackets positioning • First character in variables names • Lower case in macros names • Coherent use of new lines • Semicolons only at the end of a line Open Grid Forum 20 - Manchester 07-11 May 2007

  22. Coding convention metrics 2/2 • No Never used functions • No Not reachable code paths • No Not defined prototypes • No Never used constants • No Never used macros • No Never used types definitions • No Never used variables • Too frequent “breaks” use • No absence of variables de-alloaction • No Null pointers • Management of one instruction long “if” or one instruction long “cycle” • Shared resources management Open Grid Forum 20 - Manchester 07-11 May 2007

  23. other metrics • Errors correction ability • C7: errors trend between a test and another after corrections • Other software related metrics • (e.g. CK Metrics for OO software) Open Grid Forum 20 - Manchester 07-11 May 2007

  24. GQACM: Users benefits • Users will: • Have a clear picture about the quality and … • …will be able to select components and software of certified quality level • Concretely for different roles: • Developer will be able to certify its software • Integrator will be safe in compose new software starting from existing certified components of the appropriate quality degree • Final User will be sure to use certified software Open Grid Forum 20 - Manchester 07-11 May 2007

  25. Yes, but! • I’m not able to add any overhead to my project • This model (and the capability of automate) will reduce the effort in performing continuous build and tests (e.g. coverage tests) on different releases • How much costs adopting it? • Nothing, the model will be discussed publicly and the final version will be released under open license (e.g. Creative Commons) • The ETICS framework is provided as a service running on a dedicated infrastructure, free of charge for Research Project (until dec 2007) • My organisation is certified ISO/CMMi? • See next slides… Open Grid Forum 20 - Manchester 07-11 May 2007

  26. ISO 9126 • is an international standard for the evaluation of software. • is divided into four parts which address • quality model • external metrics • internal metrics • quality in use metrics. • The quality model established in the first part of the standard, ISO 9126-1, classifies software quality in a structured set of characteristics and sub-characteristics divided into attributes. Open Grid Forum 20 - Manchester 07-11 May 2007

  27. GQACM integration in an ISO 9126 certified organization • Facts: • ISO 9126 define quality attributes . • GQACM support the measure and evaluation of quality metrics. • Hypothesis: • ISO9126 Functionality: adequacy. • ISO9126 asks for correct functions with specific goals. • GQACM may check not dead functions and other coding convention. • ISO9126 Maintainability: all. • ISO9126 asks to develop a software that is easy to analyse, easy to modify, Stable… • GQACM metrics (such as the CK for OO) helps to verify in concrete these requirements Open Grid Forum 20 - Manchester 07-11 May 2007

  28. GQACM integration in an ISO 9126 certified organization • ISO9126 Functionality: accuracy. • ISO9126 asks that a software have to give results according with requirements. • GQACM certification can run functional test (or plug-in external test suits) of declared compliances. • ISO9126 Portability: adaptability, installability • ISO9126 asks OS adaptability and installability for software of certified organizations. • GQACM may test deployment of SW on different platforms (HW+OS) • Our thesis: • ISO9126 quality metrics can be measured and evaluated based on GQACM • Any tool implementing it and can be integrated as supporting ISO9126 adoption Open Grid Forum 20 - Manchester 07-11 May 2007

  29. CMMI • It provides organizations with guidelines for effective organisational processes • It can guide organisational process improvement across a project, a division, or an entire organization • It helps to integrate traditionally separate organizational functions • It sets goals and priorities Open Grid Forum 20 - Manchester 07-11 May 2007

  30. GQACM integration in a CMMi certified organization • Facts • CMM is process oriented (focused on the organisation). • GQACM is product oriented. • Hypotesis: • CMM2 • asks to “control and value” • GQACM propose the measures and metrics to be collected, hence controlling and evaluating specific sw related items • CMM3 • asks to “use internal standards and have a reference model” • GQACM proposes code and implementation conventions (how to organise and structure code, how to define tests, etc…) Open Grid Forum 20 - Manchester 07-11 May 2007

  31. GQACM integration in a CMM certified organization • CMM4 • asks to “have measured and controlled processes using quantitative and statistical techniques” • GQACM propose the continuous collection of quality data to perform trends analysis. • CMM5 • asks to have an “improvement process and quality based on measurement” • GQACM provide numerical reference to set the improvement processes!! • Our Thesis • GQACM is not in contrast with CMMI and can be integrated in organization as a tool to support CMMI adoption Open Grid Forum 20 - Manchester 07-11 May 2007

  32. ETICS B&T tool and GQACM • ETICS: • Can perform many measurements and calculate metrics  code perspective • Can build and test software on several distinct platforms  platform perspective • Can test the software compliance with standards  standard perspective • Can automate all the processes (build, deployment, test) Open Grid Forum 20 - Manchester 07-11 May 2007

  33. Where to find more on ETICS? • ETICS portal: http://eu-etics.org Open Grid Forum 20 - Manchester 07-11 May 2007

  34. Underdevelopment Where to find more on ETICS? • ETICS wiki:https://uimon.cern.ch/twiki/bin/view/ETICS/WebHome • ETICS client getting started https://uimon.cern.ch/twiki/bin/view/ETICS/ClientHowTo • ETICS tutorialshttps://uimon.cern.ch/twiki/bin/view/ETICS/ETICSTutorials Open Grid Forum 20 - Manchester 07-11 May 2007

  35. Conclusions • ETICS provides a multi-platform environment for building, testing and checking the quality of distributed and grid applications • ETICS focus is on automation of builds, tests, reporting, collection and analyse of quality measures … to simplify life when managing complex software management tasks • We want it to be as simple and effective as possible to use. For this we need your feedback: ETICS-QAstudy@cern.ch ETICS-discuss@cern.ch ETICS-support@cern.ch Open Grid Forum 20 - Manchester 07-11 May 2007

  36. Thanks Thanks you for taking the time to learn about ETICS Have a good day Give us feedback http://www.eu-etics.org Open Grid Forum 20 - Manchester 07-11 May 2007

More Related