WP 4 Testing Tools and Methodologies

1. WP4Testing Tools and Methodologies Eva Takacs (4D Soft) ETICS Final Review CERN, Geneva - 15 February 2008

2. Contents • Goals • Major achievements • Case study • Challenges • Metrics and Statistics • Conclusions ETICS Final Review - WP4 - CERN, 15 February 2008

3. Goals The major goal of WP4 is to provide the ETICS distributed testing functionality by: • Implementation of distributed testing mechanisms and definition of a multiplatform distributed testing environment • Identification and analysis of test metrics • Identification and selection of software testing tools and metric calculation utilities • Adoption of the selected testing tools and metric calculation utilities into the ETICS build and test system (Test and Metrics Plugin Framework) ETICS Final Review - WP4 - CERN, 15 February 2008

4. Achievements ETICS Final Review - WP4 - CERN, 15 February 2008

5. Distributed testing in ETICS The aim of ETICS is to provide a distributed build/test framework where the functionalities of the framework and the testing tools are clearly separated. Having these principles in mind, the ETICS distributed testing environment consists of four major elements: • The distributed testing (multi-node test) mechanism • The Test and Metrics Plugin Framework as part of the ETICS client (command line interface) • Testing tools and tools for metrics collection as plugins of the Test and Metrics Plugin framework • The NMI/Metronome system as a multi-platform execution system ETICS Final Review - WP4 - CERN, 15 February 2008

6. Multi-node test mechanism – What is it? • The Multi-Nodes Test mechanism is the feature of ETICS that allows performing automated deployment and execution of complex distributed tests. • With this feature, ETICS can offer users the ability to define Distributed Tests, composed of Nodes (i.e. machines), containing the Services and Tests to be deployed, installed, configured and executed (i.e. instantiated). ETICS Final Review - WP4 - CERN, 15 February 2008

7. Multi-node test mechanism – Building blocks (architecture) Multi-node test mechanism brings together several components of the ETICS system. Notably: • Metronome (NMI) system as a submission layer • Condor scheduler - which allows users to define a “multi-node” submission, which trigger the allocation of several resources together on which parallel jobs can be executed • Modelling of a deployment model - i.e. composition of nodes (i.e. machines) on which services and test suites are instantiated leveraging the existing dependencies, environments and properties management together with an information service as part of ETICS client to control the order of operations and to synchronise the process • Test and Metrics Plugin framework as well as plugins ETICS Final Review - WP4 - CERN, 15 February 2008

8. Test and Metrics Plugin Framework In order to host the wide range of testing tools on client side, without breaking the ETICS’s core functionality, a plugin-framework was developed The plugin framework: • Provides an extendible framework for common actions to be performed during build and test execution • Providesnatural separation between specific build and test execution and analysis and metrics collection • Provides open interface for users to register their own plugins (commercially available tools)to be executed during build and test • Providesraw data for build and test and repository services especially for tracking trends, having statistics ETICS Final Review - WP4 - CERN, 15 February 2008

9. Why using Plugin Framework? ETICS ships with some built-in plugins but the rational behind using “Eclipse style” plug-in architecture is • Most of present and/or future ETICS users will come with some tests and testing tools already in place and of course they would like to use them when working with ETICS system • Users can plug commercially available testing tools • Commercial providers can extend the system with their products ETICS Final Review - WP4 - CERN, 15 February 2008

10. Metrics and their tool support in ETICS ETICS Final Review - WP4 - CERN, 15 February 2008

11. Installation modules • Additionally to plugins, so-called installation modules have been implemented • The aim of installation modules is to assist users in their installation and testing procedure by providing some general purpose installation modules for major tools or applications (httpd, APT, Tomcat, MySQL, etc) • The installation modules are deployable components with appropriate test commands to install, start and stop the services on which other test components can set dependencies in order to get those services running before the test is executed. ETICS Final Review - WP4 - CERN, 15 February 2008

12. Summary of achievements • Distributed testing mechanisms are in place. Multi-node test is considered one of the major achievements of the project • Multiplatform test can be run on a large number of platforms • Metrics have identified and their supporting testing tools are included as plugins • Extendibility of the system is allowed by Test and Metrics Plugin Framework ETICS Final Review - WP4 - CERN, 15 February 2008

13. Case study - Diligent system testing Remote regression testing: • In order to assure that the testing environment is integrated well and the testers can continue to work on new test cases (i.e. the environment is “testable”) it was requested to test the “testing environment” and remote regression tests have been run on the “testing environment”. • To accommodate this requirement and to take into account some property rights issues of the project, benefiting of ETICS remote testing capabilities of adding private resources to the ETICS pool has been applied. This feature is also allowed in ETICS additionally to public resources. ETICS Final Review - WP4 - CERN, 15 February 2008

14. Case study - Diligent system testing • Applying Java static analysis plugins for core Diligent services Applied plugins: JFindbugsPlugin, JDependPlugin JPmdPlugin, JCcnPlugin • The principal aim here has been to model a system testing environment in ETICS for Diligent components where the java static analysis plugins can run easily in an automatic and governed way on a periodic manner. The next screenshots show the modelled ETICS environment for applying JFindbugsPlugin in filtered mode ETICS Final Review - WP4 - CERN, 15 February 2008

15. Modelled system testing environment • One testing component corresponding to each core component Modelled system test environment ETICS Final Review - WP4 - CERN, 15 February 2008

16. Preparing the test component • To have the plugin running in filtered mode {findbugs.bug.codes} property is set ETICS Final Review - WP4 - CERN, 15 February 2008

17. Generated report • Generated metrics report Link to the generated metrics ETICS Final Review - WP4 - CERN, 15 February 2008

18. Summary metrics • Summary metrics for each jar file of the component and link to the detailed html report Link to the detailed report ETICS Final Review - WP4 - CERN, 15 February 2008

19. Detailed report • Detailed report showing the list of ‘NP’, ‘RCN’ warning containing a link to the explanation of them ETICS Final Review - WP4 - CERN, 15 February 2008

20. Explanation of the warning • Detailed explanation of the ‘high priority warning’ ETICS Final Review - WP4 - CERN, 15 February 2008

21. Explanation of the bug codes • Explanation of the ‘NP’ and ‘RCN’ bug codes ETICS Final Review - WP4 - CERN, 15 February 2008

22. Summary of the case study • It has been proved that ETICS build and test framework can be used efficiently to model complex testing processes assuring it’s maintainability and extendibility • Customisation of the process and handling multiple versions leverage the advanced property, dependency and configuration management ETICS Final Review - WP4 - CERN, 15 February 2008

23. Challenges • For the WP4 as technical work • Realising distributed testing – a quite new aspect in software engineering • Customising existing ‘traditional’ testing techniques for cutting-edge new technologies • Identifying the most suitable open source test tools that fit the best in the ETICS system and could help the users with testing facilities after performing builds and before releasing software packages • For 4D SOFT as company • Organisational challenge – being a WP leader as an SME in a project having mostly large academic institutions • Technical challenge - working with open source operation systems/tools widely adopted by academic institutions in contrast to the industry working mostly with commercially available software ETICS Final Review - WP4 - CERN, 15 February 2008

24. Metrics and Statistics • Deliverables list ETICS Final Review - WP4 - CERN, 15 February 2008

25. Metrics and Statistics • Nr of studied testing tools: 25 • Nr of identified metrics: 9 • Nr of implemented plugins: 15 • Nr of supported languages: 3 ETICS Final Review - WP4 - CERN, 15 February 2008

26. Conclusions • WP4 has provided new technologies in the distributed testing area by defining a new multi-node test mechanism for multi-platform distributed environment. • It has managed to add and customise the most appropriate testing tools that could augment efficiently the build and test process provided by ETICS framework. ETICS Final Review - WP4 - CERN, 15 February 2008

27. Thanks http://www.eu-etics.org ETICS Final Review - WP4 - CERN, 15 February 2008 27