180 likes | 336 Views
Improving Effectiveness of Regression Testing of Telecommunications Systems Software. Sami Torniainen 19.2.2008 Supervisor: Professor Raimo Kantola. Outline. Goal Background How to improve cost-effectiveness? Methods Implementation Evaluation Further research. Goal.
E N D
Improving Effectiveness of Regression Testing of Telecommunications Systems Software Sami Torniainen 19.2.2008 Supervisor: Professor Raimo Kantola
Outline • Goal • Background • How to improve cost-effectiveness? • Methods • Implementation • Evaluation • Further research
Goal • Goal: improve cost-effectiveness of regression testing at the target company • Three sub-objectives: • explore present regression testing practices • implement methods • evaluate empirically
Background - Target Company • Finnish telecommunications systems developer • Located in Espoo • Main product: IP/MPLS routers • Customers: global mobile and internet service providers
Background – Regression testing • “Regression testing is selective retesting of a system or component to verify that modifications have not caused unintended effects and that the system or component still complies with its specified requirements.” (IEEE) • Can occur on any level of the V-model • This thesis focuses on regression testing on integration testing level
How to improve cost-effectiveness? • Costs accumulate from time and resources consumed in executing tests • Two approaches mainly utilized in literature • test selection techniques • test automation
Test selection techniques • Reduce the number of tests • Mainly based on code exploration • unit testing • Also exists risk-based method • integration testing level
Test automation • Automating manual working phases • creating executable test programs i.e. scripts • Popular method • Expensive to utilize
Implementation • Limited scope (10 test suites) • Risk-based test selection technique • evaluating test suites based on risk-analysis • executing only five test suites of ten • Test automation • automating the control of Adtech AX-4000 • Tcl API • no manual intervention needed during test execution
Evaluation - Summary • Test automation reduced testing effort significantly • but was expensive to implement (94%) • Risk-based test selection method also reduced testing effort • but will likely miss errors
Conclusions • Despite the great implementation costs, test automation is recommended • break-even ”only” after 34 test rounds • Benefits • reduced testing effort • safer • reduce the overall testing elapsed time • potentially earlier time to market
Further research • Develop Tcl API • reloading of GUI-based test setups • would reduce scripting effort • Optimizing test coverage of risk-based test selection technique • threshold • minimizing probability to miss errors Vs. reducing the number of test suites
Thank You! Questions?