Efficiency of Regression Testing

1. Efficiency of RegressionTesting

2. Contents • Research Topics & Methods • Software Quality (I & II) • Software Development Process (I-III) • Regression Testing • Test Automation (I & II) • M-MGw – Ericsson Media Gateway for Mobile Networks (I-III) • M-MGw Testing Process at Ericsson (I-IV) • M-MGw Test Environment • Identified Issues • Mutating Test Cases – Statistical Testing with TTCN3 in practice • Reliable Recovery (I & II) • Log and Filehandling Improvements • Different Node Configurations • Time • Thoughts and Conclusions • Q & A

3. Research Topics & Methods • Motive: Increasing the efficiency of regression testing in Media Gateway projects at Ericsson. • The main objectives: • How to measure the efficiency of regression testing? • What is the current state of automated regression testing at Ericsson Node Function Verification? • Suggestions and methods for improving the effiency (solutions to discovered issues) • Methods • Literature studies, interviews and discussions.

4. Software Quality I • M-MGw and telecommunications equipment in general have very high quality requirements. • What is Quality • Multiple models, each with its own upsides and drawbacks ”The degree which system, component or process meets the specified requirements, customer or user needs and expectations.” (ISO 9126:1991) • In software industry quality is the eye of the beholder • Not as easy to measure as in traditional manufacturing industry • Total number of faults in software can never be known (Exception being ”Hello world”)

5. Software Quality II Usability Efficiency • Quality attributes • NFV concentrates on usability, functionality and reliability • In telecommunications systems such as M-MGw stability and error-free operation are extremely important Reliability Maintainability Quality Functionality Portability Reusability

6. Software Development Process I • Software development phases. • Testing and verification • Verifies that • Software is stable • Works in the intended environment • According to specification • Testing and verification alone can cause half of the costs of software development. • Testing needs to be executed in cost- and time-efficient fashion. Feasibility Study Requirement Analysis Specification Design Implementation Testing & Verification Deployment Maintenance

7. Software Development Process II Feasibility Study • Software development process and V-model of testing • Not a sequential process ... • ... but in parallel with the development • Each phase of development has its corresponding test phase Requirement Analysis Acceptance Specification System Integration Design Implementation Unit Test Development Deployment Maintenance

8. Software Development Process III • Benefits • Can be executed in more efficient and systematic manner. • Easier to find and track faults in smaller software units than on system level • Testing of large entities can be done more systematically when subunits have already been tested. • Faults can be found sooner and therefore they are cheaper to fix. • M-MGw, the Ericsson Media Gateway for Mobile networks is tested in multiple phases during development.

9. Regression testing • ”Selective testing to verify that modifications have not caused unintended adverse side effects or to verify that system still meets requirements”. (Scherr, 1977) • Normal testing and regression testing have a common objective – ensure the quality of the software. • However there are differencies: - In regression code has already been tested - input to regression tests is changed code and specifications. - Scope can be limited and concentrate only to changed areas of the code.

10. Test Automation I • Machines never get bored. • Machines never need sleep. • Tests are executed exactly the same way. • Machines only find faults they are programmed to find. • Efficient. • Frequency

11. Test Automation II • Humans usually keep their eyes open. • Humans drink coffee. No case is executed same way twice. • Imagination and curiosity. • Pessimism. • Expect the unexpected. • Manual test execution is time- consuming

12. M-MGw – Ericsson Media Gateway for Mobile Networks I • Development has slowed down from revolution to evolution • Convergence towards all-IP • M-MGw offers customers a cost efficient network architecture solution on road to all-IP future.

13. M-MGw – Ericsson Media Gateway for Mobile Networks II MGC MGC Control Flow BTS TDM Network BSC Media Flow UE IP or ATM UE M-MGw RBS RNC M-MGw vMGw M-MGw vMGw vMGw vMGw

14. M-MGw – Ericsson Media Gateway for Mobile Networks III • Right now multiple networks, protocols and signalling systems co-exist and they need to work together • Media Gateway nodes provide the interfaces needed for communication. • Media Gateway • Handles user data • Switching and routing between the core and access network • Media transformation and transcoding between the networks. • Functions of MGws are controlled by MGCs, Media Gateway Controllers.

15. M-MGw Testing Process at Ericsson I Component Tests Load Module Tests LMG Tests

16. M-MGw Testing Process at Ericsson II • Early Verification in design organization • Whitebox testing • Individial components, loadmodules and load module groups. • Goal – to get the trivial faults fixed before system level tests.

17. M-MGw Testing Process at Ericsson III System Verification Node Function Verification

18. M-MGw Testing Process at Ericsson IV • System level tests are executed by system testers • Black box testing in target environment • Goal – To verify functionality, characteristics, robustness and maintainability and to find and fix faults that are only seen on system level.

19. M-MGw Test Environment

20. Measuring the Efficiency of automated regression test I • Performance indicators provide valuable information that can help in developing the process. • Common sources of inaccuracy, examples • Million ways to fail • Are design tests executed? • Pesticide effect • Delays because of TRs. • One indicator is not enough – combination of indicators must be used to extract useful information.

21. Measuring the Efficiency of automated regression test II • Performance indicators provide valuable information that can help in developing the process. • Amount of Failures/Executed TCs (AoF) -Gives some indication about the quality of software -Also indicates how good our test cases are at finding faults

22. Measuring the Efficiency of automated regression test III • Regression Catch Rate (RCR) • Ratio of faults found in regression test to total number of faults found from the software. • How efficient is automated testing compared to manual testing. • Inaccurate because of coverage differencies.

23. Measuring the Efficiency of automated regression test IV • Regression Test Efficiency (RTE) • Average time to execute test case. • Time-efficiency, for common function optimization.

24. Identified Issues • Coverage • Test Environment, reliable recovery & tool issues • Different node configurations • Maintenance • User plane verfication issues (not in scope...) • Logging-related issues • Time

25. Mutating Test Cases – Statistical Testing with TTCN3 in practice • Increasing coverage by increasing variance with statistical testing • Works only in certain situations • Provides tools for stress testing • Mean-Time-To-Failure • Mean-Time-Between-Failures • Loops – executed until fault is found • Tricky to implement • Fault tracing is challenging. • rnd

26. Reliable Recovery I • Node restart in certain situations. • After one or several failures • Requires small modifications to test execution tools • Restart cures everything short of HW failure • Takes time • System state is lost, faults can be hard to reproduce.

27. Maintenance functions Start n= 1 Maintenance functions Start n = 1 Fail k = k + 1 Pass Execute n Pass Execute n Failed n == N n < N k < K k == K n = n + 1 k = 0 n = n – (K–2) n == N n < N n == N n < N n < N n == N n = n + 1 n = n +1 n = n + 1 Node Restart Node Restart Next Test Object Node Restart Next Test Object Reliable Recovery II Simple recovery solution Advanced recovery solution

28. Log and Filehandling Improvements • Not an issue yet. • Traces written to file • Enable more efficient tracing • Delete after certain period if case is closed • More efficient troubleshooting, especially in hard to trace -cases

29. Different Node Configurations • Distributed solution • More maintenance • Individual implementations • Centralized solution • Maintained by tool team • Always up to date configurations, no need for indepth understanding • Common solution for all test objects?

30. Time • Execute nightly regression test objects using multiple nodes • Nodes expensive, use vMGws instead -Would require advanced tagging scheme -Coverage lost with bad behaving cases -changes required to test tools -Fault tracking would be even more difficult -One faults causes every case to fail • No individual optimization, optimize the common functions

31. Thought and Conclusions • You get what you measure • One indicator alone does not work • Multiple performance degrading issues were found, both time and coverage-wise and feasible solutions to those issues were found. • First – Recovery methods. Then optimization of common functions to save time. • Further research and optimization still to be done, some companies claim 80% cost savings by making testing more efficient.

32. Q & A Any Questions?