1 / 16

Economics of automation in functional testing of network service platforms

Economics of automation in functional testing of network service platforms. Supervisor: Prof. Heikki Hämmäinen Instructor: M.Sc. Reima Kaitajärvi Mikko Lehtonen, Nokia Networks. Contents. Background Research Problem and Research Methods Testing, overview Test automation

patch
Download Presentation

Economics of automation in functional testing of network service platforms

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Economics of automation in functional testing of network service platforms Supervisor: Prof. Heikki Hämmäinen Instructor: M.Sc. Reima Kaitajärvi Mikko Lehtonen, Nokia Networks

  2. Contents • Background • Research Problem and Research Methods • Testing, overview • Test automation • Cost efficiency model • Flexi Platform, overview • Results of the case study • Conclusions and further study

  3. Background • Test automation is an intriguing concept, because it includes a promise of more comfortable testing. • The business interests in automation are clear, e.g. to reduce the testing effort, enable earlier release for publication and save testing costs

  4. Research Problems and Research methods • How test automation can be used to improve testing in FlexiServer’s subcomponent Basic IP Services? • How to analyze costs and benefits of test automation in FlexiServer’s subcomponent Basic IP Services? Methods: A literature study was performed to gain the required know-how of the problem domain. Case-study was performed to solve the exact research problems.

  5. Testing, overview • The main goal of testing is to verify the quality of the product • Testing is consuming about 50% of development resources and effort in many software projects • As the amount of source code increases significantly, the testing costs seem to increase even more need to cut costs

  6. Test Automation (TA) • One way to enhance the cost efficiency of testing • Problem: • Test automation is not always an appropriate solution • The cost efficiency of TA is difficult to quantify due to the intangible nature of many variables (e.g. quality, improved professionalism)

  7. Solution: the cost efficiency model The model connects three perspectives: • A procedure for choosing the most cost effectively automated test areas (cost efficiency calculations) • defines ways to analyze the quality of automated testing • provides guidelines for implementation of the testware

  8. Cost efficiency calculations • Formulas concentrate on tangible factors i.e. financial values • Intangible values must be considered separately • Two possible formulas:

  9. Calculations (cont.) • The most difficult thing is to chooce the correct cost factors for a given situation (formulas quite simple) • Principles on how to calculate the additional costs of manual and automated testing

  10. FlexiServer arhitechture • The BIPS implemented mostly in Linux + some in Nokia extensions (e.g. SCTP stack) • The BIPS implements protocols that enable the communication between network elements (IP, TCP, UDP…)

  11. The results • Reasons for the differences: • Implementation took more time than estimated • Differences in manual test time (actual_1>average)

  12. Results (cont.) • The average manual testing time is used (probably more realistic) • The figure on right side is calculated using discounted future costs (NPV)

  13. Summary result of BIPS • The automation of BIPS was a quite small scale project (2 person months) • Obtained financial savings also small, • intangible benefits (e.g. improved professionalism) support financial savings • The results are dependent from the test environment • Some of the tests of BIPS have to be still performed manually.

  14. Main conclusions • The main results: • the automation of BIPS • the information of the economy of automation in that environment • and the toolkit for analysing the cost efficiency of automation in a defined project • Some abstracted results: • The general environment (organization,SUT,test processes,test environment) affects to the economy of TA • The suitable candidates for automation are tests or supporting activities that are executed often, have a long lifetime and take long time to perform manually. • The choice what to automate first is essential, because the implementation of TA takes that much time that not all the test can be automated in the normal schedule of a test organization.

  15. Further study • The effects of automation to the quality of testing (and product!) and test thoroughness • How the possible saved testing time is consumed?

  16. Questions? Thank you for your interest!

More Related