1 / 83

Test Selection Practices in a Large IT Company

Test Selection Practices in a Large IT Company. Vincent Blondeau. Agenda. Presentation of the industrial context Identifying root causes of project failures Improving testing habits of Worldline developers Conclusion and future work. Industrial Context. Industrial Context: Worldline.

zia
Download Presentation

Test Selection Practices in a Large IT Company

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Test Selection Practices in a Large IT Company Vincent Blondeau

  2. Agenda • Presentation of the industrial context • Identifying root causes of project failures • Improving testing habits of Worldline developers • Conclusion and future work

  3. Industrial Context

  4. Industrial Context: Worldline

  5. Global Business Units Industrial Context: Worldline Merchant Services Financial Services Mobility & eTransactionalServices Benelux Germany & CEE France RegionalBusiness Units UK APAC LATAM & Iberia Transversal OperationalUnits Production Services Software DevelopmentCommunityOffice

  6. How to Predict the Health of a Project?

  7. Project Health: Definition Scope Resources Schedule

  8. Systematic Literature Review • Structure defined by [Staff Keeke et al., 2007] • Population: Software projects • Intervention: Predict health (success or failure) • Comparison: Cross projects • Outcome: Model based on metrics • Context: Industrial What metrics could help predict the success or failure of a project?

  9. Systematic Literature Review Keywords: Predict & Software & Project & Metric & (success | failure)

  10. Systematic Literature Review • Individually, metrics to make predictions exist • No unique metric for all the projects • Predictions retrospectively

  11. Data Mining • 19 projects • Average 1400 man-days • Between 900 to 2300 man-days • 91 excel files • 2012 to 2015 • Correlation of project metrics • Budget: 2 • Bugs: 8 • Slippage: 3 • Project Name (placebo metric) • From managers Excel files

  12. Data Mining Correlation method: Spearman Bugs Budget Slippage 12

  13. Data Mining Wewere not able to predictprojecthealth

  14. Pilot Study • Delay at the start of the project • Lack of team cohesion • Lack of collaboration between team and client • Misunderstanding of requirements • Misunderstanding of application domain concepts • Lack of experience with the used frameworks • Change of the framework during the project • Bypass qualification tests [Blondeau et al., IWST’15] [Blondeau et al., SATToSE’15]

  15. Questionnaire

  16. Questionnaire “In order to improve project health, what tool could help you?” …

  17. Summary • As far as we know: • Literaturedid not apply • Data miningdid not work • Interviews identifiedroot causes of failure • Not easy to measureit • Questionnaire confirmed the results Give up the projecthealthpredictionproblem

  18. Thesis Can we improve the testing habits of Worldline developers? (by giving them faster feedback)

  19. New Problem

  20. Test Selection • Example

  21. Test Selection Automatic Test Selection Select only the tests related to the changes and run only those ones 3 hours 5 minutes

  22. Strategy Can we improve the testing habits of Worldline developers? (by giving them faster feedback) • Choose a test selection approach • Establish current testing behavior of developers • Create a test selection tool

  23. Agenda • Presentation of the industrialcontext • Identifyingroot causes of projectfailures • ImprovingTesting Habits of Worldline developers • Choose a test selection approach • Establish current testing behavior of developers • Create a test selection tool • Conclusion and Future work

  24. Choose a Test Selection Approach

  25. Literature Review • Kazmiet al. “Effective regression test case selection: A systematic literature review”, ACM Comput. Surv., 2017 • Mining and Learning • Model Based Testing • Program Slicing • Oracle • Control Flow Graph

  26. Two Approaches T T T T T T T Dynamic Static

  27. Feasability Study • Which approach can we use at Worldline? • Dynamic more accurate • But should be kept up to date • Dynamic problem of failing tests • Static faster in our case • Feasability study on 3 sample projects

  28. Feasability Study

  29. Breaks • Issues to retrieve the tests from the changed code • Identified and classified as: • Dynamic breaks • Third Party breaks • Multi-program breaks • Polymorphism breaks [Blondeau et al., SoftwareQuality Journal, 2016] [Blondeau et al., IWST’16], [Blondeau et al., BENEVOL’15]

  30. Dynamic breaks: Attribute Initialization

  31. Third Party breaks: Anonymous Classes

  32. Third Party breaks: Delayed Execution

  33. Polymorphism breaks: Interfaces

  34. Research Questions • What is the impact on test selection of: • Granularity (class, method)? • Third-party breaks issue resolution? • Dynamic breaks issue resolution? • Polymorphism breaks issue resolution? • Combined solutions?

  35. Case Study • Ratio of the total test suite selected • Precision • How many selected tests are relevant? • Recall • How many relevant tests are selected?

  36. Case Study • Compared several static approaches resolving the issues • Dynamic approach is the oracle • Considered each covered method as changed T M T M M T M T M

  37. Tooling

  38. Results

  39. Results: Finer granularity? Number of Selected Tests Precision Recall

  40. Results: Third-party break issue? Number of Selected Tests Precision Recall

  41. Results: Polymorphism break issue? Number of Selected Tests Precision Recall

  42. Results: Dynamic break issue? Number of Selected Tests Precision Recall

  43. Results: Combined solutions? Number of Selected Tests Precision Recall

  44. Results • Breaks • are intertwined • impact differentlyeachproject • have different solutions for a given break class • Good final result: 3% of tests selected

  45. Results

  46. Current Testing Behavior

  47. When, How, and Why Developers TestBeller et al. (2015) • Open-source + industry + students • Monitor 416 software engineers Main conclusions • Majority of the developers rarely test in their IDE • Quick tests do not lead to more test executions • Some failing tests are fixed later: 50% of the test repairs happen within 10 minutes whereas 75% within 25 minutes

  48. Comparison of Manual and Automated Test Selection – Gligoric et al. (2014) • How developers manually select tests • Compare it to an automatic selection • Academic experiment: 14 developers: 5 professionals + 9 students Main conclusions • Need for better automated test selection techniques • Better integration with developer IDEs

  49. Field Study • Literature conclusions are not trivially transposable to our industrial case • Open source projects • Students • Need another study • Based on the replication of the 2 literature studies • In our context

  50. Field Study • Research Questions: • Do developers testtheir code changes? • Do quick tests lead to more test executions? • Do developers practice test selection? • Does manual test selection depend on sizeof code changes? • How manual test selection compares to automatic test selection?

More Related