1 / 15

Theory and Practice, Do They Match ? A Case with Spectrum-Based Fault Localization

Theory and Practice, Do They Match ? A Case with Spectrum-Based Fault Localization. Tien-Duy B. Le, Ferdian Thung , and David Lo School of Information Systems Singapore Management University. Spectrum-Based Fault Localization. Locating buggy program elements by

hinto
Download Presentation

Theory and Practice, Do They Match ? A Case with Spectrum-Based Fault Localization

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Theory and Practice, Do They Match ?A Case with Spectrum-Based Fault Localization Tien-Duy B. Le, FerdianThung, and David Lo School of Information Systems Singapore Management University

  2. Spectrum-Based Fault Localization • Locating buggy program elements by • Analyzing two sets of execution traces • Normal traces and faulty traces • Assigning suspiciousness scores to program elements • Two well-known SBFL formulas • Tarantula • Ochiai

  3. Spectrum-Based Fault Localization • Xie et al. “A Theoretical Analysis of the Risk Evaluation Formulas for Spectrum-based Fault Localization” (TOSEM, 2013) • Two families of SBFL formulas • ER1 and ER5 (5 formulas in total) • Theoretically proven to outperform Ochiai and Tarantula • Under the assumption: test coverage is 100%

  4. Our Goal Benchmark Programs vs. Theoretically Best SBFL Formula by Xie et. al. Popular SBFL Formula ?

  5. Notations

  6. Popular SBFL Formulas • Tarantula • Ochiai

  7. Theoretically Best SBFL Formulas

  8. Dataset • 10 programs, 199 faulty versions • Siemens test suite • Space, NanoXML, XML-Security • Evaluation Metric • The lower the EXAM score, the better the performance

  9. Results • Ochiai has the lowest EXAM score (21.02%)

  10. Results • Tarantula‘s EXAM score is lower than 4 out of the 5 theoretically best SBFL formulas

  11. Results • Wilcoxon signed rank test (significance level of 0.05) •  Ochiai is statistically better than ER5a, ER5b, ER5c

  12. 100% Test Coverage Assumption • For 135 out of the 199 faulty versions • Test coverage < 100% • Average test coverage of the 199 versions • 84.97%  Theoretically best SBFL formulas cannot outperform popular SBFL formulas

  13. Conclusion • We conduct an empirical study on 10 programs with 199 versions • Compare performance of 5 theoretically best SBFL formulas with Tarantula and Ochiai • We find that: • Ochiai outperforms all theoretically best formulas • Tarantula outperforms 4 out of the 5 formulas • Assumption of 100% test coverage is not valid in many cases

  14. Future work • In-depth study how test coverage and other factors affect effectiveness of SBFL formulas • Theoretically analyze performance of SBFL formulas • Assumption: test coverage < 100%

  15. Thank you! Questions? Comments? Advice? {btdle.2012, ferdiant.2013, davidlo}@smu.edu.sg

More Related