1 / 22

Evidence-Based Software Engineering and Systematic Reviews

Evidence-Based Software Engineering and Systematic Reviews. Barbara Kitchenham. Agenda. The evidence-based paradigm Evidence-Based Software Engineering (EBSE) Systematic Reviews. The Evidence-Based Paradigm. Evidence-based medicine has changed research practices Medical researchers found

niel
Download Presentation

Evidence-Based Software Engineering and Systematic Reviews

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Evidence-Based Software Engineeringand Systematic Reviews Barbara Kitchenham

  2. Agenda • The evidence-based paradigm • Evidence-Based Software Engineering (EBSE) • Systematic Reviews

  3. The Evidence-Based Paradigm • Evidence-based medicine has changed research practices • Medical researchers found • Failure to organise existing medical research cost lives • Clinical judgement of experts worse than systematic reviews • Evidence-based paradigm adopted by many other disciplines providing service to public • Social policy • Education • Psychiatry

  4. Goal of EBSE • EBM: Integration of best research evidence with clinical expertise and patient values • EBSE: Adapted from Evidence-Based Medicine • To provide the means by which current best evidence from research can be integrated with practical experience and human values in the decision making process regarding the development and maintenance of software • Anticipated benefits • Common goals for research groups • Help for practitioners adopting new technologies • Means to improve dependability • Increase acceptability of software-intensive systems • Input to certification process

  5. Practicing EBSE • Convert information need into answerable question • Track down best evidence • Critically appraise evidence • Integrate critical appraisal with SE expertise and stakeholder requirements • Evaluate and improve above steps

  6. Systematic Reviews - 1/2 • A systematic review is • An overview of research studies that uses explicit and reproducible methods • Systematic reviews aim to synthesise existing research • Fairly (without bias) • Rigorously (according to a defined procedure) • Openly (ensuring that the review procedure is visible to other researchers)

  7. Systematic Reviews – 2/2 • Support Evidence-based paradigm • Start from a well-defined question • Step 1 • Define a repeatable strategy for searching the literature • Step 2 • Critically assess relevant literature • Step 3 • Synthesise literature • Step 4 (but only partially)

  8. Advantages • Provide information about effects of a phenomenon across wide range of settings • Essential for SE where we have sampling problems • Consistent results provide evidence that phenomena are • Robust • Transferable • Inconsistent results • Allow sources of variation to be studied • Meta-analysis possible for quantitative studies

  9. Anticipated Benefits • Create a firm foundation for future research • Position your own research in the context of existing research • Close areas where no further research is necessary • Uncover areas where research is necessary • Help the development of new theories • Identify common underlying trends • Identify explanations for conflicting results • Should be a standard research methodology

  10. Disadvantages • Require more effort than informal reviews • Difficult for lone researchers • Standards require two researchers • Minimising individual bias • Incompatible with requirements for short papers

  11. Value of Systematic Reviews • Can contradict “common knowledge” • Jørgensen and Moløkken reviewed surveys of project overruns • Standish CHAOS report is out of step with other research • May have used inappropriate methodology • Jørgensen reviewed evidence about expert opinion estimates • No consistent support for view that models are better than human estimators

  12. Write Review Report Validate Report Systematic Review Process Develop Review Protocol Plan Review Validate Review Protocol Identify Relevant Research Select Primary Studies Conduct Review Assess Study Quality Extract Required Data Synthesise Data Document Review

  13. Developing the Protocol • Review protocol • Specifies methods to be used for a systematic review • Predefined protocol • Reduces researcher bias by reducing opportunity for • Selection of papers driven by researcher expectations • Changing the research question to fit the results of the searches • Good practice for any empirical study

  14. Protocol Contents -1/2 • Background • Rationale for survey • Research question • Critical to define this before starting the research • Strategy used to search for primary sources • Individual studies of the phenomenon of interest

  15. Protocol Contents – 2/2 • Strategy to find primary studies • Search terms, resources, databases, journals, conferences • Procedures for storing references • How publication bias will be handled • Grey literature • Direct approach to active researchers • How completeness will be determined • Useful to have the baseline paper to set start date • Selection Strategy • Inclusion/exclusion criteria • Handling multiple papers on one experiment

  16. Protocol Contents- 2/3 • Quality assessment criteria • Criteria used to evaluate quality of primary sources • Data extraction • What data will be extracted from each primary source • How to handle missing information • How data reliability will be addressed • Usually multiple reviewers • Where data will be stored • Procedures for data synthesis • Formats for summarising data • Measures and analysis if meta-analysis is proposed • Should tested during protocol construction

  17. Research Question – 1/2 • Question types for EBSE • Assessing the effect of an SE technology • Assessing the frequency or rate of a project development factor • E.g. Rate of project failures • Identifying cost and risk factors • Identifying impact of technology on reliability, performance, cost • Possible to have more general questions for other purposes • Review of research in software engineering (Glass, et al., 2002)

  18. Research Question – 2/2 • Question structure • Population • People, projects types, applications types affected by the intervention • Intervention • Software method, tool, procedure • Outcomes • Impact of technology in terms relevant to practitioners • Cost, quality, time to market • Experimental designs • Any constraints on type of primary studies to be included

  19. Next steps are easy!? • Conduct the review • Enact the protocol • Expect further iterations of • Search strategy • Selection criteria • Data extraction • Record any deviations from protocol • Document the Review • Using procedures defined in protocol

  20. Conclusions • Evidence-based approach • Revolutionised medicine • May be relevant to SE • Systematic reviews • Support the evidence-based approach • Valuable as a research tool • Even if we don’t accept EBSE

  21. References Australian National Health and Medical Research Council. How to review the evidence: systematic identification and review of the scientific literature, 2000. IBSN 186-4960329 . Australian National Health and Medical Research Council. How to use the evidence: assessment and application of scientific evidence. February 2000, ISBN 0 642 43295 2. Cochrane Collaboration. Cochrane Reviewers’ Handbook. Version 4.2.1. December 2003. Glass, R.L., Vessey, I., Ramesh, V. Research in software engineering: an analysis of the literature. IST 44, 2002, pp491-506 Magne Jørgensen and Kjetil Moløkken. How large are Software Cost Overruns? Critical Comments on the Standish Group’s CHAOS Reports, http://www.simula.no/publication_one.php?publication_id=711, 2004. Magne Jørgensen. A Review of Studies on Expert Estimation of Software Development Effort. Journal Systems and Software, Vol 70, Issues 1-2, 2004, pp 37-60.

  22. References Khan, Khalid, S., ter Riet, Gerben., Glanville, Julia., Sowden, Amanda, J. and Kleijnen, Jo. (eds) Undertaking Systematic Review of Research on Effectiveness. CRD’s Guidance for those Carrying Out or Commissioning Reviews. CRD Report Number 4 (2nd Edition), NHS Centre for Reviews and Dissemination, University of York, IBSN 1 900640 20 1, March 2001. Kitchenham, Barbara. Procedures for Performing Systematic Reviews, Joint Technical Rreport, Keele University TR/SE-0401 and NICTA 0400011T.1, July 2004. Pai, Madhukar, McCullovch, Michael, Gorman, Jennifer D., Pai, Nitika, Enanoria, Wayne, Kennedy, Gail, Tharyan, Prathap, Colford, John M. Jnr. Systematic reviews and meta-analysis: An illustrated, step-by-step guide. The National medical Journal of India, 17(2) 2004, pp 86-95. Sackett, D.L., Straus, S.E., Richardson, W.S., Rosenberg, W., and Haynes, R.B. Evidence-Based Medicine: How to Practice and Teach EBM, Second Edition, Churchill Livingstone: Edinburgh, 2000.

More Related