Research Synthesis

1. Research Synthesis Barbara Kitchenham David Budgen, Stuart Charters, Mark Turner, Pearl Brereton, & Stephen Linkman

2. The Evidence-Based Paradigm • Origins in clinical medicine, where it has had a major impact on both practice and teaching. • Stems from the challenge laid down by Archie Cochrane, initially in 1972, that the evidence from empirical trials was not being utilised to inform practice. Set out later in 1979 as: “it is surely a great criticism of our profession that we have not organised a critical summary, by speciality or subspeciality, adapted periodically, of all relevant randomised controlled trials” • Major bandwagon of the 1990’s for clinical medicine, now supported by the Cochrane Collaboration.

3. Systematic Literature Review • A key process in evidence-based practice. • A systematic literature review is: A summary of research studies that uses explicit and reproducible methods • Often referred to as a secondary study, the aim is to synthesise existing (empirical) research • fairly (without bias) • rigorously(according to a defined procedure) • openly (so that the process is visible to other researchers)

4. Medical Guidelines • First example of guidelines for systematic literature reviews • Meta-analysis for Quantitative aggregation • RCTs to provide high quality primary studies • Quality evaluation of primary studies based on assessing • Procedures for minimizing Bias • Procedures for maximizing Internal Validity • Procedures for maximising External Validity

5. Software Engineering • Less rigorous studies • Surveys • Case studies • Quasi-experiments • Experiments without double-blind procedures • When quantitative information available • Study protocols differ • Different measures and analyses used • Issues addressed by qualitative & quantitative studies • EBSE project is looking at practices in disciplines that are closer to SE

6. Domain Comparison • Obtained expert views from each domain about the following experimental characteristics: • Use of field experiments or quasi-random experiments • Use of laboratory experiments • Use of other types of empirical study • Whether studies involve human expertise • Whether the participants can be ‘blinded’ as to treatment • Whether the experimenters can be ‘blinded’ as to treatment

7. Characterisation: Similarity Matrix

8. Some approaches used in social sciences • Domains close to SE are sociological (except Chemistry) • Criminology • Education • Nursing & Midwifery • Much important systematic review research in these domains • Systematic Review methodology based on Social Science research • Systematic Reviews in the Social Sciences (Petticrew and Roberts, 2006) • Checklists for assessing different study types • Qualitative case studies (Atkins & Sampson, 2002) • Surveys (Crombie, 1996) • Procedures for synthesising qualitative studies • Meta-ethnography (Noblit & Hare, 1988) • Procedures for combining qualitative & quantitative results (Thomas et al., 2003)

9. Quality Checklist for Case studies • Way of thinking • Choice of method and personal viewpoint • Way of controlling • Quality control procedures • Way of working • Case study procedures • Way of supporting • Data collection & Tools • Way of communicating • Reporting procedures

10. Meta-ethnography • Based on concept of “translating” one study’s terminology and discussion into that of another • Synthesis can be based on • Comparable accounts that build a coherent picture • Accounts that are in opposition to one another and articulate conflicting views • Accounts that taken together represent a line of argument • Assisted by text analysis tools (such as NVivo) • SE example currently being undertaken by Tore Dybå and colleagues • Looking at agile methods • Propose useful quality checklist

11. Synthesising Qualitative & Quantitative Studies • Evaluation of study quality • Use different checklists & criteria for different types of study • Analyse different types of study separately • Meta-analysis may be appropriate for quantitative studies • Meta-ethnography (or similar) for qualitative studies • Synthesise results by evaluating the relationship between qualitative and quantitative summaries • Are all issues raised by qualitative studies addressed in quantitative studies? • Do issues raised by qualitative studies partition quantitative studies? • Do issues raised by qualitative studies explain quantitative results?

12. Conclusions • Interest in adapting the evidence-based paradigm is widespread across many disciplines. • Successes among these suggest that it may prove equally useful for Software Engineering. • Those domains with similar characteristics to SE tend to be at the ‘social’ end of the spectrum. • Our guidelines and protocols should be based more on the approaches adopted in these disciplines less on medical ones

13. References • Atkins, C. and Sampson, J. Critical appraisal guidelines for single case study research. ECIS, 2002, pp 100-108. • Crombie, I.K. The Pocket Guide to Critical Appraisal. BMJ Publishing Group, 1996. • Noblit, G.W. and Hare, R.D. Meta-Ethnography: Synthesizing Qualitative Studies, Sage Publications, 1988. • Petticrew, M and Roberts, H. Systematic Reviews in the Social Sciences. A Practical Guide, Blackwell Publishing, 2006 • Thomas, J., Sutcliffe, K., Harden, A., Oakley, A., Oliver. S., Rees, R., Brunton, G., Kavanagh, J. Children and Healthy Eating. A systematic review of barriers and facilitators. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London, 2003.