Evidence-Based Decision Making : The Contribution of Systematic Reviews in Synthesizing Evidence

1. Evidence-Based Decision Making:The Contribution of Systematic Reviews in Synthesizing Evidence

2. General introduction to systematic reviews • What is a systematic review? • Why do/fund one? • For which policy questions is SR useful? • What are the main international bodies?

3. Systematic reviews as defined by the experts • “A systematic, up-to-date summary of reliable evidence of the benefits and risks of an intervention” (The Cochrane Collaboration) • Reviews “sum up the best available research on a specific question” (The Campbell Collaboration) • Defines a process, not a product. The process is a systematic and transparent approach to all phases of a literature review (Jeff Valentine).

4. Systematic reviews are research • Systematic reviews are a form of research • Secondary observations • In which studies are the unit of analysis • Follow basic steps in the research process • Aim to minimize bias and error

5. Example: microcredit • Evidence used to justify development policy is often anecdotal: “Small loans enable people in poverty to earn an income and provide for their families… Each successful business feeds a family, employs more people and eventually helps empower a whole community” (Opportunity International) • Where evidence is used it is often based on single studies: “Impact studies done on the Grameen Bank by independent researchers find that 5% of borrowers come out of poverty every year…the status of women has been enhanced” (Yunus, 2005) • Where a range of studies are used, not clear how representative: “Women participants in microcredit programs often experience important self-empowerment… there is a strong indication from borrowers that microcredit improves their lives” (Microfinance Gateway)

6. So what do reviews synthesizing all the existing high quality evidence say? Duvendack et al. (2011): ‘All impact evaluations of microfinance suffer from weak methodologies and inadequate data [which] can lead to misconceptions about the actual effects of a microfinance programme’ ‘It remains unclear under what circumstances, and for whom, microfinance has been and could be of real, rather than imagined, benefit to poor people’ Vaessen et al. (2012): ‘From those studies deemed comparable and of minimum acceptable quality, we can conclude that overall the effect of microcredit on women’s control over household spending is weak’.

7. Why do we need systematic reviews? • Sheer amount and flow of information/ research • Variable quality of research outputs • Need to ‘separate the wheat from the chaff’ • Problems of publication bias • Limitations of single studies

8. What makes a systematic review ‘systematic’? • Scoping: defining answerable question, methods set out in study protocol • Rigorous search to identify published and unpublished sources, in any language • Application of study inclusion criteria (PICOS) • Critical appraisal of study quality, to assess how reliable is the evidence • Data extraction and organisation • Synthesis of evidence (outcomes along causal chain) • Interpreting results (policy and practice, research recommendations) • Improving and updating reviews as new evidence emerges

9. Types of (answerable) questions

10. Resources and international bodies • Cochrane Collaboration; 1993; www.cochrane.org • Producing high quality information about the effectiveness of health care (> 5000 published online – Cochrane library) • Campbell Collaboration; 2000; www.campbellcollaboration.org • Producing systematic reviews of the effects of social interventions (>200 published online – Campbell library) • International Development Coordinating Group (IDCG); 2010 www.campbellcollaboration.org/international_development • Producing systematic reviews of high policy-relevance focusing on social and economic development interventions in LMICs • International Initiative for Impact Evaluation, 3ie; 2008 www.3ieimpact.org/en/evidence/systematic-reviews • EPPI Centre - An Institute of Education centre focusing on systematic reviews in education, health and social policy • Collaboration for Environmental Evidence producing systematic reviews for environmental management