Philip Davies International Initiative for Impact Evaluation [3ie]

1. 3ie-LIDC Seminar Series 'What Works In International Development’ London School of Hygiene and Tropical Medicine 20th February 2013 Getting Evidence Into Policy Philip Davies International Initiative for Impact Evaluation [3ie]

2. Outline of Presentation • How evidence is used in policy • Factors other than evidence • Some features of evidence • Different notions of evidence • Barriers to getting evidence into policy • Overcoming barriers to getting evidence into policy

3. How Evidence is Used in Policy Making Conceptual Use Involves using research results for general enlightenment; results influence actions, but in less specific, more indirect ways than in instrumental use Instrumental UseInvolves acting on research results in specific, direct ways. Symbolic Use Involves using research results to legitimate and sustain pre-determined positions. Source: Carol Weiss, 1982

4. How Evidence is Used in Policy Making “Rarely does research supply an “answer” that policy actors employ to solve a policy problem. Rather, research provides a background of data, empirical generalisations, and ideas that affect the way that policy makers think about a problem.” “But to acknowledge this is not the same as saying that research findings have little influence on policy.” Source: Carol Weiss, 1982

5. Factors Other Than Evidence Values and Decision Making Context Pragmatics & Contingencies Experience & Expertise Lobbyists & Pressure Groups Evidence Judgement Bureaucratic Culture Resources

6. Some Features of Evidence • Evidence is almost always probabilistic • Evidence is often context specific • Often disagreement agreement on what counts as ‘evidence’ • Evidence is always contestable/contested • Evidence is rarely self-evident • Not all research is of equal value/sufficient quality • Single studies can misrepresent the balance of evidence • Hence, the need for systematic reviews/synthesis of evidence

7. Different Notions of Evidence Knowledge Translation and Transfer Policy Makers’ Evidence Researchers’ Evidence • Colloquial (Narrative) • Anything that seems reasonable • Policy relevant • Timely • Clear Message • ‘Scientific’ (Generalisable) • Proven empirically • Theoretically driven • As long as it takes • Caveats and qualifications Source: J. Lomas et al, 2005

8. UK Policymakers’ Views of Evidence* • Focus on the ‘end product’, rather than how the information was either collected or analysed • Use of ‘anecdotal’ evidence • Drawing on such things as ‘real life stories’, ‘fingers in the wind’, ‘local’ and ‘bottom-up’ evidence • But: • “If we try and move anywhere without having the scientific basis to do so we get fleeced in the House” *Source. Campbell, S. et al, 2007: Analysis for Policy, London, GSR.

9. Where Do UK Civil Servants Go For Evidence? Sharks Plankton Academic/Evaluation Research?

10. Barriers to the Use of Evidence • Policymakers’ lack of familiarity with the research process • Researchers’ lack of familiarity with the policy process • Trust (lack of trust) of policymakers in researchers (vice versa) • Physical access to evidence • Cognitive access to evidence (i.e. lack of understanding) • Lack of clarity in the presentation of evidence • Timeliness and availability of evidence Sources: Lomas, 2000; Petticrew et al, 2004; Lavis et al, 2005; Dobbins et al, 2007; Ouimet, et al, 2009; Rosenbaum, 2010

11. Overcoming Barriers to the Use of Evidence • Early and ongoing involvement of relevant decision makers • Interactions between researchers and policymakers increases the prospects for research use by policymakers • Identify and use interpersonal networksand face-to-face interactions • Identify willing and able knowledge brokers • Separate strategic from operational demands for evidence • Get policy makers to own the evidence – not just the policy Sources: Lomas, 2000; Petticrew et al, 2004; Lavis et al, 2005; Dobbins et al, 2007; Ouimet, et al, 2009; Rosenbaum, 2010

12. Improving Communication of Evidence • Establish what the evidence says, and does not say • Establish the policy messages and policy implications • But avoid ‘recommendations’ • Provide information about the costs and benefits, harms and risks of interventions/policies • Present contextual factors that affect local applicability • Be clear - plain English summary, unclouded by jargon • Find the ‘influencing moment’ • Aligned to decision making timescales • Use a 1:3:25 format

13. The ‘One’ in the 1:3:25 Format • A one page of main message bullets • The lessons decision makers can take from the research • Not a summary of findings • Suggest implications of findings • No details of methodology

14. The ‘Three’ in the 1:3:25 Format • These are the key findings of the study • The classic Executive Summary • Condensed to serve the needs of the busy decision maker • Focus on how the study may be useful for policy • Some brief mention of methodology • Some implications of policy and practice

15. The ‘25’ in the 1:3:25 Format This should include: • Context/Background • Approach (Methodology in Appendices, not text) • Results • Implications • Knowledge gaps • References • Additional resources • Appendices

16. Summary • The routes of evidence to policy are usually indirect and delayed • There are factors other than evidence • Evidence is seldom definitive or invariant • Nor is it self-evident • There are different notions of evidence • Physical and cognitive access to evidence is important • Evidence has to be ‘worked’ into policy/practice • Clear communication evidence is essential

17. References Campbell, S., et al, 2007 Analysis for Policy, London, GSR Dobbins et al, 2007 Public Health Decision-Makers’ Informational Needs and Preferences for Receiving Research Evidence, Worldviews on Evidence-Based Nursing, 3rd Quarter, 156-163, New York, Wiley Inc. Lavis et al, 2005 Towards systematic reviews that inform health care management and policy-making, Journal of Health Services Research and Policy, 10, Suppl 1 July 2005 S1:35. Lomas, J., 2000 Connecting Research and Policy, Canadian Journal of Policy Research, Spring, 140-144 Lomas, J. Culyer, T., McCutcheon, C., McAuley, L., and Law, S., 2005 Conceptualizing and Combining Evidence for Health System Guidance, Final Report, Canadian Health Services Research Foundation, Ottawa. .

18. References (continued) Ouimet, M., Landry, R., Ziam, S., and Bédard, P., 2009 The absorption of research knowledge by public servants, Evidence and Policy, 5, 4, 331-350. Petticrew, M., et al, 2004 Evidence for public health policy on inequalities: 1: The reality according to policymakers, Journal of Epidemiology and Community Health 2004;58:811-816 Rosenbaum, S.E., et al, 2010 Evidence summaries tailored to health policy-makers in low- and middle-income countries, Bulletin of the World Health Organization, 89, 1, 54-61. Weiss, C.A., 1982 Policy research in the context of diffuse decision making, Journal of Higher Education, 53, 6, 619-639.

19. Thank you Philip Davies Email: pdavies@3ieimpact.org +44 (0)207 958 8350 Visit www.3ieimpact.org