Realist Synthesis: Implementation Strategies for Evidence-Informed Healthcare

1. Alison Hutchinson, Australia/Canada Alyce Schultz, USA Brendan McCormack, UK Bridie Kent, New Zealand Cheryl Stetler, USA Erna Snelgrove-Clarke, Canada Jo Rycroft-Malone, UK Kara DeCorby, Canada Lars Wallin, Sweden Marita Titler, USA Tracey Bucknall, Australia Val Wilson, Australia ReS-ISRealist Synthesis – Implementation Strategies The story so far….

2. Outline • Background & methodology • Searching & retrieval • Process issues • Emerging findings

3. Background • KU colloquia • Frustration • Existing evidence base for ‘effective’ interventions

4. The state of the evidence base • Systematic review evidence* • Some indications: • Facilitative approaches, feedback, education…. • Some limitations *e.g. Thomas et al 2000, O’Brien et al , 2003, Foxcroft & Cole 2003, Grimshaw et al 2004, Thompson et al 2007

5. Limitations • Equivocal findings • Don’t necessarily relate to the professional practice of all practitioners • Use of behaviour change as outcome measure • Lack of acknowledgement about complexity and context

6. A different approach – different lens • Want to know about what is working, what isn’t working: • Different contexts (& levels) • Different stakeholders • Address meaningful questions • Theoretical driven

7. Methodology • Realist synthesis (Pawson et al, 2004) • A method for studying complex interventions using diverse bodies of data • Clear objectives and inclusion criteria • Benefits • Flexible but rigorous • Capacity for providing detailed and practical recommendations

8. The Realist Alternative • Review method based on principles of realistic evaluation • Focus is on reviewing complex social interventions e.g. policy, management, service delivery • Review takes place at the level of theories that underpin complex interventions • Explanatory focus; seeking answers to the question ‘What works, for whom, in what circumstances, in what respects and why?’

9. Characteristics of Complex Social Interventions • Consist of theories • Involve actions of people • Consist of a chain of steps or processes that interact • Constituent steps and processes are rarely linear • Embedded in social systems • Prone to modification • Open systems that change through learning (Pawson et al, 2004)

10. Rethinking the Standard Systematic Review Template • Clarifying the scope of the review • Searching for evidence • Appraisal of primary studies • Data extraction • Data synthesis • Interpretation of findings/ presentation of results and recommendations

11. Clarifying the Scope of the Review • Identify the review question • Nature and content of the intervention • Circumstances or context for its use • Policy intentions or objectives • Refine the purpose of the review • Theory integrity, adjudication, comparison • Articulate key theories to be explored • Draw up long list • Group, categorise or synthesise • Design the evaluative framework

12. Purpose of the Review What are the interventions and strategies that are effective in enabling evidence informed healthcare?

13. ReS-IS Theoretical Model OUTCOMES 1. Change Agency 2. Systems change • Change agent characteristics • Change agent interventions • Interplay between change agent characteristics and interventions • Systems change interventions • Interplay between setting and interventions DOSE LEVELS Theory and Contextual Factors what works, for whom, in what circumstances, in what respects and why 3. Technology 4. Education and Learning • Characteristics of technological interventions • Impact of technological interventions • Characteristics of education interventions • Impact of education interventions

14. Four Theories and 13 Theoretical Foci • Theory area 1 - Properties of change agency in KU • What impact do the characteristics of the change agent have on KU • What is the overall impact of the change agent intervention on KU? • What impact does the interaction between the change agent and the setting have on KU? • Theory area 2 –system change in KU • What impact do characteristics of the systems change intervention(s) have on KU? • What is the overall impact of the system change intervention(s) used? • What impact does the interaction between the system change and the setting have on KU? • What impact do senior leadership roles have in creating practice environments that integrate daily use of evidence at the point of care delivery?

15. Theory area 3 – properties of technologies (paper & electronic) used in KU • What impact do the characteristics of the technological intervention(s) have on KU? • What is the overall impact of the technological intervention(s) used? • What impact does the interaction between the technological intervention and the setting have on KU? • Theory area 4 – education interventions in KU • What impact do the characteristics of the education intervention(s) have in enabling KU? • What is the overall impact of the education intervention(s) used? • What impact does the interaction between the education intervention and the setting have on KU?

16. Core and Quality Analysis • Core Areas Analysis • Is there evidence of particular theoretical perspective(s) impacting on the effectiveness of the intervention? • Is there evidence of contextual factors impacting on the effectiveness of the intervention? • Is there evidence of the level of the intervention impacting on the effectiveness of the intervention? • Is there evidence of the intervention dose impacting on the effectiveness of the intervention? • Quality Analysis • Is the evidence provided in this theory area good and relevant enough to be included in the synthesis (consider issues of sample size, data collection, data analysis and claims made)

17. Search for Evidence • Searching by stages to: • Get a ‘feel’ for the literature • Identify key programme theories (and refine inclusion criteria) • Test and refine programme theories • Purposive and snowball sampling • Look across policy domains • Final search for additional studies when review nearing completion

18. Data Extraction • Develop ‘bespoke’ set of data extraction forms and notation devices • Extract different data from different studies to populate evaluative framework with evidence • Interest is in the ‘chains of inference’ in the original study: “this body of evidence allows me to draw the following conclusions”

19. Data Synthesis • Basic task is to refine the programme theory i.e. determine what works, for whom, in what circumstances, in what respects and why • Use contradictory evidence to generate insights about the influence of context

20. Searching, Relevance Testing, & Data Extraction • search strategy • relevance, 2 stream approach • data extraction, final relevant papers

21. Search Strategies • broad, not discipline-specific, corresponding to healthcare in general • approx 39 lines of search text used with Boolean operators

22. Search Results 6 online databases (1997-2007): (Medline, CINAHL, Embase, PsycInfo, Sociological Abstracts, Web of Science) Health Sciences Librarians (Dalhousie University, McMaster University) consulted on search strategies run March 5, 2007 in OVID

23. Search Results • 24,021 electronic references • Medline 4,530 • Embase 8,482 • CINAHL 5,683 • PsycInfo 4,993 • Sociological Abstracts 248 • Web of Science 130 • 196 potentially-relevant papers based on title and abstract (return rate of 0.8%)

24. Inclusion Criteria • relate to purpose statement: What are the interventions and strategies that are effective in enabling evidence-informed health care? • published in the last 10 years • english-language • healthcare related • all disciplines • not limited by research focus or design (e.g. papers addressing intervention application in addition to effectiveness)

25. Data Extraction • papers divided among 5 work groups • further papers eliminated based on full versions • data extraction of each paper by 2 reviewers • final data extraction resulted in a number of papers being omitted

26. Overall Process: What challenges were overcome • Methodology • Questions • asking the right question to get the literature • Search terms • Volume • Choice of one area first to refine process • Inclusion or exclusion criteria • Health & non-health papers • Empirical & non empirical papers • Theory papers • Interventional papers • Process of analysis, • 2 independent reviews • double checking all papers • Clarity, confidence in process • Assumptions on data extraction

27. Overall Process: What worked well • Communication- • face to face and phone • email • Yahoo discussion group • Variable perspectives • International group • Research methods used • Range of specialties • Variable work processes • Established process • Agree rules of engagement, responsibilities and outputs • Small and large group • Product Development • Established database • Completed preliminary review • Refined processes • Learning • Access and exposure to new literature • Realist synthesis method and philosophy

28. Preliminary Findings

29. Theory Area1: Impact of Change Agents on KU • Character of the literature • Crosses a number of theory areas • Poor linkages between processes, context and outcomes • Methodological problems and study design challenges

30. Impact of the Characteristics of the Change Agent? • Role characteristics • Knowledge characteristics • Overall impact of change agent interventions? • Potential to change practice context • Increasing knowledge of others • Providing feedback • Awareness raising • Some practice change • How?

31. Impact of Education (overlap with theory area 4) • Change agent and education interventions • Education that includes content and process • Barriers and how to overcome these • Specific projects and professional/personal growth

32. Impact of the interaction between change agent and setting? • Relationship between character of context and impact of change agent • Illogical inferences!

33. Next Steps: • Refine data extraction templates • Complete synthesis for theory area 1 • Write up process, challenges, findings • Begin review process for other theory areas • Need funding!!!!!!

34. Strengths of the Realist Approach • Firm roots in philosophy and the social sciences • Not a method or formula, but a logic of enquiry • Pluralist and flexible • Explanatory, as opposed to judgemental • Learns from, rather than attempting to control, real world phenomena • Engages stakeholders in a systematic way • Has the potential to maximise learning across policy domains

35. Some Limitations • Stage of development of the realist approach; small number of completed reviews • Cannot be used as a protocol driven approach • Handling potentially large volume of evidence • Developing/documenting reproducible methods • Demonstrating and maintaining objectivity • Requires a high level of experience on the part of the reviewer • Leads at best to tentative recommendations