Implementation and process evaluation: developing our approach

1. Implementation and process evaluation: developing our approach Gemma Moss Institute of Education Neil Humphrey University of Manchester Ann Lendrum University of Manchester

2. Starting activity • What is your understanding of the terms ‘process’ and ‘implementation’? • Try to come up with working definitions for each in your group • It may be helpful to think about how these terms could be applied and understood in relation to an intervention you are evaluating

3. Introduction • Understanding ‘what works’ is important but it is equally important to know, “why various programs do or do not work, for whom and under what conditions they work, what is needed to scale up proven programs, and what policy supports are needed to scale them up without losing their effectiveness” (Slavin, 2012,p.xv)

4. What do we mean by implementation and process evaluation? • Put very simply… • Assessment of outcomes in trials answers the question does it work? • Implementation and process evaluation (IPE) helps us to understand how and why • IPE within trials explore, “the implementation, receipt and setting of an intervention and help in the interpretation of outcomes” (Oakley et al, 2006, p.413) • Studying how the intervention is implemented (including how and why this varies) • Ascertaining the views of key participants (e.g. pupils, teachers) of critical issues (e.g. social validity) • Distinguishing between different intervention components • Investigating contextual factors that may influence the achievement of expected outcomes

5. What do we mean by implementation and process evaluation? • IPE can help to clarify whether assumptions in the intervention design about the causal links from intervention to impact work out in practice • By paying attention to the social processes involved in making the intervention happen, IPE can help us to • Know what happened in an intervention • Establish the internal validity of the intervention and strengthen conclusions about its role in changing outcomes • Understand the intervention better – how different elements fit together, how users interact et cetera • Provide ongoing feedback that can enhance subsequent delivery • Advance knowledge on how best to replicate intervention effects in real world settings (Domitrovich and Greenberg, 2000)

6. A worked example • Secondary SEAL national evaluation (Humphrey, Lendrum & Wigelsworth, 2010) • Outcomes strand • Pre-test-post-test control group design (41 schools, c.8,000 pupils) • Primary outcomes were social and emotional skills, behaviour, mental health • IPE strand • Longitudinal cases studies in 9 SEAL schools • Interviews and focus groups, observations, document analysis • Analysis of data from outcomes strand indicated that SEAL had no measurable impact • Analysis of data from IPE strand helped us to understand why this was the case • Issues with the programme theory (or lack thereof) • Implementation failure • Lack of understanding, resistance among staff

7. Why is IPE important? • Interventions are rarely (if ever!) delivered exactly as planned • Variability in implementation has been consistently shown to predict variability in outcomes • Interventions do not happen in a vacuum – so understanding context and the social processes within them is crucial

8. Putting the I in IPE • Aspects of implementation • Fidelity/adherence • Dosage • Quality • Participant responsiveness • Programme differentiation • Programme reach • Adaptation • Monitoring of comparison conditions • Factors affecting implementation • Preplanning and foundations • Implementation support system • Implementation environment • Implementer factors • Programme characteristics • See Durlak and DuPre (2008), Greenberg et al (2005), Forman et al (2009)

9. Researching IPE activity • Think about a trial you have recently completed • Did you have an IPE built into your trial? • If YES… • What information did you collect and why? • What did the data generated tell you about the intervention, the context in which it was being implemented, and the interaction between the two? • One useful piece of information you were able to feedback to the intervention designers/implementers/participants/funders about the process of implementation • If NO… • What were the main findings of the trial? • What conclusions did you draw as a result? • What could IPE have added?

10. Developing our approach to IPE • General approach - quantitative, qualitative or both? • 62% quant, 21% qual, 17% both in health promotion research (Oakley, 2005) • Where to target resources? • Intervention, context, and the interaction between the two • Which aspects to assess when examining implementation? • In assessment of implementation the focus is predominantly on fidelity and dosage, but this can lead to a Type III error • What you see and what people tell you • Implementer self-report or independent observation?

11. Benefits of IPE • For implementers • explains and illuminates findings from the impact evaluation, strengthening the conclusions impact evaluators draw • For intervention designers • clarifies whether assumptions embedded in the programme about “what works” are warranted • indicates how the programme might need to adapt to unforeseen contextual factors that influence implementation • enables designers to engage with programme participants’ perspectives on the programme, giving participants a voice • For EEF, helps to clarify • the relative effectiveness of different “theories of change” • the necessary and sufficient conditions under which different programme logics might work best

12. Sources of further information and support • Some reading • Lendrum, A. & Humphrey, N. (2012). The importance of studying the implementation of school-based interventions. Oxford Review of Education, 38, 635-652. • Durlak, J.A. & DuPre, E.P. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327-350. • Kelly, B. & Perkins, D. (Eds.) (2012). Handbook of implementation science for psychology in education. Cambridge: CUP • Oakley, A. et al (2006). Process evaluation in randomised controlled trials of complex interventions. British Medical Journal, 332, 413-416. • Organisations • Global Implementation Initiative: http://globalimplementation.org/ • UK Implementation Network: http://www.cevi.org.uk/ukin.html • Journals • Implementation Science: http://www.implementationscience.com/ • Prevention Science: http://link.springer.com/journal/11121