Maintaining research rigour in evaluations of complex interventions

1. Maintaining research rigour in evaluations of complex interventions Laurence Moore

2. Learning Objectives (1) • To be aware of frameworks for the development and evaluation of complex interventions • To be aware of value (and added value) of complementary mixed methods • To understand value of pragmatic CRTs with embedded process evaluation • To be aware of examples of successful CRTs of complex interventions

3. Learning Objectives (2) • To understand why cluster randomised trials can be the preferred design • Situations when individual randomisation is not feasible • Contamination • Limitations of Quasi-experimental designs • To be aware of specific issues in the conduct and analysis of CRTs • Risk of baseline imbalance • Intra cluster correlation and the design effect • To be aware of design and analysis strategies to respond to these issues • Sample size calculations • Randomisation methods • Analysis of cluster randomised trials

4. Mixed methods Quantitative • Methods: • Outcome / summative • Intermediate outcomes / process measures • Research questions: • What works? • What is effect? • How many, how much?

5. Qualitative • Methods: • Observations, field notes, diaries, records, videos, interviews, focus groups • Process • Research questions: • Why? • How? • Barriers / facilitators

6. Complementary use of mixed methods: frameworks in health • Fit for purpose • Match method to question • Staged, series of studies • PRECEDE / PROCEED • Nutbeam model • MRC framework

7. Phase 5 Administrative & policy assessment Phase 4 Educational & ecological assessment Phase 2 Epidemiological assessment Phase 1 Social assessment Phase 3 Behavioral & environmental assessment Predisposing Factors Health Program* Health Education Behavior Reinforcing Factors Quality of Life Health Policy Regulation Organization Environment Enabling Factors PRECEDE-PROCEED Framework* Formative evaluation & baselines for outcome evaluation* Intervention Mapping & Tailoring* Phase 6 Implementation Phase 7 Process evaluation Phase 8 Impact evaluation Phase 9 Outcome evaluation *New in 4th ed., Green & Kreuter, Health Promotion Planning, in press.

8. Stages of Research and Evaluation for Health Promotion Programs Problem Solution Innovation Intervention Intervention Program definition Generation Testing Demonstration Dissemination Monitoring Epidemiology and demography Social, behavioural and organisational research Community needs analysis Intervention theory development Intervention literature search, meta- analysis Assessment of cost and benefits (financial, social., political) Performance monitoring Assessment of outcome Pre- Testing methods and materials Understanding of Process What is the How might it Did the Can the Can the Can the problem? be solved? solution program be program be program be work? repeated/ widely sustained? refined? reproduced? Key Research Questions

9. Phases of RCTs of complex interventions: MRC April 2000

10. Complementary use of mixed methods • Fit for purpose • Match method to question • Staged, series of studies • PRECEDE / PROCEED • Nutbeam model • MRC framework • In combination within one study • What works?, how?, for whom? and in what circumstances?

11. Public Health Improvement: Evidence base conundrum • Good quality trials successfully conducted, evaluating weak interventions. Small or zero effect sizes. • Good quality complex interventions evaluated using weak research designs. Biased effect estimates.

12. Challenges in applying RCTs to evaluation of complex social interventions • Recruitment and retention • Scale and Cost • Ethics • Research and Policy Timescales • Implementation

13. Challenges in applying RCTs to evaluation of complex social interventions • Variability in delivery • Context dependence • Generalisability / implementation

14. Types of intervention in which individual randomisation is difficult or impossible: • Interventions that entail changing the organisation of services in a given unit or area • Interventions targeted at changing the behaviour of professionals • Community programmes • ‘Settings’ based interventions, such as workplace or school interventions • Interventions targeted at individuals but based on social processes

15. Risk of contamination • (though see Puffer & Torgerson)

16. Problems with quasi-experimental designs • Selection bias – external & internal validity • Imbalance at baseline, often not measurable • Different trends at baseline • Ethical considerations

17. Cluster (group) randomised trials • ASSIST Peer-led smoking intervention -MRC • 59 schools randomised • Fruit tuck shops - FSA • 43 schools randomised • Free Breakfast Initiative - WAG • 111 schools randomised • Emergency contraception - DH • 25 schools randomised

18. Variability in delivery • RCTs traditionally require that interventions are standardised and uniformly delivered • (efficacy trial) • Social interventions highly dependent on quality of delivery • Value of efficacy trials limited • eg. school smoking education • Results of efficacy trials involving enthused teachers not replicated in roll-out

19. Efficacy and effectiveness • Efficacy trial • To test whether the treatment does more good than harm when delivered under optimal conditions • Effectiveness trial • To test whether the treatment does more good than harm when delivered via a real-world program in realistic conditions • Pragmatic, allowing variability in delivery as would be experienced in real world

20. Context dependent • Social interventions often highly dependent on the context within which they are delivered • Argued therefore that RCTs not suited to their evaluation • However, RCT design has the advantage that randomisation process ensures that systematic differences in external influences between groups do not occur • Generally use stratification or minimisation to minimise imbalance due to small no. of units • Will achieve unbiased estimate of average effect

21. Generalisability • Efficacy trials may demonstrate that intervention has ‘active ingredients’ that work • Effect unlikely to be reproduced in real world • Attenuated by context and implementation • Generalisability of small trials with (e.g.) one educator will be limited

22. Effectiveness trials with embedded process evaluation • Effectiveness trials, implementing interventions in a manner reproducible in real world • Realistic level of flexibility allowed, but not adaptation or reinvention • Crucial to conduct a comprehensive process evaluation (largely qualitative) within such a trial • Monitor variability in context and delivery • Identify barriers / facilitators • Relate variability in these factors to variability in intervention impact

23. MRC Assist TrialPeer-led smoking intervention • Theory based (Diffusion of innovations) • Developed from similar approach used in sex education • Extensively piloted • Feasibility trial conducted in 6 schools • Funding for main trial (59 schools) sought and obtained from MRC

24. ASSIST Trial • Intervention led by specialists, as would be the case if rolled out in the real world • Not to be implemented by untrained, unmotivated teachers • Process evaluation in all 30 intervention schools, with parallel measures in the 29 control schools • In-depth process evaluation in sub-sample • Observations, field notes, diaries, records, interviews with pupils, teachers, staff

25. Challenges in embedding process evaluation within trial • Hawthorne effects • Distinguishing team roles • Differentiating intervention and evaluation activities • Volume of data • Sampling • Analysis plan • Power balance

26. Randomised trials of health promotion interventions: feasible? valuable? • Not always! • Cluster randomised design • Pragmatic, effectiveness trials • Unbiased estimate of overall intervention effect • Additional qualitative and quantitative data collection to measure variation in context, process, delivery and outcome • Identifies issues for further development of intervention / further testing of its (variable) effect • Crucial for implementation stage • Hypothesis generation, not testing

27. Workshop: Analysis of trials (cluster randomised) • Statistical issues • Design effect, context, implementer, cluster effects • Multilevel analysis • Synthesis of qual/quant data

28. Cluster randomisation • Randomise the cluster rather than the individual • Generally a small number of clusters • Four per group an absolute minimum • Use restricted randomisation to ensure balance in number of clusters per group • Use stratification or minimisation to minimise imbalance in group characteristics • Matched pair design popular, but some drawbacks

29. Standard statistical methods, when applied to cluster randomised trials, will (usually) lead to: • Sample size calculations that are too small • Confidence intervals that are too narrow • P-values that are too small

30. Intra-cluster correlation • The proportion of the true total variation in the outcome that can be attributed to differences between the clusters:

31. Design effect • The ratio of the variance of the outcome under the cluster sampling strategy to the variance that would be expected for a study of the same size using simple random sampling: deff = 1 + (n-1)

32. Sample size inflation=0.01, m=20 n` = deff * n n = 360 (or 18 classes) per group deff = 1+(m-1) = 1+(20-1).01 = 1.19 n` = 1.19 * 360 = 428.4. i.e. 429 pupils per group Number of classes required = 429/20 = 21.4 i.e. 22 classes per group

33. Sample size inflation=0.02, m=200 n` = deff * n n = 360 (or 2 schools) per group deff = 1+(m-1) = 1+(200-1).02 = 4.98 n` = 4.98 * 360 = 1792.8. i.e. 1793 pupils per group Number of schools required = 1793/200 = 9.0 i.e. 9 schools per group

34. Further reading • Donner A, Klar N. Design and analysis of cluster randomization trials in health research. London: Arnold, 2000. • Murray DM. Design and analysis of group-randomized trials. Oxford: OUP, 1998. • Ukoumunne OC, Gulliford MC, Chinn S, Sterne JAC, Burney GJ, Donner A. Evaluation of health interventions at area and organization level. BMJ 1999:319:376-379. • Ukoumunne OC, Gulliford MC, Chinn S, Sterne JAC, Burney GJ. Methods for evaluating area-wide and organisation-based interventions in health and health care: a systematic review. Health Technol Assess 1999;3(5). (http://www.hta.nhsweb.nhs.uk/). • Elbourne DR, Campbell MK. Extending the CONSORT statement to cluster randomised trials: for discussion. Stats Med 2001;20:489-496. • Puffer S, Torgerson D, Watson J. Evidence for risk of bias in cluster randomised trials: review of recent trials published in three general medical journals. BMJ, Oct 2003; 327: 785 - 789

35. www.cardiff.ac.uk/schoolsanddivisions/academicschools/socsi/staff/acad/moore/ • F. Starkey, L. Moore, R. Campbell, M. Sidaway, M. Bloor. Rationale, design and conduct of a comprehensive evaluation of a school-based peer-led anti-smoking intervention in the UK: the ASSIST cluster randomised trial [ISRCTN55572965]. BMC Public Health 2005, 5:43. 22nd April 2005. http://www.biomedcentral.com/1471-2458/5/43 • L. Moore, A. Graham, I. Diamond. On the feasibility of conducting randomised trials in education: case study of a sex education intervention. British Education Research Journal 2003;29:673-689. • L. Moore, R. Campbell, A. Whelan, N. Mills, P. Lupton, E. Misslebrook, J. Frohlich. Self-help smoking cessation in pregnancy: a cluster randomised controlled trial. British Medical Journal 2002;325:1383-1386. • A. Graham, L. Moore, D. Sharp I. Diamond. Improving teenagers’ knowledge of emergency contraception: results of a cluster randomised trial. British Medical Journal 2002;324:1179-1183. • L. Moore, C. Paisley, A. Dennehy (2000) Are fruit tuck shops in primary schools effective in increasing pupils’ fruit consumption? A randomised controlled trial, Nutrition and Food Science 30(1) 35-38.

36. Analysis of trials (cluster randomised) • Analysis plan • Multiple outcomes • Primary and secondary analyses – adjust for baseline / stratifiers • A priori plan (register and publish) – not baseline testing • To include effect modifiers • Design effect • Clustering • Practitioner effect • Even in individual RCTs across clusters (context effect) • Multilevel analysis • Hypothesis generation / informing implementation • Synthesis • Triangulation, discordance

37. Nothing worse than a poorly conducted trial • Complex intervention trials very challenging • DO IT WELL – GET ADVICE!!

38. Cluster randomised trials Laurence Moore Cardiff Institute of Society, Health and Ethics Email: MooreL1@cf.ac.uk Tel: 02920 875387