Challenges of undertaking RCTs in Education SIG Workshop Monday 18 th March 2013

1. Challenges of undertaking RCTs in EducationSIG Workshop Monday 18th March 2013 Hannah Ainsworth, Research Fellow, York Trials Unit

2. School Based Educational Trials • ECC Trial 1 – one-to one numeracy intervention for Year 2 children struggling in maths • ECC Trial 2: Pairs and Triplets – small group numeracy intervention for Year 2 children struggling in maths • 3 Writing Trials – all looking at the impact of a writing intervention on children in the transition from Year 6 to Year 7, focus on children performing below national average

3. Trials with University Students • Student Nurses Numeracy – ‘Authentic World’ computer based instruction aimed at improving maths skills needed in the workplace • Hypnotherapy Trial – hypnotherapy to reduce exam anxiety and improve exam performance students

4. School Based Health Trials • CLASS – Children Learning About Second-hand Smoke – reducing exposure to second-hand smoke (Year 5 and Year 6) • PIP – Pre-schoolers in the Playground – increasing exercise • ASSSIST – Autism Spectrum Social Stories in Schools Trial – improving social skills and behaviour (ages 5 – 15, primary and secondary schools)

5. School Recruitment and Engagement • Recruiting schools to take part in trials can be challenging • Competing demands • Relevance to ‘their’ school • Getting past the gatekeeper • Keeping schools engaged throughout the duration of the trial • Disappointed at allocation • Staff turnover • Competing demands

6. School Recruitment and Engagement

7. School Recruitment and Engagement • Hold a recruitment conference • Visit schools • Clearly explain randomisation and the need for a control group • Use existing contacts and networks • Ask them to sign an agreement to participant form • Manage expectation • Be clear about their responsibilities and how you will help • Keep in contact

8. Use examples to explain

9. Produce Clear Evaluation Diagrams Primary Schools n = 24 Secondary Schools n = 3 • Children in target group n = 288 • (based on average 12 children per school) • Inclusion criteria: • Yr 6 • predicted to achieve Level 3 or an insecure level 4 in English by the end of Key Stage 2 (based on teacher assessments conducted at the end of Autumn term 2012) Baseline data collection Information on all Year 6 pupils including Key stage 2 English Teacher Assessments from Dec 2012 Primary School Randomisation Control Group Primary Schools N = 12 No intervention (Primary Schools will receive intervention next academic year) Intervention Group Primary Schools N = 12 Improving Writing Quality intervention in Year 6 continued intervention in Year 7 in Secondary Schools. • Follow up data collection Dec 2013 • Progress in English 11 (long form)(Conducted in Secondary School) Long term follow up Routine test results and pupil characteristics recorded in National Pupil Database

10. School Agreement From

11. Clear instructions for schools

12. Pupil Recruitment and Parent Engagement Notoriously difficult to recruit participants in to trials generally • Limited access/face to face contact with parents • Schools have the contacts, they need to help • Information can be overwhelming • Parents need to see a reason ‘why’ this is good for ‘their’ child

13. Pupil Recruitment and Parent Engagement ECC Trials: Each school had to identify approximately 12 eligible children and gain parental consent, almost all participating schools were able to achieve this – very little problems with gaining parental consent

14. Student Recruitment

15. Pupil Recruitment and Parent Engagement • Resources to put people on the ground • Provide simple information in creative ways • Use existing relationships (trust) • Consider incentives • Consider collecting anonymous data • Consider an opt-out approach to consent

16. Information for Parents • Detailed information to meet ethical requirements • Simple version • Use a short DVD • Use assemblies and parents evenings

17. Information for Children

18. Data Collection • Needs to fit in with academic timetable • Delays to study set up can seriously impact the whole trial • Data collection often needs to occur at the beginning and end of terms – difficult time for schools • You might not be able to be there in person – placing trust in others. • May need to coordinate large scale testing in a short period • Unexpected events!

19. Data Collection • Allow plenty of time in funding plan for study set up • Give schools as much notice as possible • Try to do as much of the donkey work as possible • Keep testing and data collection burden to a minimum • Be clear and repeat instructions • Don’t leave anything to chance • Expect the unexpected!

20. Relationship with intervention developer • Can be a challenging relationship • Developer has a lot invested in the intervention • They may try to influence trial design and conduct

21. Relationship with intervention developer • Manage expectations • Try to explain that it important you remain impartial • Refrain from voicing your own opinions about the intervention – let the research speak for itself • Stand your ground

22. Relationship with Funder • Political context • Bigger picture / Wider aims • High staff changeover • May be more familiar with trials in health • May be underfunded

23. Understand the question that can actually be answered • Example ECC Trial • We really wanted to conduct long term follow up, but the funders and developers were not supportive • Positive short-term impact of receiving Numbers Count (NC) compared with not receiving NC

24. Ethics and Other Approvals • Need to ensure you acquire the necessary approvals – can be time consuming • CLASS trial – collecting saliva samples, had to go through NHS ethics approval process, square peg in a round whole • Hypnotherapy for exam anxiety trial • Examination board concerned students would claim unfair disadvantage

25. Outcome Mesaures • Find a suitable outcome measure • Ideally this outcome measure needs to be able to be conducted independently and blind to allocation • Ideally not a treatment/intervention inherent measure (designed specifically for use with the intervention e.g. questions use same format as questions in intervention)

26. ECC Example • Primary Outcome was conducted by independent testers blind to allocation • Secondary Outcome conducted by teachers and was a treatment inherent measure

27. ASSSIST Example • Very difficult to identify an appropriate outcome measure • Unable to find any standardised tools that are measure the changes we hope to capture • Each Social Story will have a different aim for each child • Using a combination of measuring frequency of behaviour and using a goal rating scale

28. Intervention Fidelity • Fidelity is the extent to which the intervention, as realised, is “faithful” to the pre-stated intervention model • Little consensus on meaning of the term “intervention fidelity. • But Dane & Schneider (1998) identify 5 aspects: • Adherence– program components are delivered as prescribed; • Exposure – amount of program content received by participants; • Quality of the delivery – theory-based ideal in terms of processes and content; • Participant responsiveness – engagement of the participants; and • Program differentiation – unique features of the intervention are distinguishable from other programs (including the counterfactual)

29. Intervention Fidelity Student Nurses Numeracy Example • Very low compliance • Actually 2 very similar trials in 2 different universities • Only 24% and 12% of students allocated to the intervention group spent more than 15 minutes using the programme

30. Intervention Fidelity • Measure fidelity to intervention • Always conduct ITT analysis • But can also use CACE analysis to take into account non-compliance • Remember if you are undertaking a ‘pragmatic trial’ then compliance/fidelity is an important outcome in itself

31. BUT RCTs in Education are Important! • Robust design for answering questions of effectiveness and cost effectiveness • Although challenging they are possible! • Historically has been a serious lack of robust evidence from RCTs in education but things are changing • We need to be prepared for the challenges and ready to find solutions

32. Any Questions? Hannah Ainsworth hannah.ainsworth@york.ac.uk 01904 328158