Research into research use:

1. Research into research use: A ten-arm cluster RCT 10th September 2014 Dr Ben Styles, Dr Anneka Dawson (NFER), Dr Lyn Robinson and Dr Christine Merrell (CEM)

2. Background on evidence- based education What research use experts said: • How are you going to measure outcomes? • Can you expect to see any difference?

3. Research dissemination: competing ideas Active methods can be expensive Measurement of attainment is cheap (National Pupil Database) Passive methods are rarely effective Passive methods are cheap i.e. a tiny effect is still cost effective Several interventions trying to achieve the same goal Measurement of research use is complex and expensive Passive methods may not require consent

4. Trial design Two RCTs focused on improving Key Stage 2 literacy through teachers’ research use: • Active trial- 780 primary schools randomised to experience one of varied activities from CPD to online communities (or control) • Passive trial- Most other English primary schools randomised to receive one of various research-based communications (or control)

5. Outcomes

6. How many arms in the active trial? • Four providers of research evidence, each with an active and passive approach • Control group • CEM need three arms

7. Active trial 780 schools recruited 240 60 60 60 60 60 60 60 60 60

8. Passive trial 12,500 schools 2500 2500 2500 2500 2500

9. Timeline

10. CEM intervention

11. Challenge 1-Recruitment Starting point- why do schools want to take part in research?: • Having their say • Doing something good for education • The process being easy and sounding easy • Trust in the organisation approaching them • Security of data • Clear concise communication • Offer of feedback

12. Our recruitment approach • Promotion at Research Ed and other provider events • Word of mouth between schools and providers, NFER, EEF • Website with detailed information on the trial and what is expected of schools and form to express an interest • Letters, phoning, emails targeted response • Helping inform schools about RCTs including our guide for schools

13. Recruitment (continued)

14. Challenge 2- Secondary outcome measurement • Secondary outcomes are ‘awareness’, ‘understanding’ and ‘action’ • Measure using a bespoke survey • Schools involved in other research projects using this survey cannot be in this trial

15. Research use outcomes • Baseline survey developed without pilot • ‘Awareness’, ‘understanding’ and ‘action’ assessed through a series of questions in a survey of teachers • Follow-up survey will be piloted in autumn • Questions will be retained if they discriminate well and contribute to reliable measures

16. Intention-to-treat analysis • Attainment (multilevel model, attainment at age 7 as covariate) • Research use is also hierarchical but low numbers per cluster (research use at baseline as a covariate)

17. Challenge 3- Coordination of the delivery organisations With 6 delivery partners carrying out different interventions things to consider: • Effective, timely communication (inc one main point of contact at NFER and one at each provider) • Funder as mediator for more basic updates • Monitoring information planned in advance

18. Acknowledgements • Education Endowment Foundation • NFER Education Trials Unit

19. For more information For Literacy Octopus project information: www.nfer.ac.uk/octopus For our guide for schools: http://www.nfer.ac.uk/publications/RESM07/RESM07_home.cfm For information on RCTs for educational researchers: http://www.nfer.ac.uk/publications/RCT01/RCT01_home.cfm For the NFER Education Trials Unit with details of our other trials http://www.nfer.ac.uk/research/trials-unit/