TRANSFORMING EDUCATION THROUGH EVIDENCE.

1. TRANSFORMING EDUCATION THROUGH EVIDENCE. The Centre for Effective Education SCHOOL OF Education Conducting Educational Randomised Control Trials in Disadvantaged Community Settings Andy Biggart & Liam O’Hare

2. Presentation Overview • Introduction to research context and programmes • Overview of Key Results • Conextualising Results in Wider Literature • Some methodological and ethical reflections of conducting RCT’s in community settings The Centre for Effective Education | SCHOOL OF Education

3. Research Context • Educational Research & Randomised Control Trials • The Atlantic Philanthropies • Some of first contemporary trials in UK and Ireland • The two progammes grew out of a consultation with the local community in Tallaght (Dublin) The Centre for Effective Education | SCHOOL OF Education

4. Two After-schools Programmes • Doodle Den • Newly developed after-schools programme • Aimed at 5 and 6 year olds identified as struggling readers • Adopted a balanced literacy framework • Delivered by teacher and youth worker/child care assistant • Ran throughout the school year • 3 days a week for 1.5 hours afterschool • Highly structured programme with strong emphasis on professional development The Centre for Effective Education | SCHOOL OF Education

5. Two After-schools Programmes • Mate Tricks • Very similar in delivery to Doodle Den • Hybrid Pro-social BehaviourProgramme: Coping Power and Strengthening Families Program • Aimed at 9 and 10 year olds • Universally applied in schools in Tallaght • Parental Components • Designed to enhance children’s pro-social development, reduce anti-social behaviour, improve parenting skills and parent/child interactions The Centre for Effective Education | SCHOOL OF Education

6. Common Methodology • Individually randomised trials • Qualitative Process Evaluation • Children recruited on basis of teacher referrals • 3 year rolling cohort design • Over 600 children allocated to either intervention or to control: ‘business as usual’ • Child, parent and teacher measures • Outcomes measured according to pre-validated measures where possible. The Centre for Effective Education | SCHOOL OF Education

7. Summary Outcome Results – Doodle Den • Primary Outcomes: • Statistically significant effects on both direct child and teacher assessed measures of literacy (d=0.17-0.3) • Secondary Outcomes: • Statistically significant effect on children’s behaviour in regular school class (d=0.18) • No impact on attendance rates at school • Exploratory Analysis • No significant differences in terms of literacy • Boys behaviour improved relative to control group • Greater attendance at the programme led to more positive outcomes The Centre for Effective Education | SCHOOL OF Education

8. Summary Outcome Results – Mate Tricks • Primary Outcomes: • No statistically significant effects on pro-social behaviour • Two significant negative effects on anti-social behaviour (d=0.18; 0.20) • Secondary Outcomes: • No statistically significant positive effects on wide range of secondary measures • Some significant negative effects in relation to child reported parenting practices • Exploratory Analysis • Child programme attendance not related to outcomes • Parental engagement with the programme significant predictor of a range of positive outcomes The Centre for Effective Education | SCHOOL OF Education

9. Conextualising Results – Doodle Den • First randomised trial of an after-school program in UK or Ireland focused on academic outcomes • Mixed results of previous trials of the effectiveness on academic outcomes • e.g. 21st Century schools U.S. national evaluation • Reviews of evidence highlight complexity of the field • Recent promising practice guides have concluded a need for much more research on what works in after-school settings • Potential features driving positive outcomes: • Age of children; targeted at those most in need, well structured clearly defined curriculum, qualified staff with focus on professional development and fun engaging activities The Centre for Effective Education | SCHOOL OF Education

10. Conextualising Results – Mate Tricks • Hybrid programme with some evidence of effectiveness behind the two programmes • No evidence of effectiveness of Mate Tricks in improving behavioural outcomes and some negative effects apparent • Some evidence of in-school programmes focusing on social and emotional learning having promising effects e.g. Incredible years • However, previous reviews have also highlighted null or adverse effects in after-school contexts • The current study adds to existing evidence that behavioural programmes have the potential to cause adverse effects as well as benefits • r The Centre for Effective Education | SCHOOL OF Education

11. Some Lessons Learned: Methodological Issues Despite the critics, it is possible to implement RCTs in real world community settings. Making key stakeholders aware of key principle’s of RCT evaluation designs A willingness to be flexible, without undermining the core of the study and randomisation process Developing and maintaining good relationships Keeping attrition to a minimum without overburdening participants Importance of measurement and agreeing analysis plan in advance RCT’s good at assessing intended outcomes but good process data should not be neglected The Centre for Effective Education | SCHOOL OF Education

12. Ethical Issues and RCT’s Equipose Randomisation perceived as a fair way of allocation Ethical Issues may arise at any stage Potential for burden Dissemination and not spoiling the field, especially when faced with null or negative findings The Centre for Effective Education | SCHOOL OF Education

13. EVIDENCE MATTERS!!! We intervene in children’s lives on a regular basis! We may also do damage when we intend to do good! Interventions often driven by fashion rather than sound evidence! Do we owe it to children to try and ensure that the policies they are subjected to are effective in improving their life chances? Both reports available on CEE website: www.qub.ac.uk/cee The Centre for Effective Education | SCHOOL OF Education