School Selection and Randomization for a School RCT of a Universal Social-Emotional Learning and Literacy Intervention

1. School Selection and Randomization for a School RCT of a Universal Social-Emotional Learning and Literacy Intervention Joshua L. Brown Fordham UniversityConference on National and International Perspectives on Place-Based Randomized Trials in EducationInstitute of Human Development and Social Change New York UniversityOctober 3, 2008

2. NYC Study of Social and Literacy Development Principal Investigators Joshua L. Brown Stephanie M. Jones Fordham UniversityHarvard University J. Lawrence Aber New York University

3. Acknowledgements Research Team:Genevieve Okada, Site CoordinatorSuzanne Elgendy, Vanessa Lyles, Emily Pressler, RAsWendy Hoglund, Postdoctoral FellowMaria LaRusso, Postdoctoral FellowJuliette Berg, Catalina Torrente, GRAsProgram Partners:Tom RoderickAudrey MajorMorningside Center for the Teaching of Social Responsibility Funders:Institute for Education Sciences, DOENational Center for Injury Prevention and Control, CDCWilliam T. Grant Foundation National Institute of Mental Health

4. Outline Background/Context of Study Program and Study Design Pre-Randomization Activities Matching and Randomization Implications of Approach Current Preliminary Findings (Y1, Y1-2) Conclusions and Future Challenges

5. Background (1) Co-occurrence of social-emotional and behavioral problems with low academic achievement. Theoretical and initial empirical links between self-regulation and reading/math. Emphases on standardized testing and instructional improvement have crowded out attention to social-emotional-character development (among other things).

6. Background (2) Early efforts at whole school strategies to prevent behavior problems, violence, and substance use plagued by intervention design and implementation fidelity problems. Early research on whole school strategies plagued by low power, and inappropriate statistical analyses. Need to rigorously test promising but unproven approaches to SEL/SACD.

7. Background (3) Birth of “Social and Character Development” Research Network. 7 different interventions in 7 different sites. 7 Local Evaluations and 1 National Evaluation (Mathematica Policy Research).

8. Reading, Writing, Respect and Resolution (4Rs)Program and Study Design

9. The 4Rs Program • Universal, school-based intervention in literacy development, conflict resolution, and intergroup understanding. • 3 Primary components: • 7-unit literacy-based curriculum in conflict resolution and social-emotional learning. • Each unit organized around grade-appropriate book, includes 2 literacy activities, and 3-5 SEL lessons (21-35 total lessons). • Total possible activities per unit = 5-7 • Total possible activities per year = 35-49 • Training and ongoing coaching of teachers in the delivery of the 4Rs curriculum. • 25 hours introductory training • Ongoing classroom coaching, minimum 12 contacts • Learning kit • Family Connections • 1 parent-child “homework” per unit

10. Heuristic Model: 4Rs Child-Level Study Teacher Development Social-Emotional Skills & Behaviors 4Rs Experimental (classroom and parent) vs. Control Literacy Skills & Academic Achievement Extended Opportunities & Supports

11. Heuristic Model: 4Rs Setting-Level Study School Culture and Climate The Classroom System: Culture and Climate Teacher Affective & Pedagogical Processes & Practices Classroom Emotional, Instruct. & Org. Climate Child Developmental Outcomes: SEL & Academic Achievement 4Rs: Instruction, Teacher Training & Coaching Teacher- Child Relationships Child Behavioral Dispositions & Normative Beliefs

12. Overall Study Design 3-year, 6 wave longitudinal experimental design 18 NYC elementary schools matched and randomly assigned to 4Rs and control group (9 assigned to each group) Intervention is implemented school-wide, grades K-6 for 3 years All 3rd grade children in each school followed over three years through 5th grade Schools represent demographic character of NYC public elementary schools Racially/ethnically diverse; School lunch receipt ~70%; Mobility/Stability = ~18%/60%; Suspensions = 23%

13. Pre-Randomization Activities

14. Identifying Candidate Schools Planning Year (January-March, 2004) • History of practitioner’s work in NYC led to letters of support from Regional Superintendents • Facilitated direct contact with Local Instructional Superintendents who recommended schools based on: • no prior history of implementing 4Rs • willingness to implement program (and research) activities: • all Teachers participate in Intro. Training, teach curriculum ~1 lesson/week, and receive ongoing coaching from 4Rs Staff Dev’s • Principals attend 2-3 workshops/year and appoint “4Rs Liaison” • School administration and teachers cooperate with data collection • Resulted in LIS recommendations of 41 schools

15. Assessing Candidate Schools Planning Year (March-June, 2004) • Goal: assess/recruit “viable” schools for program implementation in context of research study, (i.e., capacity for sustained, high-quality implementation, but room to improve; willing to be randomly assigned) • Process: Meetings and “walk-throughs” of all 41 schools: • Individual meeting with Principals to present program model and overall research design • Visits to classrooms • “Organizational Readiness” assessment completed by practitioners (co-developed with research team)

16. Organizational Readiness • Principal Leadership • Organizational skills • Enthusiasm for/compatibility with 4Rs • Rapport with Students & Staff • Teachers & School Leadership Team • Relationship with Principal • Enthusiasm for/quality of questioning about 4Rs • Stress, morale & attitudes toward children • School Environment • Tone of adult-child interaction • Engagement and behavior of students • Physical environment (e.g., use of bulletin boards, etc.)

17. Recruiting/Selecting Schools • 17 schools eliminated from initial pool: • Grade structure other than K-5 (e.g., no 5th grade, K-3) • Lack of Principal and/or teacher interest in program • E.g., can’t support school-wide implementation requirement and/or balance competing academic demands • Highly chaotic environments (e.g., adult-adult/adult-child yelling, extensive behavior problems, frequent crises) • Unwilling to risk random-assignment to Control condition (one preferred to purchase program) • 24 schools held staff vote, signed letters of agreement for random assignment

18. Pairwise Matching of Schools • Given potential for “bad draw” with small number of schools, IES grantees agreed to pairwise match schools to ensure balance on key variables and increase precision • 24 schools pairwise matched and rank-ordered based on “distance” of each school from every other eligible school across 20 key school characteristics  12 pairs School Characteristics Include: • Size (total N) • Race/ethnic and gender composition • School lunch receipt • Attendance (Students and Teachers) • Reading achievement (% of students at or above proficiency on ELA test) • Within year student mobility/two-year stability • Teacher full licensure and years of experience • Expenditures • Organizational Readiness (Overall)

19. Matching  Random Assignment • Funding for 18-schools, kept 9 best matching pairs, but maintained 3 back-up pairs during lead-up to program implementation/data collection • Random numbers generator used to assign 1 school in each pair to intervention and 1 to control conditions • Post-random assignment, 2 schools and their respective matches were dropped and 2 back-ups engaged • Principal of Tx school had been previously trained in RCCP • LIS ultimately did not condone RCT design for her schools • Note, pairwise matching can protect the experimental design (from selection bias) in case of schools dropping out after start of study (King et al., 2007)

20. Implications of Matching for Analyses Should “blocks” (e.g., matched pairs) be regarded as fixed or random effects? Current debate in field, depends on: Number of units per block (when only 2, need RE model) Treatment effect heterogeneity (i.e, across matched pairs) If large, RE model allows heterogeneity of Tx effect to contribute to standard errors and tests for the average effect of Tx. Interest in generalizability In FE model, the blocks constitute the population or universe of generalization In RE model the blocks are seen as representing a larger universe of possible blocks (or settings) in which Tx might be implemented. See Schochet, 2004; Raudenbush, 2004; and Bloom, 2005; all SACD internal network communications We estimate blocks as random effects at the school-level – most conservative

21. Implications of Approach Evidence of effective matching process -- no Tx/Control differences in 20 initial matching variables, or baseline constructs assessed via child, teacher, and parent-reports Focus on initial identification and subsequent selection of “viable” schools limits generalizability to district- or citywide population However, features of school organizational capacity and support have been clearly linked to schools’ ability for quality program implementation (Payne, Gottfredson & Gottfredson, 2006) A fully representative sample of NYC elementary schools would yield many schools with weak engagement and early withdrawal from study Although generalizable only to “willing” implementers, we see preliminary evidence of Tx impacts

22. Year 1 Child Impacts (Jones et al, under review) Main effects of Tx on 2 of 9 child outcomes (2L HLM) Controlling for baseline levels, children in the Tx group had lower mean levels of Hostile Attribution Bias and Depression than those in the control group at the end of Y1 Tx by baseline covariate interactions for 5 of 9 outcomes E.g., Tx by baseline Behavioral Risk (“elevated” on teacher-report aggression and/or conduct problems at baseline) Children with the highest level of baseline behavioral risk show the greatest positive difference in Aggressive Fantasies, Teacher-Report of Academic Skills, Reading Scale Score, and Attendance between the intervention and control groups

23. Year 1 Classroom-Level Impacts (in Effect Sizes) • Classrooms in the Tx group had higher mean Overall Classroom Quality scores, accounted for by higher mean Emotional Support • and Instructional Supportscores, than the control group * * * n.s.

24. Preliminary Year 1-2 Program ImpactsChild-Level

25. Results: Child-LevelTX Main Effects Local: significant impacts for 2 of 6 constructs Child Self-Report Hostile Attributional Biases Child Self-Report Depression Multisite: significant impacts for 3 constructs Teacher-Report of Aggression Teacher-Report Social Competence Teacher-Report ADHD Symptoms

26. Y1-Y2 TX Main Effects: Summary

27. Control Treatment TX on HAB Slope

28. Conclusions and Future Challenges Trade-offs between generalizability and design feasibility in school selection process for school RCTs. We opted for selectivity based on: LIS perspective of school need/capacity Pre-randomization assessment by program practitioners of school organizational readiness, used to identify final sample Need rigorous and field efficient assessment tools that tap multiple dimensions of school organizational capacity How might RCT design innovations enable the inclusion of disorganized and at-risk schools most in need of intervention?