Lessons Learned About Random Assignment Evaluation Implementation in Educational Settings

1. Lessons Learned About Random Assignment Evaluation Implementation in Educational Settings SREE Conference March 4, 2010 Raquel Sanchez and Fannie Tseng

2. Introduction • Hands-on overview of our experiences implementing random assignment evaluations in the classroom. • Extending list of lessons discussed in past literature. • Brief description of the random assignment evaluations upon which our experiences were drawn. • Discuss difficulties with implementing random assignment in classroom settings. • Discussion of lessons learned

3. Past Literature on Random Assignment Implementation • Gueron (2002) • Ritter and Holley (2008) • Raudenbush (2005) • Burghardt and Jackson (2007)

4. Overview of Our Random Assignment Evaluation Studies • Two school-level random assignment studies of the effectiveness of professional development programs that focus on developing the reading comprehension skills of English language learners (ELLs) • One center-level random assignment study of a professional development program targeting caregivers of children ages 0-3 • One student-level random assignment study of a curriculum that combines explicit and implicit approaches to instruction in increasing the literacy skills of adult ESL students

5. Challenges to Implementing RCTs in Educational Settings • Threats to integrity of random assignment • Crossovers and contamination • Recruiting and obtaining buy-in • Dilution of intervention effectiveness • Lack of teacher buy-in • Effect of crossovers and contamination • Documenting treatment dosage • Conflicting interventions on the ground • Local conditions and circumstances

6. Lessons Learned • Perform in-person recruiting visits at all levels of school administration to maximize buy-in • Foster good communication between school staff, program developers and research team • Follow-up data collection requires persistence, patience and adequate funding • Keep in touch with assessment data administrators, even in off-months of study • If possible, retain local research staff • Use conservative statistical power calculations to factor in potential implementation challenges

7. Statistical Power Example 1

8. Statistical Power Example 2

9. Lessons Learned (2) • Throughout the course of the study, establish a separate identity from the program you are evaluating • Hand out separate evaluation study materials with research organization’s logo • Gift cards help • Be proactive about developing plans for documenting treatment dosage • If possible, use more than one source of data • Importance of qualitative studies to accompany impact studies

10. Contact Us • Raquel Sanchez, Ph.D.raquel@bpacal.com • Fannie Tseng, Ph.D.fannie@bpacal.com Berkeley Policy Associates 440 Grand Ave., Suite 500 Oakland, CA 94610-5085 Ph: 510-465-7884 Fax: 510-465-7885 www.berkeleypolicyassociates.com