Fidelity of Implementation in Scaling up Science Curriculum Units

1. Fidelity of Implementation in Scaling-up Highly Rated Science Curriculum Units for Diverse Populations Carol O’Donnell and Sharon Lynch The George Washington University Michael Szesze, Suzanne Merchlinsky Montgomery County Public Schools Prepared for the IERI PI Meeting September 10, 2004 Do not cite, quote, or distribute without permission from authors.

2. SCALE-uP: A Quasi-Experimental Design Design: 5 pairs of middle schools matched on demographic characteristics. Highly rated curriculum units randomly assigned to treatment or matched comparison schools. Pre- and posttesting. Sample: Approximately 3,000 MCPS middle school students for each grade level (6-8) per year.

3. Implementation Research Questions • Experimental - Are highly rated science curriculum units more effective than “standard menu”? If yes, what happens when data are disaggregated (gender, ethnicity, and FARMS, ESOL, and SPED eligibility)? • Ethnographic- How do these units function in middle school classrooms?

4. Scale-up Research Questions • Experience--Do students in schools in the first year of implementation have better outcomes than those in the second year? • Scale--Do students in schools at small scale (5 schools) have better outcomes than students in schools at large scale (37 schools)? • Fidelity of Implementation(FOI)--Do students whose teachers enact the curriculum as it was intended have higher outcomes than those whose teachers enact with less fidelity of implementation (measured by a classroom observation instrument currently under development).

5. Fidelity of Implementation: Definitions • Determination of how well a program is implemented in comparison with the original program design (Mihalic, 2002). • Degree to which program providers implement programs as intended by the program developer (Dusenbury, Brannigan, Falco, & Hansen, 2003).

6. Work to Date • Developed a classroom observation protocol to capture quality of delivery and participant responsiveness. • Piloted protocol before implementation, then during implementation (analysis in process). • Piloted pre- and post-observation interviews between teachers and observers. • Preliminary observations show that some aspects of classroom implementations are consistent with the unit’s instructional “intent”—high fidelity. Others were not—lower fidelity.

7. Teachers’ Questions about FOI • Can we modify the unit to meet the needs of diverse student populations (SPED, ESOL, etc.)? • What if we have requirements to meet state indicators (e.g. vocabulary) not covered by unit? • Can we use instructional practices we typically use in the classroom (e.g. exit cards, warm ups)? • How do we deal with student behavior issues? • Can we add supplemental readings?

8. Teachers' Questions about FOI Prompted a Set of FOI Guidelines:Fidelity Is… • Adhering to unit and lesson purpose, goals, and objectives. • Adhering to unit pedagogical approaches. • Following lesson sequence. • Using the recommended equipment or materials. • Making an adaptation to the lesson that does NOT change the nature or intent of the lesson.

9. Fidelity Is Not… • Reducing or modifying unit goals and objectives. • Reconfiguring the lesson so that other instructional practices gradually replace parts of the new unit. • Reducing student expectations inherent to the unit. • Varying grouping strategies outlined in the unit. • Changing the unit’s organizational patterns. • Substituting other curriculum materials or lessons for those described by the unit.

10. Teachers' Questions Prompted More Expansive Definition of FOI: Components • Adherence to the unit– Unit delivered as designed. • Exposure- Number of lessons; length of time. • Quality of delivery– The manner in which a teacher implements an intervention. • Participant responsiveness– The extent to which students are engaged by the intervention. • Program differentiation– Whether critical features (e.g. instructional strategies) that distinguish the intervention from the traditional curriculum are present or absent from implementation. Adapted from: Dane & Schneider (1998); Dusenbury, Brannigan, Falco, & Hansen (2003)

11. Questions to Ponder • Can we expect teachers and students to only exhibit instructional qualities inherent to the intervention? • How does a measure of FOI account for pre-existing good teaching practices vs. those prompted by the intervention curriculum units? • Why are certain aspects of instructional delivery consistently absent, despite unit support?

12. Next Steps • Finalize instrument that captures quality of delivery and program responsiveness. • Develop means of measuring adherence, exposure, and program differentiation. • Validate all FOI instruments. Test for reliability. • Quantify classroom FOI observation scores. • Analyze correlation between FOI and student outcomes.