Dae Y. Kim, Principal Investigator Amy Cassata, Co-Principal Investigator

1. Scientific Validation Of A Set Of Instruments Measuring Fidelity Of Implementation (FOI) Of Reform-based Science And Mathematics Instructional Materials Dae Y. Kim, Principal Investigator Amy Cassata, Co-Principal Investigator Jeanne Century, Co-Principal Investigator IES Grantee Panel on Fidelity of Implementation September 7, 2011

2. Instrument Development • NSF-Funded Development Project (2007-2010) • Developed a framework for measuring the enactment of five reform-based mathematics and science curricula • Created a suite of instruments based on this framework • Driving Research Questions • What are the enacted components of reform-based instructional materials in mathematics and science that are common across specific programs? • How can we rigorously and specifically measure multiple components of implementation? • How can we collect data on and accumulate knowledge about implementation across multiple programs?

3. Instrument Development • Reform-based Curricula • Full Option Science System (FOSS), Science and Technology for Children (STC), Science Companion, Science Education for Public Understanding Program, and Everyday Mathematics • “Critical Components” • The measurable elements of the intended program model that are essential to its implementation • May be shared across programs or unique to specific programs • Can be organized into general categories • Structural: design, procedures, organization, and built-in supports • Instructional: expectations for participants’ actions, behaviors and interactions as they enact and engage in the intervention (users and recipients)

4. Framework for Measurement • Process of Critical Component Identification • Review of written materials to identify explicit and implicit components • Interviews with program developers to articulate the “intended” program model and critical components • Conversations with program users about critical components • Iterative revision, reconciliation, and modification of critical components list and framework for organizing them

5. FOI Framework for Instructional Materials

6. Measuring Implementation • Suite of instruments measuring the operationalization of the identified critical components • Teacher Instructional Questionnaire • Teacher Instructional Log • Teacher Instructional Observation Protocol • Teacher Interview Protocol • School Leader Questionnaire • School Leader Interview Protocol • School-Wide Observation Protocol • Pilot tested in 200 classrooms in CPS (2008) and revised • Field-tested in 265 classrooms in CPS (2009)

7. Measuring Implementation

8. Classroom-Level Instruments

9. Next Steps: Validation • IES Funding: Goal 5 (Measurement) Goals • Establish validity and reliability within and across the three classroom-level instruments • Develop a student questionnaire (grades 3-5) to triangulate measures of student engagement • Participants • 50 schools across 4 sites (IL, WA, MA) • All teachers grades K-5 (N~1000) • All consented students grades 3-5 (N~4500) • Schools use both Everyday Mathematics and a reform-based science curriculum

10. Year 1: Student Questionnaire • 20 to 25-item questionnaire • Literature review for items measuring Student Engagement critical components • Item development and pilot testing • Cognitive interviews • Consult with experts • Field testing • 300 students in grades 3-5 • Analyze factorial validity and internal consistency • Instrument revision

11. Year 2: Data Collection for Validation • All teachers will complete • Teacher Instructional Questionnaire – Math • Teacher Instructional Questionnaire – Science • All students (grades 3-5) will complete • Student Questionnaire – Math • Student Questionnaire – Science • 300 teachers with additional observations and logs • 100 teachers: 2 Math observations and 4 Math logs • 100 teachers: 2 Science observations and 4 Science logs • 100 teachers: 1 Math & 1 Science observation, 4 Math & 4 Science logs • Student achievement data (grades 3-5)

12. Year 3: Analysis • Inter-rater agreement (observation) • Internal consistency of items • Factorial structure of items • Differences in scores across different groups/content areas • Science vs. mathematics • Grade level • Study site • Cross-instrument consistency • Teacher Instructional Questionnaire vs. Student Questionnaire • Teacher Instructional Observation Protocol vs. Teacher Log • Predictive validity • Mathematics and science achievement (grades 3-5)

13. Predicting Student Achievement • To what extent do composite indices on the four subcategories (SP, SE, IP, ISE) predict student achievement in science and mathematics? • Planned analysis for predictive validity • 3-level fixed effects HLM model will explore the effect of the FOI composite indices measured at the teacher (classroom) level on individual students’ science and mathematics achievement (Grades 4-5) • Prior year’s science or math score will be entered as a covariate • Other variables: student demographics, classroom characteristics • Data for science and mathematics will be analyzed separately, then pooled • Data for questionnaires, logs, and observations will be analyzed separately

14. Summary • The FOI framework clearly and specifically describes the nature of reform-based mathematics and science instructional materials so these elements can be measured • The framework has resulted in a set of instruments which are currently being validated • The framework, and related instruments, have the potential to inform our understanding reform-based STEM curricula and to accumulate knowledge about the elements that make them most effective