Methods and Tools for Measuring Fidelity

1. Methods and Tools for Measuring Fidelity Greg Roberts, PhD. Vaughn Gross Center & National Center for Instruction The University of Texas at Austin

2. Notes on Fidelity • Fidelity is the match between intended and actual • Considerations for conceptualizing fidelity • Multilevel nature of many interventions • Person and group • Outcome and process • Level and intensity of measurement aligned with need and with likely quality of data • Alignment with desired outcomes

3. More Notes on Fidelity • Capacity for monitoring fidelity • Burden of monitoring fidelity • Tools for monitoring fidelity • Labor and cost intensity…for what? • Prospective versus retrospective purpose

4. Fidelity-related Tools • Teacher self reports • Teacher logs • Observation protocols • Little evidence supporting their equivalence (see for example, Burstein, McDonnell, Van Winkle, Ormseth, Mirocha & Guiton, 1995; Porter, Kirst, Osthoff, Smithson, & Schneider, 1993)

5. Teacher Self Reports • Methods • Survey • Focus group • Interview • Relatively cost efficient • May be externally valid

6. Teacher Self Reports • Response bias • Recall • Social desirability • Limited range of sampled behaviors • Suspect predictive validity

7. Teacher Self Reports • Research on enacted curriculum • Federal efforts to increase utility • Item type • Item content • Response format • Informal efforts

8. Teacher Logs • Standardized materials • Initial and ongoing training re: use • Prospective analysis plan • Loads of data • Data entry and management • Analysis

9. Teacher Logs • Study of Instructional Improvement (Camburn & Burns, 2003) • Evidence on the validity of these logs in represented enacted curricula. • Literacy instruction (Rowan, 2004) • varies day-to-day • varies across teachers (greatly) • differs (predictably) across educational reform programs (e.g., Success For All, America’s Choice)

10. Classroom Observations • External to teacher • Eliminates response bias • Greater breadth of sampled behavior • Rater reliability • Training • Monitoring • Drift • Less external validity • High inference, low inference, descriptive

11. Classroom Observations • Center for the Improvement of Early Reading Achievement Observation System (Taylor et al., 2003). • focus on teacher pedagogy, • student learning processes, and the • implementation of evidence-based instructional practices

12. Classroom Observations • Five-minute observation cycles • Record uses narratives and specific codes: • who is teaching • grouping (e.g., whole class, small group, pairs, individual) • reading/language arts activities (e.g., reading, writing, etc.) • focus of the instruction (e.g., comprehension, phonics, etc.) • material used (e.g., text books, video, computers, board/chart) • teacher interaction (e.g., telling, modeling, discussion, coaching/scaffolding) • expected student response (e.g.., reading, reading turn taking)

13. Classroom Observations • Instructional Content Emphasis (Edmonds & Briggs, 2004) • Substantive and applied research and evaluation • Teach for Success Classroom Observation Protocol (WESTED) • More oriented towards formative • School Observation Measure (Center for Research in Educational Policy) • School-wide measure

14. Questions?