Construct Validity of Classroom Observations: Items, Factors, Raters, and Achievement

1. Construct Validity of Classroom Observations: Items, Factors, Raters, and Achievement Lee Branum-Martin, Coleen D. Carlson, Angelia Durand, Christopher Barr Texas Institute for Measurement, Evaluation, and Statistics University of Houston www.times.uh.edu Society for Research on Educational Effectiveness March 4, 2010

2. A Generalizability Theory Approach (following Raudenbush) Classroom Quality = items + construct + rater + time + school + grade . . . Generalizability Raudenbush (2008). Statistical inference when classroom causality is measured with error. Context measurement Raudenbush & Sampson (1999). Ecometrics. Raudenbush, Rowan, & Kang (1991). A multilevel, multivariate model for school climate with estimation via the EM algorithm and application to US high school data.

3. An Ecometric Approach (in response to Raudenbush) Item score = construct + rater + time + school + grade . . . Development Conversation Oral Language Uses Vocabulary Furnishings Arrangement Organization Engagement

4. Rater Differences Item score = construct + rater + time + school + grade . . . Ratings are valid, but differ in severity (factor means) Rater Development λ11 λ12 Conversation Oral Language λ13 Uses λ14 Ratings are valid, but differ in factor variances Vocabulary Ψ11 Ψ21 Furnishings λ25 λ26 Raters differ in validity Arrangement Organization λ27 Engagement Ψ22

5. TimeDifferences (School year, semester, month, class session, segment of session) Item score = construct + rater + time + school + grade . . . Classroom factors differ over time Time Development λ11 λ12 Conversation Oral Language λ13 Uses The variances & relations among classroom factors differ over time λ14 Vocabulary Ψ11 Ψ21 Furnishings λ25 Factor composition differs over time: the nature of classroom ecology changes λ26 Arrangement Organization λ27 Engagement Ψ22

6. Sample & Design

7. Example Items for Classroom Management adapted from the ELLCO (Smith et al., 2002)

8. Fit Statistics: Grades, Years, Semesters

9. Model Results: Factor Loadings & Thresholds Factor z-score Item Factor .62 Organization .84 .90 Reading Instruction .70 .85 .70 Writing Instruction .77 .77 .86 Threshold between high and medium quality Threshold between medium and low quality Factor loading

10. Model Results: Factor Loadings & Thresholds Factor z-score Factor Item Assessment .85 .90 Climate .94 .94 Curriculum 1.00 Management .77 .92 — Oral Language .48 .84 .88 .77 medium/high low/medium

11. Factor Correlation Matrix

12. Interobserver Agreement on Factor Scores Interobserver correlation Mean differences Cross-classified

13. Correlations to campus mean achievement * * * * * * * * * * * * * * * * * * * * * * p < .05

14. Conclusions • Confirmatory factor models can be applied to observational data to examine relations among items, constructs, raters, time, and sites • CFA on categorical items reveals functional information about the items. • CFA incorporates theory and design in a falsifiable way. • CFA is complex, but can serve as a first-stage validity analysis to be exported into other analyses, such as G-theory or multilevel models of outcomes.

15. Questions, Comments? Lee.Branum-Martin@times.uh.edu References Raudenbush, S. W. (2008). Statistical inference when classroom causality is measured with error. Paper presented at the SREE. Raudenbush, S. W., Rowan, B., & Kang, S. J. (1991). A multilevel, multivariate model for school climate with estimation via the EM algorithm and application to US high school data. J Ed Stat, 16, 295-330. Raudenbush, S. W., & Sampson, R. J. (1999). Ecometrics: Toward a science of assessing ecological settings, with application to the systematic social observation of neighborhoods. Soc Methodology, 29(1), 1-41. Smith, M. W., Dickinson, D. K., Sangeorge, A., & Anastasopoulos, L. (2002). Early Language & Literacy Classroom Observation (ELLCO) Toolkit, Research Edition. Baltimore, MD: Paul H. Brookes.