Agenda

1. Florida Value-Added Models:2012 Results and Summary of Optional and EOC VAMsStudent Growth Implementation Committee (SGIC) December 4, 2012

2. Agenda

3. Structured Review Process • Are the input data accurate and sensible? • Examine the descriptive statistics • Are there any red flags? • Do the models behave as expected? • Examine the variance components • Examine R-squared to determine model fit • Precision of the VAM scores • Do the results suggest advantages to certain groups? • Impact data based on correlations on VAM scores and class characteristics

4. Thoughts on Covariates • Ideally, the predictor variables should have the followingproperties: • A high statistical correlation with the outcome • A high curricular relationship with the outcome • Correlated with factors that contribute to student learning but are not in the control of teachers and schools • A high correlation with the unobservable processes by which students are sorted into schools and classes • If predictors do not fully capture selection effects, teacher and school value-added estimates may be biased.

5. Covariates Included in Most Models: • Prior test scores • Students with Disabilities (SWD) status • Gifted status • English Language Learner (ELL) status (time as ELL) • Attendance • Mobility (number of transitions) • Difference from modal age in grade • Class size • Homogeneity of entering test scores in the class. • Percentage in each grade, when appropriate • Percent gifted in class • Number of subject-relevant courses

6. Optional Value-Added Model:SAT-10

7. SAT-10 SAT-10 Background Information • SAT-10 scores are used to create VAM scores for Grade 2 teachers • Grade 1 scores are used as predictors for the Grade 2 outcome variable • SEMs were not provided, as a result, measurement error is not accounted for • If SEMs are available, they should be used to account for measurement error • The VAM implemented for SAT-10 is the same statistical model used for the FCAT VAMs

8. SAT-10 2010-11 SAT-10 Scores: All Students and by Subgroup

9. SAT-10 Prior-Year SAT-10 Scores: All Students and by Subgroup

10. SAT-10 Summary of Descriptive Statistics • The differences between groups are typical for in level score analyses • All discrepancies appear normal • Correlation between current and prior score (0.77) is typical

11. SAT-10 Variance of Teachers and Schools • The next slide shows the teacher and school variance • The teacher component is typically expected to have more variability than the school component

12. SAT-10 Teacher Standard Deviation Is Larger than the School Standard Deviation

13. SAT-10 Model Fits Data Well, but Does Not Distinguish Exceptional Teachers • R-squared is one indicator of model fit. • The closer the value is to 1, the better the model predicts outcome scores • For the SAT-10, the R-squared is 0.62 • This is on par with the FCAT R-square

14. SAT-10 Impact Data Results • Impact data slides show the relationship of the teacher VAM score with various classroom characteristics • There are two ways to interpret a non-zero relationship: • Teachers are not distributed randomly across students • Classroom characteristics affect the rate of student learning and lead to biased VAM estimates

15. SAT-10 Correlation Between Teacher Score and Share of Students with Disabilities

16. SAT-10 Correlation Between Teacher Score andShare of English Language Learners

17. SAT-10 Correlation Between Teacher Score and Share of Who Are Gifted

18. SAT-10 Correlation Between Teacher Score and Share Who Are Economically Disadvantaged

19. SAT-10 Correlation Between Teacher Score and Share Who Are Non-White

20. SAT-10 Correlation Between Teacher Score and Mean Prior SAT-10 Score

21. SAT-10 Observed Correlations with Teacher Score

22. SAT-10 Impact Data Summary • The impact data correlations are larger when the teacher score includes some of the school component • In this instance, it suggests the school component adds back in some of the systematic differences between schools that a VAM is trying to account for

23. End of Course Value-Added Model:Algebra I

24. Algebra I Algebra I Background Information • 2010–11 was the first administration of the Algebra I End of Course Exam • Previously, students took the comprehensive Mathematics FCAT grades 9 and 10 (which had been administered for over a decade).

25. Algebra I Algebra I Background Information • All students who were enrolled in an Algebra I course in 2010–11 were required to take the exam • Per state law, the Algebra I EOC is required to count as 30% of the final grade for students entering 9th grade in 2010-11 • At the state level, stakes were not attached to any other test takers. Stakes may have been attached to other test takers at the local level. • Student effort may differ by grade level, thereby affecting VAM scores

26. Algebra I Number of Students by Model

27. Algebra I Prior Test Scores Depend on the Student’s Current Grade Prior scores available for each grade

28. Algebra I Descriptive Statistics • The following descriptive statistics are presented to show that the observed patterns in the level scores are also observed in the VAM scores.

29. Algebra I Comparison of Current Year Algebra I EOC Scores by Grade

30. Algebra I Comparison of Prior Year FCAT Scores by Grade

31. Algebra I Algebra I EOC Scores by Gifted Status

32. Algebra I Algebra I EOC Scores by SWD Status

33. Algebra I Correlation Between Current Year Algebra I EOC Scores and Prior Year FCAT Scores

34. Algebra I Summary of Descriptive Statistics • The data show that students in lower grades score higher on the Algebra I EOC and FCAT relative to students in the higher grades • There are large systematic differences between student groups • The correlations for Algebra I EOC are somewhat lower than those observed for the FCAT VAM

35. Algebra I Standard Deviations of Teachers and Schools • The next slide shows the teacher and school standard deviations • The teacher component is typically expected to have more variability than the school component • This is observed in the FCAT model

36. Algebra I Magnitude of Teacher and School Component Standard Deviations Other than in grade 9, the standard deviation between teachers is smaller than the standard deviation between schools.

37. Algebra I Thoughts on Variance Components • Again, the pattern observed in the variance components is the same pattern observed with the level scores

38. Algebra I R-Square • The R-square is one indicator of model fit • With the FCAT VAM, the R-square is 0.60 to 0.64

39. Proportion of Variance in Current Year Test Score Explained by Control Variables The R-square is relatively low for the Grade 10 and above model. Grades 8 and below and Grade 9 have R-square values much lower than those observed for the FCAT VAM.

40. Algebra I Teacher VAM Score by Grade Teachers with students in the lower grades tend to have higher VAM scores relative to teachers in the upper grades (using data from the model where all students are included).

41. Algebra I Impact Data Results • Impact data slides show the relationship of the teacher score with various classroom characteristics • There are two ways to interpret a non-zero relationship: • Teachers are not distributed randomly across students • Classroom characteristics affect the rate of student learning and lead to biased VAM estimates

42. Algebra I Correlation of Teacher VAM Score and Percent Students with Disabilities Correlation is virtually a flat slope, indicating no relationship with SWD variable for All Students.

43. Algebra I Correlation of Teacher VAM Score and Percent English Language Learners There is a small but noticeable negative correlation with ELL. The grade 8 and below correlation is –0.08.

44. Algebra I Correlation of Teacher VAM Score and Percent Economically Disadvantaged There is a clear negative correlation with percent FRL in class. For grade 8 and below, the correlation is -0.34.

45. Algebra I Correlation of Teacher VAM Score and Percent Gifted • The relationship is smaller than what it was previously when there was no control for percent gifted. • However, it is still positive and noticeable.

46. Algebra I Correlation of Teacher VAM Score and Percent Non-White Students There is virtually no relationship with the percent of non-white students.

47. Algebra I Correlation of Teacher VAM Score and Mean Prior Achievement Correlation with mean prior achievement is positive. For grade 8 and below, the correlation is 0.39.

48. Algebra I Observed Correlations with Teacher VAM Score

49. Algebra I Practical Effect of Correlation • Teachers of students with higher entering FCAT scores tend to also have higher value-added components • In practical terms, a teacher with a class at the 25th percentile (prior FCAT) would have a predicted teacher component of –0.45 and a teacher at the 50th percentile would have a predicted teacher component of 0.15 • This difference of 0.6 is about 0.13 of the standard deviation between teachers

50. Algebra I Summary of Impact Data • The correlation between the teacher VAM score and the percent gifted has decreased, as would be expected when using this correlation as a control • The negative correlation with economically disadvantaged remains, and is large • The variance components still reflect a sorting issue not captured by use of FCAT prior score • The R-square shows the prior scores and other covariates do not predict EOC scores as well as FCAT