Growth in Reading

1. Growth in Reading Curriculum – Based Measures and Predicting High Stakes Outcomes

2. Questions • Are there differences in the growth in Reading (CBM) between students who meet and do not meet standards on high stakes tests? • If differences exist can they be predicted from the beginning of the year? • What is the cost of setting hard targets for performance on CBM?

3. R-CBM for G3 students taking ISAT Mean 135 (35.6) Median = 129 Median = 115 Median = 97, Nm = 388 Mean =91 (27.9) Median = 94 Median = 61 Nnm = 137 Median = 85 Gain ~ .97 WPW Gain ~ 1 WPW

4. Distribution of R-CBM Scores by Time (for Gr. 3students who took ISAT) Typical Range Fall 78 – 113; Md 97 Winter 99 – 140; Md 115 Spring 108 – 156; Md 129

5. R-CBM for G3 students taking IMAGE Mean 115 (27.8) Median = 113 Median = 100 Median = 82, Nm = 56 Mean = 78.5 (21.0) Median = 80 Median = 69 Median = 56Nnm = 65 Gain ~ .94 WPW Gain ~ .73 WPW

6. Distribution of R-CBM Scores by Time (for Gr. 3 students who took IMAGE) Typical Range Fall 70 – 101; Md 82 Winter 82 – 119; Md 100 Spring 94 – 131; Md 113 Note. N’s less than 100 no whiskers are displayed

7. What does this relationship look like for Grade 5

8. Distribution of R-CBM Scores by Time (for Gr. 5 students who took ISAT) Typical Range Fall 126 – 168; Md 145 Winter 144 – 183; Md 164 Spring 158 – 196; Md 176

9. R-CBM for G5 students taking ISAT Mean 178 (29.3) Median = 176 Median = 164 Median = 145 Nm = 375 Mean = 139 (28.5) Median = 142 Median = 130 Median = 110Nnm = 126 Gain ~ .94 WPW Gain ~ .98 WPW

10. R-CBM for G5 students taking IMAGE Mean 146 (24) Median = 140 Median = 139 Median = 115 Nm = 30 Mean = 114 (34) Median = 118 Median = 112 Median = 92Nnm = 47 Gain ~ .76 WPW Gain ~ .79 WPW

11. Distribution of R-CBM Scores by Time (for Gr. 5 students who took IMAGE) Typical Range Fall 96 – 132; Md 115 Winter 112 – 154; Md 139 Spring 128 – 168; Md 140 Note. N’s less than 100 no whiskers are displayed

12. Can we use this information for … Educational Decision - Making?

13. ISAT Gr. 3 r = .70 R2 ~ 48%

14. Medical decision - making to Educational decision - making From In medicine indices of diagnostic accuracy help doctors decide who is high-risk and who is not likely at risk for developing a disease Can we borrow this technology for  tracking adequate growth and  educational decision-making

15. Diagnostic Indices • Sensitivity • the fraction of those who fail to meet standards who were predicted to fail to meet standards • Specificity • the fraction of students who meet standards who were predicted to meet • Positive Predictive Power • the fraction of students who were predicted not to meet who failed to meet standards • Negative Predictive Power • the fraction of students who were predicted to meet who met standards • Correct Classification • the fraction of students for whom predictions of meeting or not meeting were correct

16. 98% 88% 84% 80% 67% 54% 39% 25% ISAT Gr. 3 Sensitivity considers only students whodid not meet standards. As WRC increases sensitivity increases

17. 98% 74% 86% 30% 62% ~100% 68% 93% ISAT Gr. 3 Specificity considers only students whomeet standards. As WRC increases specificity decreases

18. 98% 88% 78% 67% 57% 53% 50% 38% Positive PredictivePower considers the fraction of students who scored Less than a particular cut who did not meet standards. As WRC increases PPV decreases

19. 75% 78% 82% 86% 90% 91% 92% 98% Negative PredictivePower considers the fraction of students who scored more than a particular cut who met standards. As WRC increases PPV decreases

20. Decisions, decisions How should we determine where to draw the line?

21. At 60 we classify 77% of students correctly

22. At 80 we classify 80% of students correctly

23. At 90 we classify 81% of students correctly

24. At 100 we classify 80% of students correctly

25. At 110 we classify 76% of students correctly

26. At 115 we classify 73% of students correctly

27. At 120 we classify 70% of students correctly

28. At 150 we classify 51% of students correctly

29. Why do we have to draw just one line? Maximize Correct Classification Admit the limitations of the tool

30. Two statistical methods for group determination • Logistic Regression • Maximum likelihood method for predicting the odds of group membership • Appears to maximize specificity in cut-score selection • Linear Discriminant Function Analysis • Least Squares method for predicting the linear relation between variables that best discriminates between groups • Appears to maximize sensitivity in cut score selection

31. Cut scores set by LR and LDFA Both LR and LDFA can use multiple predictors to determine group membership, but for this analysis, only R-CBM in the spring was used. • Logistic regression ~ 92 WRC • LDFA ~ 112 WRC

32. U N C L A S S I F I E D NPV = 83% Spec = 93%, The LR & LDFA (Cut L92 - H112) classifies 78% of students in the data set. Of these students who were classified, 86% were classified correctly, with a rate of  14% error in classification. Note that the 86% classified correctly in this model is 78% of the total group.  The reduction of error in identification comes at a cost of failing to identify risk status for 127 of 565 students. PPV = 77% SENS = 81%

33. What to do with the “Unclassified” student? • R-CBM does not attempt to tell us everything about a student’s reading, it is a strong indicator. • Use of convergent data may be able to provide us with a more fine-grained prediction

34. At 80 NPV = 83% At 59 Spec = 94%, U N C L A S S I F I E D • The fall r-cbm (LR & LDFA) (Cut L59 - H80) classifies 80% of students in the data set. Of these students who were classified,  87% were classified correctly, with a rate of 13% error in classification. • The 87% classified correctly in this model is 80% of the total group. • The reduction of error in identification comes at a cost of failing to identify risk status for 114 of 565 students. At 59 PPV = 81% At 80 SENS = 80%

35. At 101 NPV = 89% At 79 Spec = 93%, U N C L A S S I F I E D • The winter r-cbm (LR & LDFA) (Cut L79 - H101) classifies 75% of students in the data set. Of these students who were classified,  86% were classified correctly, with a rate of 14% error in classification. • Note that the 86% classified correctly in this model is 75% of the total group. • The reduction of error in identification comes at a cost of failing to identify risk status for 143 of 565 students. At 79 PPV = 77% At 101 SENS = 80%

37. The FALL G 5 R-CBM (LR & LDFA) (Cut L95 - H131) serves to classify 71% of students in the data set. Of these students who were classified, • 92% were classified correctly, with a rate of 8% error in classification. • The 92% classified correctly in this model is 71% of the total group. • The reduction of error in identification comes at a cost of failing to identify risk status for (166 of 565 students).

38. The WINTER G 5 R-CBM (LR & LDFA) (Cut L122 - H144) serves to classify 79% of students in the data set. Of these students who were classified, 87% were classified correctly, with a rate of 13% error in classification. Note that the 87% classified correctly in this model is 79% of the total group. The reduction of error in identification comes at a cost of failing to identify risk status for (118 of 565 students).

39. The Spring G 5 R-CBM (LR & LDFA) (Cut L137 - H157) serves to classify 80% of students in the data set. Of these students who were classified, •  86% were classified correctly, with a rate of 14% error in classification. • The 86% classified correctly in this model is 80% of the total group. • The reduction of error in identification comes at a cost of failing to identify risk status for 111 of 565 students. r = .62, R2 = .38

40. Examination of aggregated trajectories Adult Readers Annual Targets

41. The range between the 90th and 10th percentile for each sub group is an empirical, non-parametric 80% confidence interval (for the individual of) quartile growth ˘ cross-sectional between grades ˘ longitudinal within grade 170 WRC By Spring of Grade 8 160 WRC By Spring of Grade 5 115 WRC By Spring of Grade 3

42. The Spring VM (LR & LDFA) (Cut L5 - H9) serves to classify 79% of students in the data set. Of these students who were classified, 89% were classified correctly, with a rate of 11% error in classification. Note that the 89% classified correctly in this model is 79% of the total group. The reduction of error in identification comes at a cost of failing to identify risk status for (117 of 565 students).

43. Big Ideas • There are reliable quantitative differences in the average performance of Meeters versus Non – Meeters • These differences can be predicted at least from Fall if not earlier • There are limitations to the “hard and fast” cut score approach with R-CBM that can be partially addressed by admitting some students may require additional assessment to make a determination of status.

44. Directions • Set up a model for English and Spanish Reading for Students who are ELLs. • Expand progress monitoring with Vocabulary Matching

45. Questions? ben@measuredeffects.com