Rate of Improvement Calculation and Decision Making

1. Rate of Improvement Calculation and Decision Making Caitlin S. Flinn, EdS, NCSP Andrew E. McCrea, MS, NCSP

2. Why we’re here… • While there exists a wealth of convincing research supporting the implementation of a response-to-intervention (RtI) framework, there are many questions yet to be empirically answered. • Within multi-tiered model of assessment and instruction/intervention, how do we know whether a student is learning?

3. Measuring Learning • Class tests • Quizzes • Assignment/homework completion and accuracy • Ask students questions in class • Grades/report cards • State/local assessments • Universal screening, benchmark assessments • Progress monitoring

4. With Progress Monitoring Data… • How do we know if a student is learning? • Look at the data points • Where are they on the graph? • Are the data points getting closer to the goal or benchmark? • Is there a way to measure growth? • Make an aimline toward goal • Look to see where data points are compared to aimline • Calculate Rate of Improvement (RoI)

5. Today’s Objectives • Explain what RoI is, why it is important, and how to compute it. • Establish that Simple Linear Regression should be the standardized procedure for calculating RoI. • Discuss how to use RoI within a problem solving/school improvement model.

6. RoI Definition • Rate of Improvement can be described algebraically as the slope of a line • Slope is defined as: the vertical change over the horizontal change on a Cartesian plane. (x-axis and y-axis graph) • Also called: Rise over run • Formula: m = (y2 - y1) / (x2 - x1) • Describes the steepness of a line (Gall & Gall, 2007)

7. RoI Definition • Finding a student’s RoI is determining the student’s learning • Creating a line that fits the data points, a trendline • To find that line, we use: • Linear regression • Ordinary Least Squares

8. How does Rate of Improvement Fit into the Larger Context?

9. School Improvement/Comprehensive School Reform Response to Intervention Dual Discrepancy: Level & Growth Rate of Improvement

10. School Improvement/ Comprehensive School Reform • Grade level content expectations (ELA, math, science, social studies, etc.). • Work toward these expectations through classroom instruction. • Understand impact of instruction through assessment.

11. Assessment • Formative Assessments/High Stakes Tests • Does student have command of content expectation (standard)? • Universal Screening using CBM • Does student have basic skills appropriate for age/grade?

12. Assessment • Q: For students who are not proficient on grade level content standards, do they have the basic reading/writing/math skills necessary? • A: Look at Universal Screening; if above criteria, intervention geared toward content standard, if below criteria, intervention geared toward basic skill.

13. Progress Monitoring • Frequent measurement of knowledge to inform our understanding of the impact of instruction/intervention. • Measures of basic skills (CBM) have demonstrated reliability & validity (see table at www.rti4success.org).

14. Classroom Instruction (Content Expectations) Measure Impact (Test) Proficient! Non Proficient Use Diagnostic Test to Differentiate Content Need? Basic Skill Need? Intervention Progress Monitor Intervention Progress Monitor With CBM If CBM is Appropriate Measure Rate of Improvement

15. So… • Rate of Improvement (RoI) is how we understand student growth (learning). • RoI is reliable and valid (psychometrically speaking) for use with CBM data. • RoI is best used when we have CBM data, most often when dealing with basic skills in reading/writing/math. • RoI can be applied to other data (like behavior) with confidence too! • RoI is not yet tested on typical Tier I formative classroom data.

16. RoI is usually applied to… • Tier One students in the early grades at risk for academic failure (low green kids). • Tier Two & Three Intervention Groups. • Special Education Students (and IEP goals) • Students with Behavior Plans

17. RoI Foundations • Deno, 1985 • Curriculum-based measurement • General outcome measures • Technically adequate • Short • Standardized • Repeatable • Sensitive to change

18. RoI Foundations • Fuchs & Fuchs, 1998 • Hallmark components of Response to Intervention • Ongoing formative assessment • Identifying non-responding students • Treatment fidelity of instruction • Dual discrepancy model • One standard deviation from typically performing peers in level and rate

19. RoI Foundations • Ardoin & Christ, 2008 • Slope for benchmarks (3x per year) • More growth from fall to winter than winter to spring • Might be helpful to use RoI for fall to winter • And a separate RoI for winter to spring

20. RoI Foundations • Fuchs, Fuchs, Walz, & Germann, 1993 • Typical weekly growth rates in oral reading fluency and digits correct • Needed growth to remediate skills • Students who had 1.5 to 2.0 times the slope of typically performing peers were able to close the achievement gap in a reasonable amount of time

21. RoI Foundations • Deno, Fuchs, Marston, & Shin, 2001 • Slope of frequently non-responsive children approximated slope of children already identified as having a specific learning disability

22. How many data points? • 10 data points are a minimum requirement for a reliable trendline (Gall & Gall, 2007) • Is that reasonable and realistic? • How does that affect the frequency of administering progress monitoring probes? • How does that affect our ability to make instructional decisions for students?

23. How can we show RoI? • Speeches that included visuals, especially in color, improved recall of information (Vogel, Dickson, & Lehman, 1990) • “Seeing is believing.” • Useful for communicating large amounts of information quickly • “A picture is worth a thousand words.” • Transcends language barriers (Karwowski, 2006) • Responsibility for accurate graphical representations of data

24. Reading Oral Reading Fluency Word Use Fluency Reading Comprehension MAZE Retell Early Literacy Skills Initial Sound Letter Naming Letter Sound Phoneme Segmentation Nonsense Word Spelling Written Expression Behavior Math Math Computation Math Facts Early Numeracy Oral Counting Missing Number Number Identification Quantity Discrimination Skills for Which We Compute RoI

25. Guidelines? • Visual inspection of slope • Multiple interpretations • Instructional services • Need for explicit guidelines

26. Ongoing Research • RoI for instructional decisions is not a perfect process • Research is currently addressing sources of error: • Christ, 2006: standard error of measurement for slope • Ardoin & Christ, 2009: passage difficulty and variability • Jenkin, Graff, & Miglioretti, 2009: frequency of progress monitoring

27. Future Considerations • Questions yet to be empirically answered • What parameters of RoI indicate a lack of RtI? • How does standard error of measurement play into using RoI for instructional decision making? • How does RoI vary between standard protocol interventions? • How does this apply to non-English speaking populations?

28. How is RoI Calculated? Which way is best?

29. Multiple Methods for Calculating Growth • Visual Inspection Approaches • “Eye Ball” Approach • Split Middle Approach • Tukey Method • Quantitative Approaches • Last point minus First point Approach • Split Middle & Tukey “plus” • Linear Regression Approach

30. The Visual Inspection Approaches

31. Eye Ball Approach

32. Split Middle Approach • Drawing “through the two points obtained from the median data values and the median days when the data are divided into two sections” (Shinn, Good, & Stein, 1989).

33. Split Middle X(14) X (9) X(9)

34. Tukey Method • Divide scores into 3 equal groups • Divide groups with vertical lines • In 1st and 3rd groups, find median data point and median week and mark with an “X” • Draw line between two “Xs” (Fuchs, et. al., 2005. Summer Institute Student progress monitoring for math. http://www.studentprogress.org/library/training.asp)

35. Tukey Method X(14) X(8)

36. The Quantitative Approaches

37. Last minus First • Iris Center: last probe score minus first probe score over last administration period minus first administration period. Y2-Y1/X2-X1= RoI http://iris.peabody.vanderbilt.edu/resources.html

38. Last minus First

39. Split Middle “Plus” X(14) X(9) (14-9)/8=0.63

40. Tukey Method “Plus” X(14) X(8) (14-8)/8=0.75

41. Linear Regression

42. RoI Consistency?

43. RoI Consistency? • If we are not all using the same model to compute RoI, we continue to have the same problems as past models, where under one approach a student meets SLD criteria, but under a different approach, the student does not. • Hypothetically, if the RoI cut-off was 0.65 or 0.95, different approaches would come to different conclusions on the same student.

44. RoI Consistency? • Last minus First (Iris Center) and Linear Regression (Shinn, etc.) only quantitative methods discussed in CBM literature. • Study of 37 at risk 2nd graders:

45. Technical Adequacy • Without a consensus on how to compute RoI, we risk falling short of having technical adequacy within our model.

46. So, Which RoI Method is Best?

47. Literature shows that Linear Regression is Best Practice • Student’s daily test scores…were entered into a computer program…The data analysis program generated slopes of improvement for each level using an Ordinary-Least Squares procedure (Hayes, 1973) and the line of best fit. • This procedure has been demonstrated to represent CBM achievement data validly within individual treatment phases (Marston, 1988; Shinn, Good, & Stein, in press; Stein, 1987). Shinn, Gleason, & Tindal, 1989

48. Growth (RoI) Research using Linear Regression • Christ, T. J. (2006). Short-term estimates of growth using curriculum based measurement of oral reading fluency: Estimating standard error of the slope to construct confidence intervals. School Psychology Review, 35, 128-133. • Deno, S. L., Fuchs, L. S., Marston, D., & Shin, J. (2001). Using curriculum based measurement to establish growth standards for students with learning disabilities. School Psychology Review, 30, 507-524. • Good, R. H. (1990). Forecasting accuracy of slope estimates for reading curriculum based measurement: Empirical evidence. Behavioral Assessment, 12, 179-193. • Fuchs, L. S., Fuchs, D., Hamlett, C. L., Walz, L. & Germann, G. (1993). Formative evaluation of academic progress: How much growth can we expect? School Psychology Review, 22, 27-48.

49. Growth (RoI) Researchusing Linear Regression • Jenkins, J. R., Graff, J. J., & Miglioretti, D.L. (2009). Estimating reading growth using intermittent CBM progress monitoring. Exceptional Children, 75, 151-163. • Shinn, M. R., Gleason, M. M., & Tindal, G. (1989). Varying the difficulty of testing materials: Implications for curriculum-based measurement. The Journal of Special Education, 23, 223-233. • Shinn, M. R., Good, R. H., & Stein, S. (1989). Summarizing trend in student achievement: A comparison of methods. School Psychology Review, 18, 356-370.

50. So, Why Are There So Many Other RoI Models? • Ease of application • Focus on Yes/No to goal acquisition, not degree of growth • How many of us want to calculate OLS Linear Regression formulas (or even remember how)?