Graphing, Calculating, and Interpreting Rate of Improvement

1. Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3, 2010

2. Objectives • There needs to be a standardized procedure for calculating RoI • We’re proposing a method using Simple Linear Regression

3. Importance of RoI RoI Research A Need for Consistency Calculating RoI Individual Student Graphs Programming Excel Decision Making Grounding the Data Interpreting Growth Individual Student Student Groups Considerations Resources Overview

4. Importance of Graphs • Vogel, Dickson, & Lehman, 1990 • Speeches that included visuals, especially in color, improved: • Immediate recall by 8.5% • Delayed recall (3 days) by 10.1%

5. Importance of Graphs • “Seeing is believing.” • Useful for communicating large amounts of information quickly • “A picture is worth a thousand words.” • Transcends language barriers (Karwowski, 2006) • Responsibility for accurate graphical representations of data

6. Reading Oral Reading Fluency (ORF) Word Use Fluency (WUF) Reading Comprehension MAZE Retell Fluency Early Literacy Skills Initial Sound Fluency (ISF) Letter Naming Fluency (LNF) Letter Sound Fluency (LSF) Phoneme Segmentation Fluency (PSF) Nonsense Word Fluency (NWF) Spelling Written Expression Behavior Math Math Computation Math Facts Early Numeracy Oral Counting Missing Number Number Identification Quantity Discrimination Skills Typically Graphed

7. Importance of RoI • Multi-tiered model • Progress monitoring • Data for decision-making • Goal setting (Shapiro, 2008)

8. Importance of RoI • Visual inspection of slope • Multiple interpretations • Instructional services • Need for explicit guidelines

9. RoI Research • Deno, 1985 • Curriculum-based measurement • General outcome measures • Short • Standardized • Repeatable • Sensitive to change

10. RoI Research • Fuchs & Fuchs, 1998 • Hallmark components of Response to Intervention • Ongoing formative assessment • Identifying non-responding students • Treatment fidelity of instruction • Dual discrepancy model • One standard deviation from typically performing peers in level and rate

11. RoI Research • Ardoin & Christ, 2008 • Slope for benchmarks (3x per year) • More growth from fall to winter than winter to spring • Might be helpful to use RoI for fall to winter • And a separate RoI for winter to spring

12. RoI Research • Fuchs, Fuchs, Walz, & Germann, 1993 • Typical weekly growth rates • Needed growth • 1.5 to 2.0 times typical slope to close gap • Example • Bob is below benchmark on ORF • Typical slope is 1 wcpm per week growth • Bob would need slope of 1.5 to 2 to close gap in a reasonable amount of time

13. RoI Research • Deno, Fuchs, Marston, & Shin, 2001 • Slope of frequently non-responsive children approximated slope of children already identified as having a specific learning disability

14. RoI Research • Algebraic term: Slope of a line • Vertical change over the horizontal change • Rise over run • m = (y2 - y1) / (x2 - x1) • Describes the steepness of a line (Gall & Gall, 2007)

15. RoI Research • Finding a student’s RoI = finding the slope of a line • Using two data points on that line • Finding the line itself • Linear regression • Ordinary Least Squares

16. RoI Research • Gall & Gall, 2007 • 10 data points are a minimum requirement for a reliable trendline • How does that affect the frequency of administering progress monitoring probes?

17. RoI Research • Using RoI for instructional decisions is not a perfect process • Research is currently looking to address sources of error: • Christ, 2006 – standard error of measurement for slope • Ardoin & Christ, 2009 – passage difficulty and variability • Jenkin, Graff, & Miglioretti, 2009 – frequency of progress monitoring

18. RoI Research • Questions yet to be empirically answered • What parameters of RoI indicate a lack of RtI? • How does standard error of measurement play into using RoI for instructional decision making? • How does RoI vary between standard protocol interventions? • How does this apply to non-English speaking populations?

19. How is RoI Calculated? Which way is best?

20. Multiple Methods for Calculating Growth • “Eye ball” Approach • Last point minus First point Approach • Split Middle Approach • Linear Regression Approach

24. 1.1 Words Per Week

25. RoI Consistency?

26. RoI Consistency? • If we are not all using the same model to compute RoI, we continue to have the same problems as past models, where under one approach a student meets SLD criteria, but under a different approach, the student does not. • Hypothetically, if the RoI cut-off was 0.65 or 0.95, different approaches would come to different conclusions on the same student.

27. Technical Adequacy • Without a consensus on how to compute RoI, we risk falling short of having technical adequacy within our model.

28. So, Which RoI Method is Best?

29. Literature shows that Linear Regression is Best Practice • Student’s daily test scores…were entered into a computer program…The data analysis program generated slopes of improvement for each level using and Ordinary-Least Squares procedure (Hayes, 1973) and the line of best fit. • This procedure has been demonstrated to represent CBM achievement data validly within individual treatment phases (Marston, 1988; Shinn, Good, & Stein, in press; Stein, 1987). Shinn, Gleason, & Tindal, 1989

30. Growth (RoI) Research using Linear Regression • Christ, T. J. (2006). Short-term estimates of growth using curriculum based measurement of oral reading fluency: Estimating standard error of the slope to construct confidence intervals. School Psychology Review, 35, 128-133. • Deno, S. L., Fuchs, L. S., Marston, D., & Shin, J. (2001). Using curriculum based measurement to establish growth standards for students with learning disabilities. School Psychology Review, 30, 507-524. • Good, R. H. (1990). Forecasting accuracy of slope estimates for reading curriculum based measurement: Empirical evidence. Behavioral Assessment, 12, 179-193. • Fuchs, L. S., Fuchs, D., Hamlett, C. L., Walz, L. & Germann, G. (1993). Formative evaluation of academic progress: How much growth can we expect? School Psychology Review, 22, 27-48.

31. Growth (RoI) Researchusing Linear Regression • Jenkins, J. R., Graff, J. J., & Miglioretti, D.L. (2009). Estimating reading growth using intermittent CBM progress monitoring. Exceptional Children, 75, 151-163. • Shinn, M. R., Gleason, M. M., & Tindal, G. (1989). Varying the difficulty of testing materials: Implications for curriculum-based measurement. The Journal of Special Education, 23, 223-233. • Shinn, M. R., Good, R. H., & Stein, S. (1989). Summarizing trend in student achievement: A comparison of methods. School Psychology Review, 18, 356-370.

32. So, Why Are There So Many Other RoI Models? • Ease of application • How many of us want to calculate OLS Linear Regression formulas (or even remember how)?

33. An Easy and Applicable Solution

34. Get Out Your Laptops! Or Kindly Look Over Your Neighbor’s Shoulder! I love ROI

35. Open Microsoft Excel • Microsoft Office 2003 for PCs • Microsoft Office 2007 for PCs • Microsoft Office for Macs

36. Graphing RoI For Individual Students

37. Setting Up Your Spreadsheet • In cell B2, type School Week • In cell C2, type Benchmark • In cell D2, type WPM (or Student Scores)

38. Labeling School Weeks • In cell B3, type 1 • Continue entering numbers through 36 in column B • Week 36 will be in cell B38

39. Entering Benchmarks • In cell C3, type the fall benchmark 77 • In cell C20, type the winter benchmark 92 • In cell C38, type the spring benchmark 110

40. Entering Student Data Points • Student data points are entered between cells D3 and D38. • Type the student’s score next to the corresponding week that it was administered.

41. Entering Student Data Points • Week 1 – 41 • Week 8 – 62 • Week 9 – 63 • Week 10 – 75 • Week 11 – 64 • Week 12 – 80 • Week 13 – 83 • Week 14 - 83

42. Entering Student Data Points • Week 15 – 56 • Week 17 – 104 • Week 18 – 74 • Week 20 – 85 • Week 21 – 89 • Week 22 – 69 • Week 23 – 85

43. Entering Student Data Points • Week 24 – 96 • Week 25 – 90 • Week 26 – 84 • Week 27 – 106 • Week 28 – 94 • Week 32 – 100

44. *CAUTION* • If a student was not assessed during a certain week, leave that cell blank • Do not enter a score of Zero (0) it will be calculated into the trendline and interpreted as the student having read zero words correct per minute during that week.

45. Creating a Graph • Highlight the data in Columns C and D • Include cells C2 and D2 through cells C38 and D38 • Include any blank cells

46. Excel 2003/Macs Click Insert Click Chart Excel 2007 Click Insert Find the icon for Line Click the arrow below Line Creating a Graph

47. Excel 2003/Macs A Chart Wizard window will appear Excel 2007 6 graphics appear Creating a Graph

48. Excel 2003/Macs Choose Line Choose Line with markers Excel 2007 Choose Line with markers Creating a Graph

49. Excel 2003/Macs Data Range tab Columns Excel 2007 Your graph appears Creating a Graph

50. Excel 2003/Macs Chart Title School Week (X Axis) WPM (Y Axis) Excel 2007 Change your labels by clicking on the graph Creating a Graph