We’re “Doing RTI” A Closer Look at Implementation

1. We’re “Doing RTI”A Closer Look at Implementation Rebecca Zumeta Technical Assistance Coordinator National Center on RTI • Mike Jacobsen • Director of Assessment • White River School District

2. Overview • Rationale & Purpose • Prior work • Description of measurement tool • Description of interview process & reports • Interrater agreement • Summary of findings to date • Illustrative Example: White River’s Implementation Journey • Next steps

3. Rationale • 171 of 295 Washington school districts reported implementing RTI in 2010. • Unknown implementation quality • Unclear implementation metric in prior evaluation of RTI pilot sites

4. Purpose • Desire to identify areas of relative implementation strength/weakness to: • Help schools/districts plan future professional development and school improvement efforts • Help OSPI staff determine areas requiring broader state-level support • Begin to determine extent to which implementation may affect changes in: • Student achievement • Number/accuracy/type of special education referrals/eligibility

5. Prior Approaches • General staff interviews • Few scripted questions • Variable turnaround of reports (typically several months) • General recommendations/feedback provided • Surveys that went to all staff

6. Measuring Implementation • National Center on RTI (NCRTI) Implementation Integrity Rubric (Pilot) • Essential Components • Overarching Factors • Ratings of 1-5 across factors • Structured interview based on rubric language • 28 items • Precise questioning

8. Interview Form (p. 3 of 13)

9. Pilot Sample • 11 Elementary Schools (pilot districts) • 11 Middle Schools (pilot districts) • Building RTI Teams • Administrator • RTI Coach/District Coordinator • Specialist(s) • Classroom teacher(s) • Reading & Math (when relevant) • Teams asked to bring & discuss evidence

10. Procedure • 4 external evaluators & OSPI staff person • 2-3 hour interview with building RTI team • Structured interview with 28 items covering: • Essential Components • Fidelity (assessment & instruction) • Instruction across levels (Primary, Secondary, Intensive/Tertiary) • Screening & Progress Monitoring • Data-based decision-making • Overarching Factors (leadership, parents, staff perceptions, training etc.)

11. Scoring • Scored against specific language/definitions listed in the rubric • When in doubt, rounded scores down • Used scores of “2” and “4” when implementation did not exactly match definitions of a “1,” “3,” or “5.” • Scores averaged across implementation area • Components not in place scored a “1.” • Considered evidence when determining scores

12. Interview Form (p. 13 of 13)

13. Reports to Schools • Targeted feedback & scores • Notes across questions • Specific recommendations • Identification of relative strengths & challenges • Faster turnaround (average 1-2 weeks) • Also revised the survey process to better target appropriate staff

14. Interrater Agreement • 2 Interviewers • Both took detailed notes • Discussed items following interview • Scored independently following discussed items • Required agreement within 1 • If discrepancy >1, discussed and agreed on score

15. Average Agreement • Within 1 Point • Reading (n = 20*): 99.8% • Math (n = 17): 99.6% • 100% Agreement • Reading (n = 20*): 88.3% • Math (n = 17): 87.2% *Scores in one district were not assessed for reliability because we were piloting the interview and scoring process.

16. Elementary Mean(SD)

17. Middle School Mean(SD)

18. Other Factors

19. District Team 2006-2007 Membership: School Psychologists, Elementary Principal Rep, Director Student Support Services, Title One/LAP Reading Specialists, Special Education & General Education PSESD RTI Grant RTI Readiness Surveys Interviews w/Building Teams Site Visit-Tigard-Tualatin-April 07 Team: Elem. Reading Specialist, School Psych. HS Assist. Principal, MS Principal, Elem. Counselor, & Director SSS. General overview-Visitation to an elementary, middle and high school RTI in White River: The Implementation Journey

20. Implementation Journey • TTSD Two Day Training-April 30/May 1, 2007 • Staff included District RTI Team, additional members included special education and other building principals • Agenda • RTI Overview • Three-Tiered Model • Universal Screening • Data system • Team Process • TTSD Data

21. Our data indicates that we have significant numbers of students not being successful in the core curriculum We have significant equity issues as assessment and interventions vary building to building and grade to grade Research and best practice support and integrated approach to change Rationale for change?

22. We currently have a model that waits for the student to come to us rather than identifying early those students who are at risk We have limited resources and need to focus intensive support on those most in need WAC Revisions will require many RTI components for special education eligibility as learning disabled Rationale for change?

23. PSESD Cohort Training Foothills & Elk Ridge Elementary RTI Teams Four days of training with Wayne Callender During the 2007-2008 School Year The Journey

24. A systemic intervention process for every student who struggles A seamless, tiered intervention system Better results, behaviorally and academically for students who struggle Interventions are evaluated for effectiveness Effective core programs implemented with fidelity Use of assessment data to make instructional decisions The Vision

25. WRSD: Major emphasis upon development of a Professional Learning Communities model Administrative staff attend Dufour, PLC Summit-Everett-August 2006 Leadership of Janel Keating-Deputy Superintendent Implementation of PLC Teams in each building The Journey

26. RTI is fundamentally linked to the third question in a PLC model: -How will we respond when some students don’t learn? RTI is a process that systemically and empirically addresses the extra supports needed for students who are not learning. PLC and RTI are cultural shifts in how districts educate their students Integration with PLC Process

27. Phase One: Submitted April 2008 Phase Two: Submitted June 2008 Three year plan-Extended to five $225,000 Decrease grant support each year Six district awards Application addressed PD, Sustainability, RTI Coach, District support The Journey

28. The Grant-Initial Implementation

29. The Grant-Initial Implementation

30. The Grant-2009-2010

31. White River-Universal-Benchmark Screening M-COMP PSF M-CAP M-CAP M-CAP M-CAP M-CAP M-CAP M-CAP LSF R-CBM M-CAP MAZE MAZE MAZE MAZE MAZE MAZE MAZE R-CBM R-CBM NWF R-CBM LNF R-CBM R-CBM R-CBM R-CBM R-CBM PSF: Phonemic Segmentation Fluency, LSF: Letter Sound Fluency, LNF: Letter Naming Fluency, M-COMP: Math Computation, R-CBM: Reading Curriculum Based Measurement (ORF), NWF: Nonsense Word Fluency, M-CAP: Math Computation & Application & MAZE: Reading Comprehension. (AIMSWeb)

32. Standardized Reading & Math Pathways have been developed Evidence-based interventions @ Tiers 2 & 3 for K-12 RTI Data team protocols AIMSWeb has added additional content areas in math, K-5, and comprehension 4 & 5. RTI Updates 2010-2011

33. AIMSWeb has added additional content areas in comprehension 6-8. Fidelity checklists have been developed and initial implementation is moving forward Use of Kid Graphs has increased Data is routinely being used to change instruction RTI Updates 2010-2011

34. We are beginning to build “in program “ assessments from our interventions into SOAR AIMSWeb is increasingly being used to display data The district has had visitation teams from Edmonds, Auburn, Yelm, Hood River, Carnation, Richland, Puyallup, and North Thurston School Districts Activities since February, 2010

35. WRSD RTI Math Pathway

36. WRSD RTI Reading Pathway

37. Foothills Elementary Mean

38. Glacier Middle School Mean

39. Overall Summary Overall Strengths noted in: • Elementary Reading • Screening • Fidelity of Assessment • Progress Monitoring in Elementary Reading • Several individual buildings with very comprehensive systems in place

40. Overall Summary Overall Challenges noted for: • Fidelity of Instruction • Data-based decision-making (i.e., moving students and/or changing instruction in a timely manner) • Consistent implementation of tertiary interventions • Parent Involvement/Notification • Progress Monitoring in Mathematics Vague information for: • Cultural & Linguistic Responsiveness • Evidence of differentiation

41. Feedback to Date • Coordinators & teams noted finding feedback: • More reflective of their current status • Timely & helpful for planning • Targeted specific concerns • Facilitated useful conversations with RTI teams as they prepared for interviews • Some need to clarify purpose of interview process

42. Next Steps • Incorporate findings into annual evaluation report • Regression analysis to determine: • Impact on overall achievement • Impact on special education referral trends • Revisions to clarify language in some interview questions • Expand data-based decision-making section • Interview additional schools • Plan additional PD opportunities reflecting implementation challenges observed • Investigate options for evaluating behavior • Implementation monitoring i

43. Implementation Monitoring & Training • 5 trained evaluators • Available to provide support to schools/districts upon request • Hope to train an additional 5-10 people to conduct interviews with other interested districts during the coming year

44. Questions, Comments? Contact… Rebecca Zumeta rzumeta@air.org Mike Jacobsen mjacobse@whiteriver.wednet.edu