School Based Leadership Team Year Two, Day Three (Corrected Version)

1. School Based Leadership TeamYear Two, Day Three (Corrected Version) A collaborative project between the Florida Department of Education and the University of South Florida

2. Advance Organizer • Data & Homework Review • Intervention Evaluation Protocol • Resource Maps • Evaluation Plan • Resource Mapping Activity • Intervention Integrity • Types • Barriers • Improving • Assessing • Evaluation Planning Activity

3. Intervention Protocol Evaluation Linking Intervention Planning with Evaluation Planning

4. Team Data Review 1. How much information do you think your team could obtain from reviewing referral patterns (e.g., referrals to Problem-Solving Team, School-Based Intervention Team, Student Services Team, Intervention Assistance Team referrals) at your school? What implications would those data have for providing interventions to your students? 2. Given the utility of the information gathered from referral patterns and your team’s performance on the first part of the case study, how helpful would it be for your team to code and analyze referrals for assistance at your school? What resources would you use/need to make coding and analyzing referrals feasible in your building? 3. Students in need of additional intervention were identified in the scoring Summary document. To what degree do you agree with the identification of those particular students? Please explain. 4. What universal screenings (e.g., DIBELS, ODRs, county assessments) are available to identify students at-risk at your school? What standard interventions are available to serve students identified as at-risk?

5. Homework Review Examples of: Data Sources Used Criteria for Decisions of which students are in need of Tier II instruction/intervention.

6. Protocols • A planned course of treatment… • prepared in advance for anticipated events… • that have a high probability of success… • based on research or history of evidence. • Example • Safety Protocols such as fire drills or evacuation plans • Medical Protocols (ex.Physician’s Desk Reference.) • Established social rules of behavior for given situations.

7. What are the components to an Intervention Protocol? • Identify Student Need • Assessment Data • Identify Resources and Intervention(s) available to address student need • Resource Maps • Intervention Implementation Plan • Who is responsible? What will be done? When will it occur? Where will it occur? • Intervention Support Plan • Who is responsible? What will be done? When will it occur? Where will it occur? • Integrity Measurement Plan • Who is responsible? What data will be collected and how documented? When or how often collected/documented? Where will data be collected? • Evaluation Plan • Who will evaluate plan effectiveness and collect data? When will data be collected, graphed, and reviewed? What is the criteria for determining success?

8. Developing Intervention Protocols Evidence based intervention linked to verified hypothesis planned How will your intervention be implemented? How will you evaluate progress? What data will you collect? Evidence based intervention implemented Student Outcomes (SO) Assessed Treatment Integrity (TI) Assessed Continue Intervention +SO +TI Data-based Decisions -SO -TI Implement strategies to promote treatment integrity -SO +TI Modify/change Intervention From Lisa Hagermoser Sanetti, 2008 NASP Convention

9. Comprehensive Intervention Plan Development Evaluation Plan Components Integrity Documentation Plan Intervention Plan Components

10. RESOURCE MAPS Linking Student Needs and School Resources

11. Benefits to Resource Mapping RESOURCES Resources Matched to Student Needs Resources Not Used, Planned, or Necessary Needs Potentially Not Met All Identified Student Needs Matched to Available Resources STUDENT NEEDS Develop Intervention Protocols

13. Resource Map Benefits Increase efficiency in planning and coordinating services for students Increase efficiency in communicating with staff Inform professional development Inform purchasing/hiring needs Inform infrastructure needs

14. Resource Maps and Intervention Protocols • Step 1: • Update resource maps to include all available resources for each tier of service and content area for all grades; for example: • Personnel (recommend use names) • Curricula/materials • Space/Facilities • Assessments/Measurements/Data Available • Meeting Structures • Consider involving all staff through a survey of expertise and knowledge about various research-based or evidence-based interventions. • Remember not all resources are tangible.

15. Resource Maps and Intervention Protocols • Step 2: • Review Student Data for a particular grade level at Tier 1 for a particular content area – including demographic groups across last 2-4 years. Look for recurring or predictable patterns/problems. • Step 3: • Where data review indicates needs at tier 1… • Engage in Problem-Solving to identify intervention needs for Tier 1. • Where data review indicates needs at tier 2 with little or no needs at tier 1… • Engage in Problem-Solving to identify intervention needs for tier 2.

16. Resource Maps and Intervention Protocols • Step 4: • Develop Intervention Protocol using Resource Map and based on verified hypotheses from Problem Analysis step. • Include in Intervention Protocol all components of plan: • Intervention plan, • Support plan, • Evaluation Plan, • Integrity Measurement Plan. • Develop integrity measures and ensure inventory of materials for data collection and intervention use.

17. Resource Maps and Intervention Protocols • STEP 5: • Train all appropriate staff on how to implement intervention protocols that are developed… • including how student outcomes and implementation integrity will be measured. • Also, include in training any additional building procedures (if any additional are needed) for staff to access support from the school-based team.

18. Resource Maps and Intervention Protocols • STEP 6: • Identify gaps in Resource Maps where students based on data are not supported with existing resources and/or resources available are not being used, planned, or needed.

19. Summary of Resource Maps Can be effective tool for aligning resources to match student population needs. Can be used to inform professional development activities Can be used to inform infrastructure needs Can be used to efficiently communicate with staff about available resources. Should be updated each year.

20. EVALUATION PLAN Evaluating Student Outcomes and Intervention Effectiveness

21. Evaluation Plan Components Who will be responsible for evaluating student outcomes and intervention effectiveness? What data will be collected, by whom, and at what frequency? Where will data be collected? What is the criteria for determining effectiveness?

22. Planning for Data Use “Research has found that up-front planning helped make data collection and use more efficient in many case study schools by clarifying what data were needed, aiding with integration of multiple data sources, and ensuring that data collection processes were on track (Keeney, 1998; Lachat, 2001).” -Kerr, et al., (2006). Strategies to promote data use for instructional improvement: Actions, outcomes, and lessons learned from three urban districts. American Journal of Education, 112, 496-520.

23. Data • Challenge: • Balance quality and utility of information with time away from instruction for assessments. • 4 Types of Assessment Data • Screening • Progress Monitoring • Diagnostic • Outcomes

24. Mapping Your Assessments

25. How valuable are the assessments you are using? • Traditional use vs. functional use • Why are you using the assessments you are using? • Evaluate type or purpose of each assessment. • Ex. Running Records? • Ex. Lexile? • Ex. FCAT? • Ex. Chapter Math Tests? • Ex. District Assessments? • Ex. DIBELS/CBM? • Ex. FAIR • How are teachers prepared for interpreting conflicting results of multiple assessments that seemingly measure the same thing?

26. Assessment Summary Some measures are great for some things, but not all things. Some useful for screening, but offer no diagnostic or progress monitoring value. Some useful for diagnostic purposes, but not screening or progress monitoring. In many cases, may need to coordinate use of multiple data sources to assess student need and progress.

27. Valuing the Use of Data • Barriers to teachers’ use of data: • Technical skills needed for management system • Lack of quantitative prowess • Philosophical commitments and political necessities • Data management policies and assessment practices that do not yield useful information • Lack of consensus around organizational goals • School norms that are inconsistent with accountability policies. • Immediacy of availability for using the data • Perceptions of validity and reliability. • Organizational influence by uncoordinated district Curr/Inst/Assess Plans Kerr et al., 2006; Young, 2006

28. Supports for Teachers Use of Data • Young, (2006) – Teacher’s Use of Data • Make data easy to access and use • Provide training and support to accurately interpret • Furnish instructional resources to help act upon data • Facilitate meetings for teachers to ask “so what” questions • Follow up with teachers on responses to data analyses

29. Resource Mapping Activity • No ID required • Choose: Option 1: Completion of a Resource Map for Tier II Services Option 2: Planning Services Based on Tier II Resources Identified • Work as a team – only 1 form needs to be completed.

30. Intervention Integrity

31. The “I” in RtI RtI-- based on the actuality of interventions delivered as intended: • Cannot assess RtI if the intervention is not implemented as designed • Integrity must be ensured and documented • Procedural safeguards

32. Two types of integrity: • Integrity of process • Integrity of intervention plan

33. Integrity of process… • All components implemented with integrity • Comprehensive PS approach • Intervention implementation • Valid educational decisions

42. Integrity of intervention plan Intervention integrity is the degree to which an intervention is implemented as originally designed. Gresham (1989)

43. Interventions in schools • Poor track record • Resistance • Lack of resources • Special Ed as magic bullet • No guarantees

44. Research tells us… • Interventions not implemented correctly • Inaccurate self-report • Implementation short-lived

45. “The reason most interventions fail is because they are not used…and the reason they are not used is because they are not feasible for teachers.” Joe Witt (2006)

46. Integrity and student outcomes Positive correlation: When interventions are implemented correctly, student performance improves

47. What degree of implementation is required? Consider: • Integrity may not impact performance • Poor implementation may cloak effective intervention • Level of necessary integrity varies • Goal should be effectiveness

48. Barriers to knowing needed level of integrity: • Lack of integrity data in research • Heterogeneity—population, interventions, outcomes, errors • “Unknown” critical steps

49. Hmmmmm….. Considering the importance of intervention plan implementation, and the difficulty of assuring that implementation occurs, some may surmise that the Achilles’ heel of RtI has finally been exposed. Noell and Gansle (2006)

50. Ensuring Integrity What factors diminish intervention integrity? What can be done to improve integrity?