Establishing a District Evaluation Plan

1. Establishing a District Evaluation Plan Susan Barrett sbarrett@midatlanticpbis.org Director, Mid-Atlantic PBIS Network Sheppard Pratt Health System

2. Outcomes • District Team • Clear function/purpose (aligned with SPP) • Members with clear role and function • Communication/feedback loops • Implementation/Evaluation Plan

3. Professional Development • Training Curriculum • Training Capacity • Site Selection • Readiness • Coaching Data Base PBS Surveys, PBS Eval Local data base Implementation and Evaluation Plan • What data point should change? How will we track progress and show impact? • Selection of theTools Start here Marketing and Dissemination Annual Reports Presentations Newsletters

4. If something is worth doing once, it's worth building a tool to do it. • Evaluation Template- What will the impact be? • Identify Progress Monitoring Tools • Tier 1 • Tiered Fidelity Inventory • School-wide Evaluation Tool (annual for 20%) • Tier 2/3 • Tiered Fidelity Inventory • I-SET

5. Implementation Tools • How can we organize the work? • Can we work smarter to provide coaching and technical assistance by tracking progress across phases? • Can district and state use Phases of Implementation (POI) to track progress and assign resources?

6. Exploration Stage Need for change identified, possible solutions are explored, learning about what it takes to implement the innovation effectively, stakeholders are identified and developed, and decision is made to move forward Installation Resources needed to implement innovation with fidelity and desired outcomes are in place Initial Implementation Innovation is in place in schools, implementation largely guided by external TA providers Full Implementation- Innovation is implemented and sustained by local stakeholders and is well-integrated into policy/written documentation Innovation and Sustainability Innovation is adapted to fit local context, innovation becomes more efficient and is integrated with other initiatives

7. POI Tools-Coach • POI- Illinois- All Tiers • PIC- Florida- All Tiers • Implementation Phase Inventory- MD Tier 1- strong reliability

8. Lessons Learned • Training sequence and materials are developed • Refinement of implementation and deepening content expertise in the State, District and schools continuous and areas of improvement are identified by data • A unified, expanded cadre of trainers and implementation coaches prepared to deliver professional development • Trainers understand content and implementation science

9. Lessons Learned • An aligned training and support system for both trainers and implementation coaches. • Standardized training and coaching materials. • Standardized data collection system, tools and processes with practice-level feedback loops at each level of the system for planning and continuous improvement.

10. Readiness Structures4-6 months of “Pre-Work” Tier 1 • Leadership Forum- • Planning Phase Checklist • SWIS Readiness • Leaders, Coaches, New Teams walk in prepared with basic knowledge, data, SIP Advanced Tiers • Readiness- completed by Local Coordinator and Team • CICO Readiness, CICO SWIS Readiness • Readiness T2/T3

11. Training Content- Example “Anchored on the tool” Overview Critical Feature Modules • Trainer Presentation • Workbook (self assess, activity , action step) • Implementation Snapshot • Team Workbook

12. Professional Development Blueprint… • Lists and describe minimum core readiness, skill, and performance competencies for SWPBS trainers. • Lists and describe minimum outcome competencies for SWPBS implementation by (a) school staff, (b) school leadership teams, and (c) coaching personnel. • Provides self-assessment guides for monitoring training and coaching progress. • Provides evaluation procedures for assessing training and coaching fidelity and integrity. • Recommends schedules and sequence for training and coaching activities. • Assumes that trainers have implementation experience and fluency with SWPBS practices and systems. • Focuses on the school, district/region, and state as the context for sustained training implementation and support.

13. Rethinking Technical Assistance • Moving from a case by case expert model to building expertise in the school • Focus of all TA is on teaching the school team to solve problems or address challenges for themselves • Shift from providing answers to asking questions • Shift from developing plans to prompting plan development • Shift from being viewed as the expert to being viewed as a facilitator • Will not replace need for specialist, re-focus all to building capacity

14. Build parallel systemic processes • Provide school/district teams with a process to address the presenting challenge (SWPBS) • Develop a parallel process for districts/states to support school implementation and continue to expand with integrity (District /State Leadership Team)

15. CSBS Training ModelCreating the Conditions for Skill Development • Intro to Administrators • Readiness/Planning Phase • Team and Coach Training Event • Facilitated Action Planning- Team Time Workbook • Track Fidelity/Progress • Access to Community of Practice-knowledge sharing • Track Outcomes Building Training Capacity- Scope and Sequence Training Event Implementation Team Workbook

16. PBISBuilding Capacity and Sustainability • Project Leadership Team School Staff, Families, Transportation, Communities • State Implementation Team • Regional Implementation Team • District Coordinators • Coaches Team Leaders • Problem-Solving Teams

17. State Example March 30 Spring Forum Administrator attends July 18-22 New Team Training Feb- June Planning Phase for New Teams Checklist Completed Sept. 1st – Feb. 28th New SWIS and CICO-SWIS subscriptions setup and invoiced Jul Jan Oct Feb Apr Dec Jun Aug Nov Mar Sept May Oct 21 State Coaches Meeting Nov 10 IPI Due Dec 6 State Coaches Meeting April 7 State Coaches Meeting April 10 IPI Due August Team Planning SWIS Readiness Completed ** Regional Returning Team Events held during Summer- Dates TBA **CICO Training completed Regional by request

18. Training CapacityFidelity • Delivering the training content as it is intended to be delivered is essential • Participants who attend a PBIS training regardless of what part of the state they reside, should have the same training experience • This is a challenge as scaling occurs • TOT developmental process • Learner, Content Fluent, Co-trainer, Trainer

19. Implementation Snapshot • Used during training to clearly define each critical feature and the research that supports use and potential benefit to overall school climate. • The trainer facilitator will provide training slides for each feature. Teams will have an opportunity to use their guidebook and items brought to the training to use during action planning sessions. Roles of each participant are clearly described each snapshot to ensure clear expectations. • Administrators are expected to play an active role. Coaches are expected to guide process, communicate with the district coordinator and share implementation products and examples as needed. • Teams will walk away from the training with an annual action plan that will help guide the effort.

20. Building Coaching Capacity Occurs at ALL Levels • Systems • Conditions that support skill development • for staff • Policy and Procedures alignment • Budget Re-allocation • Recruitment and Selection of Coaches • Supervision of Coaching within Organization • Training Curriculum and Scope and Sequence • Access to certification • Facilitative Administrator Supports • Practices/Skills • The technical skill set required to achieve fidelity • Problem Solving (Team, Classroom, Staff, Student) • Team Building/Collaboration • Delivering Feedback • Behavioral Consultation State District Building Classroom Staff Student Family • Data • Information required to guide skill development process • Action Plan with short/long term measurable goals • Self Assessment • Process Measures/Fidelity Checks • Performance Feedback Measure • Progress Monitoring Tools • Evaluation Tools • Student Outcomes • Data used for continuous regeneration (PEP/PIP)

21. Breaking it down … • Stages of Implementation • State and District Level Implementation Guide

22. Stages of Implementation Exploration/Adoption Installation Initial Implementation Development Commitment Full Implementation Establish Leadership Teams, Set Up Data Systems Innovation and Sustainability Provide Significant Support to Implementers Embedding within Standard Practice Should we do it? Improvements: Increase Efficiency and Effectiveness Doing it right Doing it better

23. Exploration Stage Need for change identified, possible solutions are explored, learning about what it takes to implement the innovation effectively, stakeholders are identified and developed, and decision is made to move forward Installation Resources needed to implement innovation with fidelity and desired outcomes are in place Initial Implementation Innovation is in place in schools, implementation largely guided by external TA providers Full Implementation- Innovation is implemented and sustained by local stakeholders and is well-integrated into policy/written documentation Innovation and Sustainability Innovation is adapted to fit local context, innovation becomes more efficient and is integrated with other initiatives

24. State District Implementation Guide • Workbook organized by POI • Specific outcomes for each Phase • Guiding Questions to build case/consensus • Facilitated Activities that guide similar TA process • Never meant to be a “paper product”

25. Professional Development/ Technical Assistance Guiding Questions Facilitated Exploration Activities Outcomes

26. Exploration Phase Need for change identified, possible solutions are explored, learning about what it takes to implement the innovation effectively, stakeholders are identified and developed, and decision is made to move forward

27. Before you add one more thing…. www.safetycenter.navy.mil

28. Exploration Phase • Major emphasis is on long term commitment, innovation and implementation are discussed so that stakeholders understand system-level features necessary for initial and sustained implementation of the innovation

29. TIER I: Core, Universal GOAL: 100% of students achieve at high levels Tier I: Implementing well researched programs and practices demonstrated to produce good outcomes for the majority of students. Tier I:Effective if at least 80% are meeting benchmarks with access to Core/Universal Instruction. Tier I: Begins with clear goals: What exactly do we expect all students to learn ? How will we know if and when they’ve learned it? How you we respond when some students don’t learn? How will we respond when some students have already learned? Questions 1 and 2 help us ensure a guaranteed and viable core curriculum 29

30. TIER II: Supplemental, Targeted Tier II For approx. 20% of students Core + Supplemental …to achieve benchmarks Tier II Effective if at least 70-80% of students improve performance (i.e., gap is closing towards benchmark and/or progress monitoring standards). Where are the students performing now? Where do we want them to be? How long do we have to get them there? How much do they have to grow per year/monthly to get there? What resources will move them at that rate? 30

31. Tier III For Approx 5% of Students Core + Supplemental + Intensive Individual Instruction …to achieve benchmarks Where is the students performing now? Where do we want him to be? How long do we have to get him there? What supports has he received? What resources will move him at that rate? Tier III Effective if there is progress (i.e., gap closing) towards benchmark and/or progress monitoring goals. TIER III: Intensive, Individualized 31

32. Installation • Resources needed to implement innovation with fidelity and desired outcomes are in place Identify: • Coordinator • State Team • Implementation Team • Trainers, Coaches • Roles and Functions must be clear • Use the Blueprint and Team Workbook

33. Initial Implementation • Commitment to Capacity Building • Demonstrated High Fidelity/High Impact • Demand Increases • State Team won’t be able to keep up with demand

34. Features • Point of Contact and Coaches become Local Coordinators • Transfer role to local person • Use phase of implementation to guide decision points • Meet with local team to build action plan- model after state team

35. District Level Phases

36. Full Implementation Innovation is implemented and sustained by local stakeholders and is well-integrated into policy/written documentation • “New” is now standard practice

37. Innovation and Sustainability • Innovation and Integration • Demonstrated impact throughout • Change/Adapt to fit culture every year • Renew Commitment • Easier, More Efficient, Cost Reduces • Organizational Framework allows for integration • Educators as better consumers

38. Hybrid Training: Face to Face combined with Use of Technology • Training – always an evolving process…. • Continuous opportunities to deepen both content and implementation knowledge and skills. • Establish training cycle that includes: • Face to face training and planning • Storage space • Adobe connect

39. Using a Google Drive, Dropbox • “Fast” access to content/tools/resources • Targets group needs • Build with group • Interactive or Storage Space