Using PBIS Assessment to Create School and District PBIS Evaluation Plans

1. Using PBIS Assessment to Create School and District PBIS Evaluation Plans Northern California PBIS Symposium November 18, 2013 Are we doing what we said we would do?

2. Agenda • Overview of available surveys/tools on the PBIS Assessment website • Developing an Evaluation Action Plan • Decisions on which tools to choose • Decisions on timelines • Assessment Example Plans • Developing Evaluation Reports • Questions/Comments

3. Acknowledgments • School-wide Positive Behavior Support: Implementers' Blueprint and Self-Assessment • Sugai, G., Horner, R.H., Algozzine, R., et.al. (2010). • Evaluation Blueprint for School-Wide Positive Behavior Interventions & Support • Algozzine, R., Horner, R.H, Sugai, G., et.al. (2010). • Fidelity Measures to Improve Implementation of Positive Behavioral Support • Tobin, T. J., Vincent, C. G., Horner, R. H., Rossetto Dickey, C., & May, S. A. (2012).

4. PBIS Evaluation Blueprint “A hallmark of School-wide Positive Behavior Support (SWPBS) is a commitment to formal evaluation.” “Evaluation is the process of collecting and using information for decision making.

5. Foundations • Evaluation is the process of gathering data to • Answer questions • Make decisions • Define the core questions/decisions, before selecting measures. • Provide an adequate context for interpreting the data. • Effective evaluation is cyclical (repeated) • Use the evaluation process for improvement

6. Evaluation Cycle • Core Questions Evaluation Plan Data Collection Evaluation Report

7. Core Evaluation Questions Context • What are/were the goals and objectives for SWPBIS implementation? (state/district capacity; school adoption; student outcomes) • Who delivered the training and technical assistance for SWPBIS implementation? Input • What professional development events were part of SWPBIS implementation support? • Was projected level of TA capacity provided (training/coaching)? • Who received the professional development (schools/ cohorts)?

8. Core Evaluation Questions Fidelity • To what extent was Tier I SWPBIS implemented with fidelity ? • To what extent were Tier II and Tier III SWPBIS implemented with fidelity? • To what extent is the Leadership Team establishing core functions?

9. Visibility Political Support Funding Policy Leadership Team Active Coordination Training Coaching Behavioral Expertise Evaluation Local School/District Teams/Demonstrations Sugai et al., www.pbis.org

10. Core Evaluation Questions Impact • To what extent is SWPBIS associated with changes in student behavioral outcomes? • To what extent is SWPBIS associated with changes in academic performance, and dropout rates? • To what extent is district/state capacity established? (local training, coaching, evaluation, behavioral expertise) • To what extent is leadership and policy structure established?

11. Core Evaluation Questions Replication, Sustainability, and Improvement • To what extent do Fidelity and Impact outcomes sustain across time? • To what extent does initial SWPBIS implementation affect Implementation with later cohorts? • To what extent did SWPBIS implementation change educational/behavioral capacity/policy? • To what extent did SWPBIS implementation affect systemic educational practice?

12. Evaluation Questions • Fidelity • To what extent was SWPBS implemented as designed? • To what extent was SWPBS implemented with fidelity? “Are we doing what we said we would do?”

13. To What Extent Can • PBIS Applications.org • PBIS Assessment • PBIS Evaluation • Help us answer the question: “Are we doing what we said we would do?”

14. Current Tools in PBIS Assessment

15. Current Tools in PBIS Assessment

16. A Tour of at www.pbisapps.org

17. Evaluation Action Planning Decisions: • Evaluation Tools • Which tools answer your district/schools’ evaluation questions? • Evaluation Timeline • When/how often will you administer the tools? • Evaluation Reporting • How will you report outcomes to schools and to the District? • How often will you report outcomes to schools and to the District?

18. Evaluation Action Planning • Decisions about Evaluation Tools • Does your District/School need a research tool? • (SET/ISSET) • Which annual assessment tool(s) will you use? • BoQ, SAS, SSS, BAT, ECBoQ • Which progressing monitoring tool(s) will you use? • TIC, MATT • Does your District/School need to evaluate Tier I or Tier II & III implementation or all three?

19. Evaluation Action Planning It is about… CONTEXT Your settings => Your needs

20. Which Surveys Meet Your District’s Needs?

21. Example District A Evaluation Questions • What is our baseline, and what has been our progress through the year? • Are we improving on our implementation of PBIS? • How do our individual staff members rate our implementation? • Are our self-assessments consistent with data from outside evaluators? Newly implementing, grant needing specific research data as well as fidelity of implementation data

22. District A: Evaluation Plan

23. Example District B Evaluation Questions • In what specific PBIS areas are we strong, and in which areas do we need more training? • Are we improving on our implementation of PBIS? • How do our staff members actually rate our PBIS implementation? • How is our implementation of Tiers II and III? ImplementingPBIS for three years, have been using the TIC/SET, and have been consistently over 80%.

24. District B: Evaluation Plan

25. Example District C Evaluation Questions • Which schools are not meeting the SET benchmark? • In what specific PBIS areas are we strong, and in which areas do we need more training? • How can we tailor our training and evaluation to meet the needs of various schools? • Are we improving on our implementation of PBIS? • How do our individual staff members rate our implementation? Implementing for five years, some schools are doing very well, but others are not making progress.

26. District C: Evaluation Plan: Cohort 1 Schools still not performing well

27. District C: Evaluation Plan: Cohort 2 Schools performing well

28. Example District D Evaluation Questions • In what specific PBIS areas are we strong, and in which areas do we need more training? • In what academic areas are we strong, and in which areas do we need more improvement? • How can we tailor our training/evaluation to meet the needs of various schools? • How can we schedule our evaluations so that we are not overwhelming our schools? The district wants to bring RtI Behavior and Academics together and create one evaluation plan.

29. Nieves Flores, 2012

30. Task: Developing Your Plan Questions to Consider: • Which surveys are your schools/districts currently using? • Do they answer your evaluation questions: • Are we implementing SWPBIS with fidelity? • Is there more information that we need? • Pbisapps.org > PBIS Assessment > Find Out More about Surveys • Do your surveys assess the Tiers you are implementing? • Do you need a research tool?

31. Task: Developing Your Plan • Are you “topping out” of the surveys you are currently taking? • Do you need to customize your evaluation plan for different schools, or will one evaluation plan fit for all schools? • Do you want to assess all tiers of PBIS implementation? • Is your Evaluation Plan manageable for schools?

32. District Evaluation Planning Template

33. SWPBIS School or District Evaluation Reports

34. Basic Contents of SWPBIS Evaluation Reports From the PBIS Evaluation Blueprint • Context • Input • Impact on SWPBIS Fidelity • Impact on Student Outcomes • Implications for the Future

35. Context • Define SWPBIS • Define goals of the specific project • Define stakeholders and evaluation questions

36. Input Who received what support and from whom? • Leadership team • Local Capacity Building • Training, Coaching, Behavioral Expertise, and Evaluation • School teams

37. Impact on SWPBIS Fidelity • Leadership Team • SWPBIS District Leadership Team Self-Assessment • Capacity Development • Number of trainers/coaches available to support school teams/districts • Behavioral expertise available to support Tier II and Tier III implementation • Evaluation capacity • School Teams • Tier I Implementation (TIC, BoQ, SET, SAS) • Collectively and/or by training cohort • Tier II / Tier III Implementation (MATT, BAT, ISSET) • Collectively and/or by training cohort

38. Impact on Student Outcomes Behavioral Outcomes • Office Discipline Referrals • Suspension/Expulsion • Attendance • Drop Out/Graduation • Disproportionality in ODR and suspension/expulsion Academic outcomes • Curriculum Based Measures • Oral Reading Fluency, Dibels, AimsWeb • Standardized Measures • State and National • Math, reading, writing assessments

39. Impact and Future Directions • Advances to date • Plans for next three years • Areas in need of adaptation • Narrowing of evaluation questions and concerns

40. Sharing Evaluation Reports State and District examples at PBIS.org Send in your district examples, so we can all learn!

41. Support and Resources • Portal Support Section • Search Function • Community Forum • Chat with us online • Account Questions: accounts@pbisassessment.org • Technical Assistance Questions: support@pbisassessment.org • New phone number • 1-855-455-8194

42. Questions & Answers • When you leave the webinar, please complete the survey. THANK YOU!