Evaluation Process and Tools for Schoolwide PBIS

1. Evaluation Process and Tools for Schoolwide PBIS Jim Artesani, Ed.D. University of Maine College of Education and Human Development

2. What each participant needs to bring and do! • Handouts listed in “Read This First!” • Actively participate – even if first time on adobe • Ask questions – I can’t see your facial expressions : ) • Raise your virtual hand if you have questions

3. Our Purposes Today • To provide an overview of the core components of a comprehensive evaluation of PBIS in their school. • Explain steps to complete the Team Implementation Checklist, Self-Assessment Survey, School Safety Survey, and Benchmarks of Quality (BOQ) • Develop an understanding of how the survey tools mentioned in #2 can be used in conjunction with regular existing data (attendance, office referrals, etc) for decision-making. • Provide a schedule for conducting necessary assessments during the academic year.

4. Focus Schoolwide (Tier I) Implementation and Fidelity

5. Tools • Team Implementation Checklist (TIC) • Self Assessment Survey (SAS) • Classroom Management Checklist (CMC) • Benchmarks of Quality Indicators (BoQ)

6. Where do PBIS evaluation procedures and tools come from, and how where they created? PBIS.org (Evaluation)

7. Why Evaluate PBIS (Take 1) • Large-scale process • Significant undertaking • Ask and answer meaningful questions • We have the right measures to answer them • Gather the correct information for the purposes of decision-making and dissemination

8. Why Evaluate PBIS (take 2) • To examine the fidelity of implementation - Are we really doing what we think we are doing? To document effectiveness of implementation - Is what we’re doing working? To identify and examine strengths and weaknesses of implementation - Celebrate successes - Identify areas to improve

9. Evaluation Questions for Today Fidelity • Question: Are we implementing the components of Schoolwide Positive Behavior Support with fidelity?

10. Implementation vs. Intervention • Intervention effectiveness is measured by outcomes. • Implementation defined as “A specified set of activities designed to put into practice an acitivity or program of known dimensions (Fixsen, Naoom, Blasé, Friedman, & Wallace, 2005).”

11. Measure of Implementation is referred to as: FIDELITY or TREATMENT INTEGRITY

12. Schoolwide Fidelity Measures • Team Implementation Checklist • Self Assessment Survey (SAS) • Benchmarks of Quality Indicators (BoQ)

13. Team Implementation Checklist (TIC) • Question: Are we implementing the components of Schoolwide Positive Behavior Support with fidelity?

14. Team Implementation Checklist (TIC) • This checklist provides a scope and sequence for implementing schoolwide behavior supports. Schools should not expect to see change in student behavior until at least 80% of items are achieved (scored as “2”) on the checklist. • Team Checklist results are used to adapt action plans to school positive behavior support needs by monitoring the implementation and maintenance of PBS systems. Checklist results thus help determine the most effective allocation of faculty, financial, and other school resources.

15. Tool Time ActivityTeam Implementation Checklist (TIC) • Look at “Completed TIC Checklist” • Raise your virtual hand if you have questions • Scoring • Achieved = 2 • In Progress = 1 • Not Started = 0

16. Self-Assessment Checklist (SAS) Find “Completed SAS Checklist” Completed annually by all staff at beginning of fall or early spring

17. Purpose of SAS • Survey results are summarized and used for a variety of purposes including: • awareness level of building of staff, • annual action planning, • internal decision making, • Focuses on subsystems • assessment of change over time, and • leadership team validation.

18. Self-Assessment Checklist (SAS) Methods: • All staff at a staff meeting. • Have each leadership team member be responsible for getting 5-10 staff members to complete the assessment. • Individuals from a representative group. • Team member-led focus group. • Most schools experience low response rates when they provide survey and request that staff members complete on their own time.

19. To Promote Accuracy • Decide: • When and where staff will complete the SAS? • How you will support staff in they have questions? • How staff who are absent on the scheduled day will be able to complete the SAS? • Who will be in charge of creating the reports and sharing them with staff? • Don’t forget to compare to your TIC findings to SAS Results!

20. Classroom Management Checklist

21. Tool Time Activity Take 5 minutes to review the CMC

22. Benchmark of Quality Indicators (BoQ)OVERVIEW Internal and External Review

23. Background • Developed by Florida PBS at the University of South Florida • Assesses development and implementation of school-wide PBS • Lists 53 benchmarks of quality in school-wide PBS programs within 10 critical elements • Completed by school teams annually to identify areas of strength and weakness • Used by state and districts to guide TA and training and evaluate outcomes related to level of implementation

24. Benchmarks of Quality • 53 items aligned with PBS training process addressing areas of: • Faculty commitment • Effective procedures for dealing with discipline • Data entry and analysis plan • Expectations and lesson plans developed • School-wide recognition system established • Implementation plan • Crisis plan • Evaluation

25. Benchmarks of QualityRating and Scoring Components • Team Member Rating Form • Completed by team members independently • Returned to PBS Coach • Scoring Form, Tally Sheet, and Max Scores per Element • Completed by Coach using Scoring Guide • Used for reporting back to team • Scoring Guide • Describes administration process • Rubric for scoring each item

26. BOQ Will Provide: • Summary of team members’ perceptions of PBIS implementation (scored: ++ in place, + needs improvement, and - not in place) • Objective assessment of school’s implementation based on criteria described in a rubric (100 point scale) • Comparison between the above factors which will encourage discussion of strengths and weaknesses and provides ideas for action planning

27. STEP 1 • PBIS Coach uses Scoring Guide to complete: School-Wide Benchmarks of Quality: Scoring Form

28. Benchmarks Practice:Scoring Form, Scoring Guide

29. Benchmarks Practice:Scoring Form, Scoring Guide

30. STEP 2 • PBIS Coach asks every PBIS Team Member to complete the: School-Wide Benchmarks of Quality Team Member Rating Formindependently • Do not give a Scoring Guide to Team Members • Giving a deadline may be helpful

31. Coach collects Team Member Rating Forms • Coach tallies all responses provided by team members using the Team Member Rating Form: Tally Sheet

33. Completing Step 2 • Using the Team Member Rating Form: Tally Sheet, the PBIS Coach will record on the Benchmarks of Quality Scoring Form the team’s most frequent response for each item • If there is a tie, score the higher score

34. Benchmarks Practice:Scoring Form, Scoring Guide

35. STEP 3 • PBIS Coach places a Check next to any item where there is a discrepancy between the coach’s rating and the item’s most frequent response.

36. Benchmarks Practice:Scoring Form, Scoring Guide

37. Completing Step 3 • The coach will then complete the Team Summary on p. 3 of the Benchmarks of Quality Scoring Formrecording areas of discrepancy, strength and weakness. • The Team Member Rating Form: Tally Sheet will be useful in defining and explaining the areas of discrepancy

38. Areas of Discrepancy Administrator does not actively support the process 2 ++, ++, + 0 Areas of Strength Areas in Need of Development

39. Important: • If upon sharing areas of discrepancy (formally or informally), the coach realizes that there is new information that according to the Scoring Guide would result in a different score, the item and the adjusted final score should be recorded on the Scoring Form • It is recommended that changes be noted with a different color ink

40. Adjusting scores after discussion

41. STEP 4 • Coach will report back to the team using the Team Report page of the Benchmarks of Quality: Scoring Form • Again, adjustments to scores may be made using different color ink • Coach leads team through a discussion of the identified areas of strength (high ratings) and weakness (low ratings). This information should be conveyed as “constructive feedback” to assist with action planning.

42. STEP 5 • Coach adds up all the points on the Benchmarks of Quality: Scoring Form (maximum score is 100) • Coach turns in copies of Benchmarks of Quality Scoring Form andSchool-Wide Benchmarks of Quality Team Member Rating Formsto PBIS Facilitator in their school district

43. Evaluation Schedules

44. Questions??????