Implementing an Evaluation System for SWPBS

1. Implementing an Evaluation System for SWPBS Bob AlgozzineRob HornerNational PBIS Leadership ForumChicago Hyatt Regency O’HareOctober 8, 2009http://www.dpi.state.nc.us/positivebehavior/

2. Objectives • To provide brief overview of evaluation field • To describe NC PBS evaluation model • To illustrate North Carolina’s evaluation efforts • Context • Input • Fidelity • Impact

3. Collecting and Reporting Information Who Where When Why What How “Make a Case” What do we know? What do we need to know? How good is acceptable? How bad is unacceptable? Evaluation

4. EvaluationCollecting and Reporting Information for Decision-Makers Describe Judge Stufflebeam, D. L., & Shinkfield, A. J. (2007). Evaluation theory, models, and applications. San Francisco, CA: Jossey Bass.

5. EvaluationCollection and Reporting of Information for Decision-Makers What difference did it make? What should we do? Who Where When Why Defining Outcome Product Context Analyzing Obtaining Project Input Process What How Applying Are we doing it as planned? How should we do it?

6. SWPBS/NC EvaluationCollection and Reporting of Information for Decision-Makers Who Where When Why Plan What difference did it make? Impact Context School-Wide Positive Behavior Support Compare Perform Input Fidelity What How Measure

7. NC PBS EvaluationCollection and Reporting of Information for Decision-Makers The North Carolina Positive Behavior Support Initiative is part of the North Carolina State Improvement Program funded through IDEA. VISION [IMPACT] All schools in North Carolina will implement Positive Behavior Support as an effective and proactive process for improving social competence and academic achievement for all students. MISSION [CONTEXT, INPUT, FIDELITY] Provide leadership, professional development, resources, and on-going assistance in order for schools to successfully implement Positive Behavior Support.

8. NC PBS EvaluationCollection and Reporting of Information for Decision-Makers Context Districts and schools participating in the North Carolina Positive Behavior Support Initiative regularly document the goals, objectives, and activities of their programs to establish the extent to which systems and practices that should be available are available.

9. NC PBS EvaluationCollection and Reporting of Information for Decision-Makers Who Where When Why STATEWIDE LEADERSHIP AND COORDINATION One full-time Consultant and eight part-time Regional Coordinators support implementation of PBS in NC. The Consultant is in the Behavior Support and Special Programs Section of the Exceptional Children Division at the NC Department of Public Instruction. The Regional Coordinators are hosted by LEAs or another agency in their region and spend 1/3 of their time working with the PBS implementation in that school system or Charter School. The primary role of these professionals is to coordinate training, support trainers/coaches/coordinators in LEAs, and facilitate the evaluation of the statewide initiative. http://www.ncpublicschools.org/positivebehavior/coordinator/ Context

10. NC PBS EvaluationCollection and Reporting of Information for Decision-Makers Who Where When Why Context Eighty-four of the 100 counties in the state have at least one school participating in the North Carolina Positive Behavior Support Initiative.

11. NC PBS EvaluationCollection and Reporting of Information for Decision-Makers Who Where When Why Context Currently, 104 (90%) of the 115 counties and local education agencies in the state have at least one school participating in the North Carolina Positive Behavior Support Initiative.

12. NC PBS EvaluationCollection and Reporting of Information for Decision-Makers Who Where When Why Context Steady growth has been evident in the number of schools that have implemented PBS and current estimates suggest that about 85% are still implementing.

13. NC PBS EvaluationCollection and Reporting of Information for Decision-Makers Who Where When Why Context

14. NC PBS EvaluationCollection and Reporting of Information for Decision-Makers Input The North Carolina Positive Behavior Support Initiative includes critical features of SWPBS. Districts and schools participating in the North Carolina Positive Behavior Support Initiative regularly assess the extent to which key features are being implemented and they use this information to develop and refine action plans for sustaining the effort.

15. NC PBS EvaluationCollection and Reporting of Information for Decision-Makers • The North Carolina Positive Behavior Support Initiative includes critical features of SWPBS. • Leadership Team Representative stakeholders meets regularly to support effort. • Coordination State and local coordinators manage day-to-day operations. • Funding Implementation supported with targeted state-approved funds. • Visibility Ongoing local, state, and national dissemination efforts. • Political Support Leadership team reports annually on activities and outcomes. • Training Capacity Team of trainers is available to build and sustain practices. • Coaching Capacity Individuals are available to support emerging practices. • Demonstrations Large number of schools available for model demonstrations. • Evaluation Process in place for collecting and reporting information on • Implementation of action plan. • Use of SWPBS. • Impact of SWPBS on student outcomes. Input What http://www.dpi.state.nc.us/docs/positivebehavior/implementation/brochure/brochure.pdf

16. NC PBS EvaluationCollection and Reporting of Information for Decision-Makers • The North Carolina Positive Behavior Support Initiative focuses on four key elements and schools participating in the program intervene at three distinct, but related levels. • Universal Interventions • School-wide Rules & Procedures • Systematic Reinforcement • Social Skills Instruction • Culturally Responsive Practices • Data-Based Decision-making • Parent & Community Partnerships • Targeted Group Interventions • Social Skills Instruction • Reinforcement of Specific Skills • Group Behavior Strategies • Classroom Coaching • Intensive, Individual Interventions • Functional Behavior Assessment • Behavior Intervention Planning Input What http://www.dpi.state.nc.us/docs/positivebehavior/implementation/brochure/brochure.pdf

17. NC PBS EvaluationCollection and Reporting of Information for Decision-Makers Schools implementing PBS regularly assess the extent to which key features of PBS are being implemented and they use this information to develop action plans for refining and sustaining the effort. Input What

18. NC PBS EvaluationCollection and Reporting of Information for Decision-Makers Fidelity Schools participating in the North Carolina Positive Behavior Support Initiative regularly document the implementation levels of the support being provided and they use this information to refresh, redirect, and sustain their efforts. “Outcome evaluations should not be attempted until well after quality and participation have been maximized and documented in a process evaluation. Although outcome data can determine the effectiveness of a program, process data determine whether a program exists in the first place” (Gilliam, Ripple, Zigler, &Leiter, 2000, p. 56). Gilliam, W. S., Ripple, C. H., Zigler, E. F., & Leiter, V. (2000). Evaluating child and family demonstration initiatives: Lessons from the Comprehensive Child Development Program. Early Childhood Research Quarterly,15(1), 41-59.

19. NC PBS EvaluationCollection and Reporting of Information for Decision-Makers Average [08-09] scores on SET subscales varied across levels of schools (N=336) and were generally higher for elementary (n=226) than middle (n=68), high (n=24), or combined (n=18) enrollment schools. Fidelity How

20. NC PBS EvaluationCollection and Reporting of Information for Decision-Makers More schools reported SET scores and they were generally higher for 2008-2009 than for previous school years. Fidelity How

21. NC PBS EvaluationCollection and Reporting of Information for Decision-Makers Impact The North Carolina Positive Behavior Support Initiative is focused on improving social competence and academic achievement for all students. Information from impact evaluation indicators reflects the extent to which targeted outcomes are being and/or likely to be achieved.

22. NC PBS EvaluationCollection and Reporting of Information for Decision-Makers What difference did it make? Impact Office discipline referral data (majors) from schools implementing PBS in North Carolina [07-08] compare favorably with national averages.

23. NC PBS EvaluationCollection and Reporting of Information for Decision-Makers What difference did it make? Impact There has been a consistent decrease in reported suspensions across schools implementing PBS over the past five years.

24. NC PBS EvaluationCollection and Reporting of Information for Decision-Makers What difference did it make? Impact Levels of behavior risk in schools implementing PBS were comparable to widely-accepted expectations and better than those in comparison schools not systematically implementing PBS.

25. NC PBS EvaluationCollection and Reporting of Information for Decision-Makers What difference did it make? Impact Rates of office discipline referrals in schools that met SET expectations were lower than in schools that did not meet them.

26. NC PBS EvaluationCollection and Reporting of Information for Decision-Makers What difference did it make? Impact While achievement differences were not statistically significant across schools with different levels of implementation, effect sizes reflecting practical differences were large (.55 for reading and .98 for mathematics).

27. NC PBS EvaluationCollection and Reporting of Information for Decision-Makers What difference did it make? Impact [A]chievement causes [B]ehavior? [B]ehavior causes [A]chievement? [C]ontext causes [A]chievement and [B]ehavior?

28. NC PBS EvaluationCollection and Reporting of Information for Decision-Makers • Sustainability • Continuing to Build Capacity • state • district • school • Expanding Role of Local Coaches • Expanding Efforts to Make Evaluation Easier • report and presentation templates • external support • www. pbsurveys.org • www. pbseval.org • www. pbsassessment.org

29. NC PBS EvaluationCollection and Reporting of Information for Decision-Makers Web site http://www.dpi.state.nc.us/positivebehavior/ Annual Performance Reports Available http://www.dpi.state.nc.us/positivebehavior/data/evaluation/