Promises and Pitfalls in School-wide Positive Behavioral Interventions and Supports (PBIS)

1. Promises and Pitfalls in School-wide Positive Behavioral Interventions and Supports (PBIS) Catherine Bradshaw, Ph.D., M.Ed. Johns Hopkins Center for the Prevention of Youth Violence (CDC) Johns Hopkins Center for Prevention and Early Intervention (NIMH) Department of Mental Health Penn State IES Fellows: May 17, 2011

2. PBIS Model: Whole-school Prevention • Application of behavioral, social learning, & organizational behavioral principles • Clear behavioral expectations (e.g., ready, responsible, and respectful) • Positive rewards • Procedures for managing disruptions (Horner & Sugai, 2001; Lewis & Sugai, 1999; Sugai & Horner, 2006)

3. MacArthur Middle School Eagle Code Be Responsible Respect Yourself Respect Others

5. Eagle Tickets

6. PBIS Model: Whole-school Prevention • Application of behavioral, social learning, & organizational behavioral principles • Clear behavioral expectations (e.g., ready, responsible, and respectful) • Positive rewards • Procedures for managing disruptions • Focus on changing adult behavior • Emphasizes staff buy-in • Team-based & data-based process • Consistency in discipline practices (Horner & Sugai, 2001; Lewis & Sugai, 1999; Sugai & Horner, 2006)

7. Office vs. Classroom Managed

10. PBIS Model: Whole-school Prevention • Application of behavioral, social learning, & organizational behavioral principles • Clear behavioral expectations (e.g., ready, responsible, and respectful) • Positive rewards • Procedures for managing disruptions • Focus on changing adult behavior • Emphasizes staff buy-in • Team-based & data-based process • Consistency in discipline practices • Can be implemented in any school level, type, or setting • Non-curricular model – flexible to fit school culture & context • Coaching to ensure high fidelity implementation • On-going progress monitoring • Public health approach (universal / selective / indicated) • Requires a shift from punitive/reactive to preventive (Horner & Sugai, 2001; Lewis & Sugai, 1999; Sugai & Horner, 2006)

11. Maryland’s Tiered Instructional and Positive Behavioral Interventions and Supports (PBIS) Framework Behavioral Systems Academic Systems • Intensive, Individually Designed Interventions • Strategies to address needs of individual students with intensive needs • Function-based assessments • Intense, durable strategies • Intensive, Individually Designed Interventions • Address individual needs of student • Assessment-based • High Intensity • Targeted, Group Interventions • Small, needs-based groups for • at risk students who do not respond • to universal strategies • High efficiency • Rapid response • Targeted, Group Interventions • Small, needs-based groups for at- risk students who do not respond to universal strategies • High efficiency/ Rapid response • Function-based logic • Core Curriculum and • Differentiated Instruction • All students • Preventive, proactive • School-wide or classroom • systems for ALL students • Core Curriculum and • Universal Interventions • All settings, all students • Preventive, proactive • School-wide or classroom systems for ALL students and staff 1-5% 1-5% 5-10% 5-10% 80-90% 80-90% (MSDE, 2008)

12. State District School Classroom Student Maryland’s PBIS Organizational Model School Level • 826 PBIS Teams (one per school) ≈ 59% • Team leaders (one per school) • Behavior Support Coaches (560) District Level (24) • District Coordinators State Level • State Leadership Team • Maryland State Department of Education (MSDE) • Sheppard Pratt Health System • Johns Hopkins University • 24 Local school districts • Department of Juvenile Services, Mental Hygiene Administration • University of Maryland • Management Team • Advisory Group National Level • National PBIS Technical Assistance Center • University of Oregon, University of Connecticut, & University of Missouri (Barrett, Bradshaw & Lewis-Palmer, 2008; JPBI)

13. Project Target Group Randomized Trial of SW-PBIS Funding • Centers for Disease Control & Prevention • National Institute of Mental Health • Institute of Education Sciences Sample • 37 voluntary elementary schools across 5 school districts • Enrollment 227-983; 60% Caucasian; 48% suburban; 41% urban fringe; 49% Title I Design • Group randomized effectiveness trial • 21 PBIS & 16 “Focus/Comparison” • Baseline plus 4 years (spring 2002 - spring 2007) • Data from 29,423 students & 3,563 staff (Bradshaw et al., Prevention Science, 2009; School Psychology Quarterly, 2008; JPBI, 2010)

14. SET: PBIS Implementation Fidelity Notes. No significant differences between groups at baseline, but differences at all other years at p<.05. Overall SET score: Wilks’ Λ = .38, F (4,32) = 13.36, p <.001, partial η2 = .63, d = 3.22. (Bradshaw, Reinke et al., ETC, 2007)

15. Brief Summary of SW-PBIS Training Effects on Fidelity • High fidelity implementation and sustainability of PBIS • All trained schools reached high fidelity within 4 years (66% by the end of year 1) • All trained schools sustained high fidelity (Bradshaw, Reinke et al., ETC, 2007; Bradshaw et al. Prevention Science, 2009)

16. Analysis of Climate Data: Organizational Health Inventory (OHI) • OHI: 37 item staff-report measure of 5 aspects of a healthy functioning school (Hoy et al., 1991) • academic emphasis - students are cooperative in the classroom, respectful of other students who get good grades, and are driven to improve their skills • staff affiliation - warm and friendly interactions, commitment, trust • collegial leadership - principal’s behavior is friendly, supportive, open • resource influence - principal’s ability to lobby for resources for the school • institutional integrity - teachers are protected from unreasonable community and parental demands • overall OHI score (average of all items) • Analyses • Longitudinal analyses were conducted using a 3-level approach in Mplus 4.21 (Muthén & Muthén, 2006) • Intent to treat approach (Lachin, 2000) & moderated by fidelity (SET) • Adjusted for staff (sex, race, age) and school (FARMs, student mobility, faculty turnover, & school enrollment) covariates on intercept and slope

17. Effect of PBIS on Overall OHI * Sig. change (.05) Note. Adjusted means from 3-level model. * Intervention effect on slope of overall OHI significant at p<.05.

18. Effect of PBIS on Collegial Leadership * Sig. difference (.05) * Note. Adjusted means from 3-level model. * Intervention effect on slope of overall OHI significant at p<.05. Note. Adjusted means from 3-level model. * Intervention effect on slope significant at p<.05.

19. Brief Summary of SW-PBIS Training Effects on Climate • PBIS training associated with significant improvements in staff members’ report of school climate / organizational health • Principal leadership, collegial relationships, academic emphasis, recourse influence, institutional integrity, and overall OHI • Effect sizes ranged from .24 (AE) to .35 (RI) • OHI intercept and slope negatively correlated • Schools starting with lower levels of OHI tended to take longer to reach high fidelity, but improved the most (Bradshaw, et al., SPQ, 2008; Bradshaw et al. Prevention Science, 2009)

20. Brief Summary of Impacts of SW-PBISon Student Outcomes • Significant impacts for students: • Significant reduction in school-level suspensionsamong the PBIS schools • Students in PBIS schools were 32% less likely to receive an office discipline referral • A positive trend for school-level MSA academic performancewas observed (Bradshaw et al., JPBI, 2010)

21. Impact of SW-PBIS on Bullying and Rejection: HLM 3-Level Results Note. Measure is Teacher Observation of Classroom Adjustment TOCA (teacher-reported) * p<.05, ** p <.01, *** p <.001

22. Impact of SW-PBIS on Teacher-Reported Perpetration of Bullying Behaviors (N = 12,334, γ = -.02, t = -2.60, p<.05, ES= -.11)

23. Impact of SW-PBIS on Teacher-Reported Student Rejection (N = 12,334, γ = -.03, t = -2.32, p<.05, ES= -.14)

24. SW-PBIS Intervention Status by Grade Cohort on Rejection Control Younger

25. Impact of SW-PBIS on Teacher-Reported Disruptive Behaviors (N = 12,334, γ = -.02, t = -2.39, p<.05, ES = -.12)

26. Impact of SW-PBIS on Teacher-Reported Concentration Problems (N = 12,334, γ = -.03, t = -2.08, p=.046, ES=.09)

27. Impact of SW-PBIS on Teacher-Reported Emotion Regulation Skills (N = 12,334, γ = .03, t = -2.30, p=.045, ES = .10)

28. Impact of SW-PBIS on Teacher-Reported Positive Behaviors (N = 12,334, γ = .03, t = -2.11, p<.05, ES = .15)

29. Variations in the Impact of SW-PBIS • Is there variation in the impact of SW-PBIS based on the child’s baseline pattern of risk? • Is SW-PBIS more impactful for some students than others? • Parallel to ‘green’, ‘yellow’, and ‘red’ zone framework? • Used Latent Class Analysis (LCA) to examine variation • LCA is a person centered approach • Groups participants with similar patterns among indicator variables into latent classes (McCutcheon, 1987) • Analysis conducted in Mplus 6.1 (Muthén & Muthén, 1997-2010) • Fitting the classes • Substantive theory as well as statistical support (Nylund et al., 2007) • 5 indices: Akaike Information Criterion (AIC), Baysian Information Criterion (BIC; Schwartz, 1978), Sample Size Adjusted Baysian Information Criterion (SSA BIC; Sclove, 1987), Lo-Mendell-Rubin adjusted likelihood ratio test (LMR; Lo, Mendell, & Rubin, 2001), and the sample size adjusted LMR (Muthén & Muthén, 1997-2008)

30. Latent Class Analyses: Mean Baseline TOCA Scores Mean (Centered) Baseline Teacher Ratings of Student Behavior (TOCA) N = 12,334

31. Variation in Impact by LCA Membership:% within Class Experiencing (Untoward) Outcome Note. Numbers represent percents experiencing that outcome. Those sharing letters are significantly different at p<.05. N = 12,334

32. Conclusions of Main Effects • Several significant impacts of SW-PBIS on the school environment, staff, and students • Main effects on teacher reported adjustment (i.e., concentration problems, aggressive/disruptive behavior, bullying, rejection, emotion regulation, prosocial behavior) • Although students in both groups (PBIS and Comparison) tended to display higher rates of problem behaviors over time, students in PBIS schools faired better than those in comparison schools • Some indication that the intervention effects are strongest the earlier students are exposed to PBIS

33. Conclusions from Variations Results • Four classes (not 3) of risk patterns emerged • High risk (6.6%), At risk (23.3%), Normative (36.5%), Social-emotional Skilled (33.6%) • Those in the ‘high risk’ and the ‘at-risk’ faired the worst in the Control schools than in the SW-PBIS schools • Shows that main effects models may ‘wash out’ some significant program impacts • Consistent with an RtI framework, however even the higher risk students are doing ‘better’ in an SW-PBIS environment

34. Examining Contextual Factors Associated with Scale-up Research Aims • To identify school- and district-level characteristics which predict initial training, adoption, and the quality with which PBIS is implemented. • Greater need at the school- and district-level would be associated with initial training, but lower odds of adoption or quality implementation. • Greater district infrastructure and wealth would be associated with greater odds of all three outcomes. (Bradshaw & Pas, accepted pending revision)

35. Participating Schools • All 24 MD districts participate in the PBIS initiative. • 825 traditional elementary schools across 22 districts in the state were eligible to be included (i.e., two districts had three or fewer schools trained) of which 312 were trained. • 236 schools across 17 districts had been trained AND provided data, indicating active participation and could be analyzed for the implementation outcome.

36. Outcome Variables Training in school-wide PBIS • A school team of at least 4 individuals, including an administrator, attended the state’s two-day training event. Adoption of school-wide PBIS • School was trained ANDsubmitted implementation data in the spring of 2008 Implementation of school-wide PBIS • The Implementation Phases Inventory (IPI; Bradshaw, Debnam, Koth, & Leaf, 2009):implementation and maintenance scales and overall score.

37. Conclusions • Schools with greater need were more likely to receive training, and in some cases also adopt SW-PBIS. • Maryland schools self-identify for training; lower-performing schools seem to seek SW-PBIS as a way to improve their school. • Indicators of school disorganization do not serve as an obstacle to successful implementation of SW-PBIS. • The number of years since training and percent of certified teachers is associated with better implementation. • District variables are related to training and adoption, but not implementation. (Bradshaw & Pas, accepted pending revision)

38. Next Steps: Integrating PBIS with… • PATHS to PAX & PBIS (NIMH, IES; Ialongo & Bradshaw) • Integrate an evidence-based classroom management program (PAX/Good Behavior Game) and a social-emotional learning curricula (PATHS) with PBIS • Pilot work in 6 Baltimore City PBIS schools over 2 years • PBISplus Project (IES; Leaf & Bradshaw) • 45 MD elementary schools all implementing school-wide PBIS • Provides support to SSTs and teachers related to selection of evidence-based practices • 3 years of on-site support and ‘coaching’ provided through a PBISplus Liaison • Safe and Supportive Schools Grant (MDS3) (USDOE; MSDE) • Collect data on climate and student safety • Conduct a 60 high school randomized trial of the integration PBIS with evidence-based violence prevention, drug prevention, and truancy prevention programs • Double Check Cultural Proficiency & Student Engagement (IES; Bradshaw) • Develop school-wide and classroom –based management system s to reduce disproportionality in PBIS elementary and middle schools • Provide professional development, adapt the Classroom Check-up (Reinke, 2007), support data-based decision-making

39. Potential Challenges to the PBIS Model • What impact does the focus on a team-based adaptive process have on variation in implementation quality? • What are some strengths and limitations of the training model? • What is the effect on the classroom? • Is it a process or a program? • Controversial role of rewards • What is changing – students or environment? • Too behaviorally focused? What about internalizing or social-emotional factors? • Is it too adult focused/driven? • Cost benefit analysis – effect size vs. investment • What to do when the universal isn’t enough? • Will this work for urban schools or high schools?

40. Johns Hopkins Phil Leaf Katrina Debnam Chrissy Koth Mary Mitchell Elise Pas Maryland State Department of Education Milt McKenna Andrea Alexander Sheppard Pratt Health System Susan Barrett Jerry Bloom Acknowledgements Contact Information Catherine Bradshaw, Ph.D. Johns Hopkins Center for the Prevention of Youth Violence Johns Hopkins Bloomberg School of Public Health cbradsha@jhsph.edu PBIS Resources www.PBIS.org www.PBISMaryland.org Supported by NIMH (1R01MH67948-1A, P30 MH08643), CDC (1U49CE 000728 & K01CE001333-01), IES (R324A07118, R305A090307, & R324A110107 ) & USDOE