Evaluating Research on Challenging Behavior

1. Evaluating Researchon Challenging Behavior Chris Borgmeier Portland State University

2. From Spencer, Detrich & Slocum, 2012

3. www.pbis.org Horner, R., & Sugai, G. (2008). Is school-wide positive behavior support an evidence-based practice? OSEP Technical Assistance Center on Positive Behavioral Interventions and Support. http://www.pbis.org/files/101007evidencebase4pbs.pdf.

4. Evidence Basics From George Sugai

5. Why evidence-based? Maximize outcomes Minimize harm Increased accountability Increase efficiency Improve decision making Improve resource use

6. Basic Approach • Start w/ what has greatest likelihood of addressing (evidence-based) confirmed problem/question • Explained/supported conceptually/empirically • Adapt to local context/culture/need • Monitor regularly & adjust based on data • Adapt for efficient & durable implementation

7. 4 Evaluation Criteria • Effectiveness • Has/will practice produced desired outcome? • Efficiency • What are costs (time, resources, $) to implement practice? • Relevance • Is practice & outcomes appropriate for situation? • Conceptually soundness • Is practice based on theory?

8. Basic Practices Evaluation

12. Design Questions Has functional or cause-effect relationship been demonstrated & replicated? Have alternative explanations been accounted & controlled for? Have threats or weaknesses of methodology been controlled for? Was study implemented w/ fidelity/accuracy?

13. Research Designs Experimental - RCT & SSR Evaluation - Descriptive w/ baseline Case Study - Descriptive w/o baseline Testimonial - No/Limited data

14. Results Questions • Who were subjects? • How much like my participants? • Where was study conducted? • How much like where I work? • What measures were used? • Do I have similar data? • What outcomes were achieved? • Are expected outcomes similar

15. Effectiveness Logic • Significance (“believe”) • Likelihood of same effect by chance • Effect Size (“strength”) • Size of effect relative to business as usual • Consequential Validity (“meaning”) • Contextually meaningful

16. PBIS Evidence Base

17. School-wide PBIS – Let’s compare! • www.pbis.org • Click on ‘Resource Catalog’ • Then ‘Literature List’ • Evidence Base for SW-PBIS • Randomized Control Trials

18. Randomized Control Trials of SW-PBIS Tier 1/ Universal SW-PBIS • Bradshaw, C. P., Mitchell, M. M., & Leaf, P. J. (2010).Examining the effects of school-wide positive behavioral interventions and supports on student outcomes: Results from a randomized controlled effectiveness trial in elementary schools. Journal of Positive Behavior Interventions, 12(3), 133-148. • Bradshaw, C.,Koth, C., Bevans, K., Ialongo, N., & Leaf, P. (2008). The impact of school-wide positive behavioral interventions and supports (PBIS) on the organizational health of elementary schools.School Psychology Quarterly. • Bradshaw, C., Reinke, W., Brown, L., Bevans, K., & Leaf, P. (2008).Implementation of school-wide positive behavioral interventions and supports (PBIS) in elementary schools: Observations from a randomized trial. Education and Treatment of Children, 31, 1-26. • Horner, R. H., Sugai, G., Smolkowski, K., Eber, L., Nakasato, J., Todd, A. W., & Esperanza, J., (2009). A randomized, wait-list controlled effectiveness trial assessing school-wide positive behavior support in elementary schools. Journal of Positive Behavior Interventions, 11(3), 133-144. • Sprague, J., & Biglan, A., et al (in progress).A Randomized Control Trial of SWPBS with Middle Schools.

19. 90-School RCT StudyHorner et al., in press • Schools that receive technical assistance from typical support personnel implement SWPBS with fidelity • Fidelity SWPBS is associated with • Low levels of ODR • .29/100/day v. national mean .34 • Improved perception of safety of the school • reduced risk factor • Increased proportion of 3rd graders who meet state reading standard.

20. RCT Project TargetBradshaw & Leaf, in press • PBIS (21 v. 16) schools reached & sustained high fidelity • PBIS increased all aspects of organizational health • Positive effects/trends for student outcomes • Fewer ODRs (majors + minors) • Fewer ODRs for truancy • Fewer suspensions • Increasing trend in % of students scoring in advanced & proficient range of state achievement test

21. Impact Factor

22. Ask a Faculty member

23. Collaborative Problem Solving • Visit the website • http://www.livesinthebalance.org/ • What do I notice? • A canoe? • Advertising products for purchase • Lots of testimonials • Little bit of research (10 citations listed under research) • let’s take a closer, evaluative look at the research

24. Evaluating a Research Study • Abstract • Introduction & Literature Review • Research Questions • Method & Design • Subjects & Settings / Measures/ Procedures • Results • Discussion & Conclusions • References

25. Results Questions • Who were subjects? • How much like my participants? • Where was study conducted? • How much like where I work? • What measures were used? • Do I have similar data? • What outcomes were achieved? • Are expected outcomes similar

26. Collaborative Problem Solvingdata from CPS website on 6/18/12 Johnson, M., Ostlund, S., Fransson, G., Landgren, M., Nasic, S., Kadesjo, B., Gillberg, C., and Fernell, E. (2012).  Attention-Deficit/Hyperactivity Disorder (ADHD) with Oppositional Defiant Disorder (ODD) in Swedish Children:  An Open Study of Collaborative Problem Solving.  ActaPaediactrica, in press.  Ollendick, T. H. (2011). Invited Address: Effective Psychosocial Treatments for Emotional and Behavioral Disorders in Youth. University of Stockholm, Sweden.. Fraire, M., McWhinney, E., & Ollendick, T. (2011). The effect of comorbidity on treatment outcome in an ODD sample. In T. Ollendick (Chair), Comorbidities in children and adolescents: Implications for evidence-based treatment. Symposia presented at the 41st European Association for Behavioral and Cognitive Therapies, Reykjavik, Iceland. Halldorsdottir, T., Austin, K. & Ollendick, T. (2011). Comorbid ADHD in children with ODD or specific phobia: Implications for evidence-based treatments. In T. Ollendick (Chair), Comorbidities in children and adolescents: Implications for evidence-based treatment. Symposia presented at the 41st European Association for Behavioral and Cognitive Therapies, Reykjavik, Iceland. Epstein, T., & Saltzman-Benaiah, J. (2010). Parenting children with disruptive behaviors: Evaluation of a Collaborative Problem Solving pilot program. Journal of Clinical Psychology Practice, 27-40. Martin, A., Krieg, H., Esposito, F., Stubbe, D., & Cardona, L. (2008). Reduction of restraint and seclusion through Collaborative Problem Solving: A five-year, prospective inpatient study. Psychiatric Services, 59(12), 1406-1412. Greene, R.W., Ablon, S.A., & Martin, A. (2006). Innovations: Child Psychiatry: Use of Collaborative Problem Solving to reduce seclusion and restraint in child and adolescent inpatient units. Psychiatric Services, 57(5), 610-616. Greene, R.W., Ablon, J.S., Monuteaux, M., Goring, J., Henin, A., Raezer, L., Edwards, G., & Markey, J., & Rabbitt, S. (2004). Effectiveness of Collaborative Problem Solving in affectively dysregulated youth with oppositional defiant disorder: Initial findings. Journal of Consulting and Clinical Psychology, 72, 1157-1164. Greene, R.W., Biederman, J., Zerwas, S., Monuteaux, M., Goring, J., Faraone, S.V. (2002). Psychiatric comorbidity, family dysfunction, and social impairment in referred youth with oppositional defiant disorder. American Journal of Psychiatry, 159, 1214-1224. Greene, R. W., Beszterczey, S. K., Katzenstein T., Park, K., & Goring, J. (2002). Are students with ADHD more stressful to teach? Patterns of teacher stress in an elementary school sample. Journal of Emotional and Behavioral Disorders, 10, 27-37. NO studies have been conducted in school settings; all research is either with parents or in-patient clinical settings These studies to not evaluate effectiveness of CPS

27. Evaluating the Research studies

28. What does the research tell us? • So what do we know? • Based on 4 evaluation studies • All include children ages 12 or less (2008 study does not specify an age range; simply ‘school age’) • 2 are in inpatient psychiatric hospitals • 1 is an outpatient mental health clinic • 1 is a parent training program • # in school settings = 0 • The research tells us nothing about the efficacy of CPS in school settings

29. What does the research tell us? • Outcome measures • ODD Rating Scale (unpublished assessment created by the author) & improvement ratings from parent & therapist • Similar scores to parent training • Reductions in restraint & seclusion (Pre/Post) • Is this due to student behavior change or adult behavior change? • Eyberg CBI & Parent Stress Index (Pre/Post) • No studies directly measure changes in student behavior

30. Concerns • Only 4 research studies evaluating CPS in 4 years • 2 on parent training (1 individual training & 1 group training) • 2 in inpatient psyc facilities • Make sure research you are looking at takes place in settings that match your application • E.g. school settings v. treatment centers • 2 of 4 studies have been conducted by the author of the program • Concern if authors are benefiting financially from sale of the program

31. Research on CPS in Schools! Schaubman, A., Stetson, E., & Plog, A. (2011). Reducing Teacher Stress by Implementing Collaborative Problem Solving in a School Setting. School Social Work Journal, 35(2), 72-93. Abstract Student behavior affects teacher stress levels and the student-teacher relationship. In this pilot study, teachers were trained in CollaborativeProblemSolving (CPS), a cognitive-behavioral model that explains challenging behavior as the result of underlying deficits in the areas of flexibility/adaptability, frustration tolerance, and problemsolving. It was hypothesized that teacher stress would be reduced when teachers' understanding of the underlying causes of student behavior shifted to a framework of skills development, and they began using a proactive, positive approach to misbehavior (CPS), with the support of mental health consultation. Results showed a significant decrease in teacher stress, as measured by self-report. Further, discipline referrals were significantly reduced. Limitations of the study and implications for school mental health consultation are also discussed. (Contains 4 figures.)