By James L. McDougal, Psy. D., Achilles N. Bardos, Ph.D., & Scott T. Meier, Ph.D.

1. TM By James L. McDougal, Psy. D., Achilles N. Bardos, Ph.D., & Scott T. Meier, Ph.D.

2. 3-tier Behavioral Progress Monitoring System for screening and assessing changes in response to behavior intervention follows the RTI model School-aged children and adolescents (5-18) 2 main components: BIMAS Standard & BIMAS Flex Multi-informant tool with forms for teachers, parents, youth, and clinician web-based data management and reporting system. What is the BIMAS?

3. Presentation Overview School Mental Health Services BIMAS: Rationale within an RTI context BIMAS: Theory and Structure BIMAS: Development and Psychometric Properties BIMAS: Applications in a School District BIMAS: Web-based Interface and Reports. Q and A

4. Background & Development

5. Why is Behavioral/Emotional Screening Crucial? • Students with behavioral/emotional problems: - lower grades - poorer reading skills - drop-out rate > 50% - worst social & academic outcomes of any disability group (Bradley, Dolittle, & Bartolotta, 2008) • Loss of instructional time: teacher intervention & student disruption (Arnold, 1997) • Early ID – prevents pervasive problems(Kern, Hilt-Panahon, & Sokol, 2009) • Systematic universal screening  ID children at-risk for E/B problems(Kamphaus & Reynolds, 2007)

6. Early Identification Early identification and intervention with children who are at risk for EBD appear to be the “most powerful course of action for ameliorating life-long problems associated with children at risk for [EBD]” (p. 5). (Hester et al., 2004) Younger children are more likely to be responsive to and maintain the positive outcomes from early prevention and intervention programs (Bailey, Aytch, Odom, Symons, & Wolery, 1999)

7. RTI and BehaviorRationale and need for Universal Behavior Screening Strong link between behavior/emotions and academic performance How do we define health? • Schools screen for vision, hearing, speech, and academic achievement • Behavior/Emotional screening occurs in less than 2% of districts across the U.S.

8. RTI and BehaviorRationale and need for Universal Behavior Screening Screeners for children are mostly done in primary care settings; we miss many children for early identification and intervention (Pagano et al., 2000). Screening measures for children are frequently focused on one disorder to the exclusion of others, an approach that neglects large numbers of children who have problems other than the target condition (August et al., 1992; Taylor et al., 2000; Matthey & Petrovski, 2002).

9. RTI and BehaviorRationale and need for Universal Behavior Screening Early problem identification allows for prevention & intervention Negative impact of problems can be minimized Only 15-20% of children with emotional/behavioral problems receive mental health services in the U.S (Ringel & Sturm, 2001)

10. RtI and BehaviorRationale and need for Universal Behavior Screening Teachers accurately identify young children at high risk of academic and behavioral problems related to school adjustment with a great deal of accuracy(Taylor et al., 2000). Schools are the ideal setting for large-scale, broad based mental health screening of children and adolescents (Wu et al., 1999).

13. Tier 1 Universal Level Students without serious problem behaviors (80-90%) Use BIMAS to Screen How to use BIMAS within the RtI Framework

14. How to use BIMAS within the RtI Framework Tier 2 • Targeted Level • Students at risk for problem behaviors (5-15%) • Use BIMAS to monitor & assess response to intervention/treatment

15. How to use BIMAS within the RtI Framework Tier 3 • Intensive Level • Students with chronic/intense problem behaviors(1-7%) • BIMAS Data for decision making: Treatment planning, Special education eligibility, IEP review, program evaluation

16. Applications of BIMAS • Screening- To detect students in need of further assessment and to identify their respective areas of need. • Student Progress Monitoring- To provide feedback about the progress of individual students or clients. • Program Evaluation- To gather evidence that intervention services are effective.

17. Instrument Structure The BIMAS consists of two levels • Standard • Flex

18. BIMAS Standard • 34 change-sensitive items • Can be used as a brief screener, treatment monitoring or program evaluation tool • 4 forms: Teacher, Parent, Self (U.S. normed) & Clinician

19. Intervention Item Selection Rules(IISRs; Meier 1997, 1998, 2000, 2004) • BIMAS: developed using empirically derived model for designing change-sensitive measures to assess RTI: IIRSs • “State” scale as opposed to a “Trait” scale • BIMAS developed using clinical & school samples in field settings (rare combo in psychotherapy research) • Identified constructs that change as a result of emotional and behavioral intervention

20. IISRs (cont’d) • Be grounded in theory- to provide for interpretation and minimize the effects of chance; • Be aggregated across individuals- to reduce the effects of random error; • Not evidence ceiling and floor effect; • Evidence change after a psychosocial intervention; • Change in the theoretically expected direction; • Evidence change relative to control and comparison groups; • Show no difference at pre-intervention; • Have no relation to relevant systematic errors; and • Be subject to cross-validation studies. (Meier 1997, 1998, 2000, 2004)

21. Rule 1. Items Based on Theory Review existing research and theory Select pool of items from a variety of theories Ensure large Item Pool Select those thought to be influenced by intervention.

22. Rule 2. Aggregate Items Aggregation across individuals decreases error Increases chances of showing intervention effect Individual response used to inform theory

23. Rule 3. Avoid Ceiling/Floor Effects Mean item scores should not be at top or bottom of possible range Drop items with means 2 or more SD’s out

24. Rule 4. Detect Change Post-Intervention Significance vs. effect size Retain subset of items showing pre/post change Then filter those that demonstrate treatment group change

25. Rule 5. Change in the Expected Direction Retain items showing pre/post TX change in expected direction Items changing in opposite direction: retained to investigate negative effects

26. Rule 6. Change Relative to Comparison Group Compare to no TX control for: developmental and maturation effects, and TX effects Do items differentiate between TX groups? Retain items: pre/post difference in TX group Drop items with pre/post difference in control group

27. Rule 7. No Difference at Pre-Test Drop items showing significant difference between samples drawn from the same population Random assignment best

28. Rule 8. Remove Systematic Error Drop items reflecting systematic error (e.g., social desirability --Marlowe Crowne Social Desirability Scale)

29. Rule 9. Cross Validate Repeat steps 3-8 with new samples from same population Emphasize theoretically predicted change items

30. BIMAS Standard Behavioral Concern Scales: • Conduct— anger management problems, bullying behaviors, substance abuse, deviance • Negative Affect— anxiety, depression • Cognitive/Attention— attention, focus, memory, planning, organization Adaptive Scales: • Social — social functioning, friendship maintenance, communication • Academic Functioning— academic performance, attendance, ability to follow directions

31. BIMAS Flex • List of specific behavioral items corresponding to each Standard item for progress monitoring • User can select items based on elevated Standard scale score for an individual student — customized treatment goals • Ability to make notes to describe specific behaviors, response to services, or to add other comments • Teacher, Parent, Self and Clinician forms

32. Negatively worded: Argued with peers Argued with teachers Argued with parents Argued with siblings Talked back to parents Talked back to teachers Physically hurt peers Physically hurt parents Physically hurt teachers Physically hurt siblings Threatened peers Threatened teachers Threatened parents Threatened siblings Positively worded: Showed regret after a fight Was respectful to adults Walked away from a fight Prevented a fight Stopped an argument Found a positive outlet for frustration Avoided a verbal confrontation BIMAS Flex ExampleStandard Item:Fought with others (verbally, physically, or both) Or…custom create your own!

33. Psychometric Properties

34. Large Normative Sample Total Sample N = 4,855 Teacher N = 1,938 Parent N = 1,938 Self-Report N = 1,050 Normative N = 1,400 Clinical N = 538 Normative N = 1,400 Clinical N = 467 Normative N = 700 Clinical N = 350

35. Age x Gender Distribution: Normative Sample

36. Race/Ethnicity Distribution Highly comparable to the most recent U.S. Census (Weighted N’s)

37. Geographic Region Distribution • Highly comparable to the most recent U.S. Census (Weighted N’s)

38. Parental Education Level • Highly comparable to the most recent U.S. Census (Weighted N’s)

39. Norm Groups Age groups based on: age group mean differences developmental settings (K-12) Combined-gender norms (♂ flagged as often as ♀) Gender-specific norms available as option

40. Development of Standard Scores T-scores reported on every scale Raw scores  percentile rank to preserve the shape of original distribution (behavior: not normal curve but Original percentile scores smoothed by imposing empirical percentiles Final smoothed percentile scores converted to standard T-scores (M = 50; SD = 10)

41. Interpretation of BIMAS Scores : Scale-level Descriptors Behavior Concern Scales (↑ scores = ) High Risk:T = 70+ Some Risk:T = 60-69 Low Risk: T < 60 Adaptive Scales (↑ scores = ) Strength: T = 60+ Typical: T = 41-59 Concern: T≤ 40

42. Interpretation of BIMAS Scores: Item-Level Descriptors(Useful in Individualized Intervention Design) Likert Scale 0 = Never; 1 = Rarely; 2 = Sometimes; 3 = Often; 4 = Very Often Item Descriptors: Behavior Concern Scales No Concern Mild Concern = M + 1SD; ≥ 75th percentile Concern = >1 SD; ≥ 84th percentile Adaptive Scales Concern = ≤10th percentile Mild Concern = 10th - 20th percentile; M - 1SD Fair Positive = ≥ 75th percentile; M + .67SD

43. Psychometric Properties Reliability • Internal Consistency • Test-Retest (stability) Validity • Content • Construct • Scale structure • Screening accuracy • Progress monitoring

44. Reliability • Internal Consistency • The extent to which all items on the same scales measure the same content • Cronbach’s Alpha (ranges from 0.0 to 1.0; higher = more reliable) • Good range: α = .80 to .89 • Clinical cases added to the Normative Sample for reliability analyses (85% Normative; 15% Clinical) • To increase variability in the data • Most schools: mixture of students with & w/o diagnosis

45. Internal Consistency Cronbach’s Alpha

46. Reliability (cont’d) • Test-Retest Reliability • Refers to the stability of test scores when an assessment is administered on two or more occasions (without intervention) • Pearson’s Correlation (r) between Time 1 and Time 2 BIMAS scores • 2-to-4 week interval • Ranges from -1 to +1; higher = more reliable; good range: .7 or higher

47. Test-Retest Reliability Coefficients All rs significant, p < .001.; A 2-4 week interval (non-clinical sample; no intervention in between)

48. Across-Informant Correlations Correlation between parent & teacher ratings Correlation between self-report & parent/teacher Are the behaviors assessed by the BIMAS consistently detected by raters in different settings? (Diff informant: Diff observation context) Parent to Teacherr: range = .79 - .86 Parent to Self r: range = .59 - .69 Teacher to Selfr: range = .54 - .59

49. Validity

50. Validity The validity of a test refers to the quality of inferences that can be made by the test’s scores (i.e., how well does the test measures and how well are theclaims it makes for its use and applications supported by empirical evidence).