Measurement Development and Inclusion Criteria: Developing Meaningful Standards

1. Measurement Development and Inclusion Criteria: Developing Meaningful Standards Steven W. Evans, Christine Brady, Lee Kern, Christiana Andrews and the CARS Research Team

2. Defining the Population • No consistent standard that is applied across districts that would allow us to rely on school district determined labels. • ED • OHI (ADHD) • LD • Other • Criteria • High school students with social, emotional behavior problems • Cognitive ability in the normal range • Significant impairment in school functioning • Placement in special education

3. Key Aspects of Definition • Social, emotional or behavior problems • Broad band measure • At risk range • Significant impairment in school functioning • Limited assessment tools for use with high school aged students • Academic Competence Evaluation Scales (ACES):73 items with norms for adolescents • Classroom Performance Scale: 24 items with no norms or factors • Placement in special education • Cognitive ability in the normal range • FSIQ estimate equal to or greater than 75 according to most recent evaluation in school records

4. Research Plan • Collect CPS and BASC data for students in the pilot study with open eligibility criteria • Compare data to BASC norms • Gather normative data with the CPS • Identify factors (if any) • Establish norms • Compare data to findings from CPS • Identify items/factors that assess most problematic areas and issues for which we have interventions • Determine eligibility criteria

5. BASC Means & SD for CARS Participants

6. BASC Externalizing

7. BASC Internalizing

8. Number Who Met BASC Criteria

9. Classroom Impairment: Method • CARS investigators were asked to recruit staff at high schools for the study • Measurement packets were created for each teacher and placed in their mailbox • Packets contained a cover letter, student selection form, teacher demographic sheet, and six copies of the classroom performance survey (CPS), disruptive behaviors disorders scale (DBD), and impairment rating scale (IRS) • Teachers were asked to complete the CPS, DBD, and IRS for three randomly selected boys and girls from their first period classroom using the student selection form

10. Method • Those who completed packets were entered in a drawing to win one of three monetary prizes ($75, $50, $25) • Drawing were conducted within school to increase participation • Teachers were allowed two weeks to complete and return the packets

11. Classroom Performance Survey (CH.A.D.D., 1996) • Teacher report • 20 items on a 5 point Likert-type scale (1 = “always” to 5 = “never”) • Items regarding homework, note-taking, test-taking, interpersonal skills, communication, and attention behaviors

12. Teacher Demographics Overall Sample139 teachers

13. Student Demographics Overall Sample

14. Analysis Plan • Given our sample size (139 teachers and 833 students), we decided to run an exploratory factor analysis with a portion of the sample (263 students) and a confirmatory factor analysis with the others (570 students) • Data were randomly assigned to an analysis by teacher within school

15. Exploratory Factor Analysis • Results indicated the presence of two factors accounting for 78.6% of the variance • Factor 1: Academic (14 items) included items regarding homework, tests, attending to instruction, and completion of longer term assignments • Factor 2: Interpersonal (6 items) included items regarding relation to peer, relation to teachers, accepting assistance, and respecting property

16. Confirmatory Factor Analysis • Two models were compared: the two factor model found in the EFA and a one factor model • Results indicated some degree of fit, but not enough to justify the use of either model • Items on the CPS should be treated individually

17. Mean Data on the CPS • Scores on every item included the full range of scores (1 to 5) • The two most commonly endorsed problematic behaviors were recording assignments (M = 2.68, SD = 1.36) and completing homework assignments (M = 2.65, SD = 1.26) • The two least endorsed items were respecting property (M = 1.69, SD = 1.02) and relating positively to teachers (M = 1.84, SD = 1.06)

18. CPS Correlations All correlations were significant at p = .001

19. CPS Correlations All correlations were significant at p = .001

20. CPS Descriptives

21. CPS Descriptives

22. Average Z-Score by Item

23. 4 - Records Assignments Consistently

24. 6 – Completes Long-Term Assignments

25. 7 – Attends to Instructions in Class

26. 12 – Takes Notes in Class to Study

27. 14 – Completes Assigned Work with Accurate Computation/Detail

28. 18 – Relates Positively to Peers

29. 19 – Communicates Own Needs or Asks Questions

30. 20 – Accepts Assistance when Needed or Offered

31. How are Students Distributed?

32. Number of Times Teacher Endorsed a z-score of 0.5 or Greater on Any Item

33. Establishing Initial Criterion • Who are students with means better than normative sample? • Who are students with very few items rated half of a sd worse than normative sample? • Talk to CARS clinicians about students. • Criterion – 2 or fewer items rated less than .5 z-score • Best match with clinicians descriptions • Possible to have better than average mean with some areas of significant impairment

34. Criteria Applied to Students Participating in Pilot • We have parent BASC and CPS data for 32 participating students at two sites • Met neither criteria – 2 • Met one criteria – 19 • Met both criteria - 11

35. Implications • Possible reasons some students did not meet academic impairment criteria • In a setting where nothing is being expected of them • Special education teachers tend to rate problems as less severe than regular education teachers • Other consideration • Many of the most problematic students are not attending public high schools • Modifications • Consider BASC reports from both parent and special education teacher (use “or” criteria) • Gather CPS data from all teachers

36. Criteria • Social, emotional or behavior problems • 1 sd from mean on either internalizing or externalizing • Significant impairment in school functioning • One-half sd from mean on 3 or more items on the CPS • Cognitive ability within average range (broadly defined) • IQ > 75 • In special education • In special education for something other than a Pervasive Developmental Disorder

37. Next Steps • Keep eligibility criteria during next phase of pilot broadly defined (6 sites; 8 schools) • In special education for social, emotional or behavior problems • Exclude Pervasive Developmental Disorders and IQ less than 75 • Adjust data collection and use “or” criterion • BASC – gather from parent and primary special education teacher • CPS – gather from all teachers • Analyze eligibility criteria in relation to students’ responses to interventions and other evaluation data