Agreement Among Measures of Autism Spectrum Disorders in School-Age Children

1. Agreement Among Measures of Autism Spectrum Disorders in School-Age Children Presenters: Barbara Garcia-Lavin, Ph.D. & Irena Morin, M.S. Nova Southeastern University Fort Lauderdale, Florida

2. Presentation Objectives • Discuss the strengths and limitations of the current assessment methods used to diagnose/identify ASDs. • Select diagnostic instruments most likely to yield accurate diagnosis/identification of ASD in school-age children. • Explain the influence of intellectual and language functioning on ASD diagnostic profiles.

3. Autism Spectrum Disorders (ASD) • Autistic Disorder • Characterized by deficits in three distinct domains: • Social interaction • Communication • Unusual behaviors and patterns (i.e., restricted, repetitive, and stereotyped behaviors) • Asperger’s Disorder • Pervasive Developmental Disorder, Not Otherwise Specified (PDD-NOS)

4. Prevalence • Significant increases in reported cases of ASD all over the world in the past decade. • Centers for Disease Control and Prevention (CDC) Autism and Developmental Disabilities Monitoring Network indicate an average prevalence estimate of 9.0 per 1,000 children aged 8 years old or approximately 1 in 110 children identified with ASD (CDC, 2006).

5. School Prevalence • While the number of students, ages 6-21, served under IDEA, Part B, in the U.S. remained generally constant for most disability categories between Fall 1999 and Fall 2008, the number of students served who were classified with autism increase four fold from 66,043 in 1999 to 292,818 in 2008. https:www.ideadata.org/PartBData.asp

7. Identification of ASDs in schools • ASDs are frequently first identified in schools particularly in who may have less access to health care (Yeargin-Allsopp, et. al., 2003). • “School personnel are now more likely to be asked to participate in the screening and identification of students with ASD than at any other time in the past” (Wilkinson, 2010, p. 351)

8. Identification of ASDs in schools • Accurate identification of ASD is crucial to meeting educational and therapeutic needs of school-age children with ASD • Eligibility-Children are often not eligible for early intervention or Special Education services until an evaluation is completed and a formal diagnosis of an ASD has been made (Ventola et al., 2006). • Outcomes improvefor children with ASDs who receive intensive intervention services (Bryson et al., 2003)

9. Identification: Challenges • Heterogeneous symptom presentation • Other possible conditions causing disability must be considered/assessed for • Identification may be delayed (into school age) for students with higher levels of intellectual functioning and adequate language (Bryson et al., 2003 )

10. Identification Clinical Diagnosis = DSM-IV TR School System = IDEA Eligibility Uneven developmental profile Impairment in social interaction Impairment in verbal and/or nonverbal language or social communication generally before age three Restrictive, repetitive and/or stereotyped patterns of interests, behavior, or activities Demonstrates a need for special education Autism- onset of the following before age 3 • Impaired social interaction • Impaired communication • Restricted, repetitive, and stereotyped patterns of behavior, interests and activities Asperger’s-as above except • No significant delay in language (single words by 2 years, phrases by 3 years) • No delay in cognitive development or adaptive behavior PDD, NOS

11. Identification: Best Practices (Filipek, et al., 1999) Screening Diagnostic Evaluation Comprehensive assessment by experienced professional including: Thorough interviews with the child’s parents (family and child developmental and medical history and strengths/weaknesses) Direct behavioral observations Recommend use of several commonly used instruments found to have “moderate sensitivity and good specificity for autism” (p.470), including: ADI, ADOS, GARS-II, CARS, etc.) Measures of intellectual, language, sensorimotor, and adaptive functioning • Developmental surveillance and screening by physicians • Screening instruments (including Ages and Stages Questionnaire, the BRIGANCE Screens, the Child Development Inventories, and the Parents’ Evaluations of Developmental Status • Recommended against using: Denver-II (DDST-II) and Revised Denver Pre-Screening

12. Previous Studies • Although, many ASD measures have been found to demonstrate adequate sensitivity and specificity for autism, only few studies considered the agreement among various best practice diagnostic strategies and instruments used to identify ASD (Ventola et al., 2006; Papanikolaou et al., 2009). • However, these studies was have been carried out with toddlers only.

13. Current Study • Current study examining the agreement among the most commonly used assessments for Autism with children age 5-12 : • Autism Diagnostic Interview-Revised (ADI-R) • Childhood Autism Rating Scale (CARS) • Gilliam Autism Rating Scale (GARS) • Autism Diagnostic Observation Schedule-Generic (ADOS-G) • Compared against DSM-IV-TR Diagnostic Criteria (structured Interview)

14. Purpose of Study Objective of the current study was to examine the agreement between each of the four ASD diagnostic instruments (ADI-R, GARS-2, CARS, and ADOS) and a structured interview based on ASD diagnostic criteria outlined in the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, Text Revision (DSM-IV-TR; American Psychiatric Association, 2000)

15. METHOD

16. Participants • School-age children, between 5 and 12 years of age, of various ethnicities and socioeconomic status, with suspected or diagnosed ASD • Inclusion criteria included English- or Spanish- speaking • Exclusion criteria included co-morbid physical, vision, or hearing impairments. • Recruited from • UM-NSU Center for Autism and Related Disabilities (UM-NSU CARD) • other departments within the university • local area hospitals • Schools • community mental health centers • other healthcare professionals and agencies in the community

17. Procedure • In adherence with “best practice” parameters (Filipek et al., 1999), administration of the ASD diagnostic assessments took place in the context of a comprehensive evaluation, which included a parent interview and administration of measures of intellectual, language, and adaptive functioning to the child.

18. Procedure: Parent Interview • During the parent interview, consent was obtained, interview measures were administered, and information regarding the child’s current symptoms and developmental history that may indicate the presence of an autism spectrum disorder, based on DSM-IV-TR criteria (American Psychiatric Association, 2000) were collected.

19. Assessment • ASD diagnostic instruments • GARS- 2 (Gilliam, 2005) • ADI-R (Rutter, Le Couteur, & Lord, 2003) • CARS (Schopler, Reichler, & Renner, 1988) • ADOS-G (Lord et al., 2000) • Structured diagnostic interview based on DSM-IV-TR criteria • Intellectual Functioning • Kaufman Assessment Battery for Children-Second Edition (KABC-II; Kaufman & Kaufman, 2004) • Adaptive functioning • Vineland Adaptive Behavior Scales-Second Edition (VABS-2; Sparrow, Cicchetti, & Balla, 2005) • Language Abilities • Comprehensive Assessment of Spoken Language (CASL; Carrow-Woolfolk, 1999).

20. ASD Assessment Instruments (Allen, Robins, & Decker, 2008; Filipek, et al, 1999; Noland & Gabriels, 2004)

21. Childhood Autism Rating Scale(CARS; Schopler, Reichler, & Renner, 1988) • Most widely employed diagnostic instrument for autism. • Rates behavioral characteristics after observations or based on parent report of children on 15 subscales. • Designed to identify Autism, its severity (within normal limits to severely abnormal for age), and differentiate ASD from other developmental disorders (Mental Retardation in particular) • Strengths • Adequate reliability (test-rest reliability of .88 and criterion-related validity yielded a correlation of .80 [Prizant, 1992]) • Ease of administration • Weaknesses • Broad conceptualization of autism • No representative normative sample

22. Gilliam Autism Rating Scale Second Edition (GARS-2; Gilliam, 2005) • Rating scales and interview for parents • Use for identifying autism in individuals ages 3 through 22. • Items are based on the definitions of autism adopted by the Autism Society of America and the Diagnostic and Statistical Manual of Mental Disorders: Fourth Edition-Text Revision (DSM-IV-TR). • Helps estimate the severity of the child's disorder • Autism Quotient (AQ) - Classified on an ordinal scale from “Very Low” to “Very High” probability of autism • Correlation coefficient of .84 for test-retest reliability (Ward-Fairbanks, 2007) • Strengths • It is a norm-referenced Instrument (normed on a representative sample of 1,107 persons with autism from 48 states within the United States) • Ease of administration • Defined conceptual framework • Weaknesses • Informant report vs. direct observation • Clinical standardization sample limits interpretive value

23. Autism Diagnostic Interview-Revised (ADI-R; Lord, Rutter, & LeCouteur, 2003) • A standardized parent interview that collects information (93 questions) about the child’s early development and current functioning. • Interrater reliability ranging from .62 to .89 (Lord, Rutter, & Le Couteur, 1994, p. 681) • 4 areas: • Communication • Social interaction • Repetitive behaviors • Age of onset of symptoms • Strengths • Accounts for changes over lifespan • Comprehensive • Scoring based on clinician’s judgment of parent report • Weaknesses • Time consuming • Significant training required for use • Based solely on care-giver report • No representative normative sample • Algorithm likely to change based on new DSM-V criteria

24. Autism Diagnostic Observation Schedule (ADOS; Lord, Rutter, Di Lavore, & Risis, 2002) • Semi-structured assessment consisting of a standard set of interactions and activities that sample the social, communication, and play behaviors of children and adults suspected of having an ASD. • There are two sets of material for most tasks so that individuals of various ages and language levels can be tested. • Interrater agreement range between 90-100% across Modules • 4 Modules • Module 1 – Pre-verbal / Single Words • Module 2 – Phrase Speech • Module 3 – Fluent Speech (Child/Adolescent) • Module 4 – Fluent Speech (Adolescent/Adult) • Strengths • Observations based on typical interactions • “Gold Standard” • Weaknesses • No representative normative sample • Subjective ratings • Algorithm likely to change based on new DSM-V criteria • Based solely on observations during isolated period • Significant training required for use • Cost

25. Procedure: Testing & Feedback • Child participants underwent 2-4 testing sessions, depending on the child’s levels of motivation, attention, and concentration. • After all the data was collected for each parent and child participant, an evaluation report was completed within 4 weeks, and the participants’ parents were offered feedback about the findings during a 1-2 hour feedback session.

26. RESULTS

27. Data Analysis • Cohen’s kappa (k) was used to evaluate the pair-wise agreement among the four instruments and the DSM-IV-TR criteria • Sensitivity, specificity, and positive predictive values were calculated for the four measures against the DSM-IV-TR criteria, based on the structured interview

28. Sample • Thirty-seven school-age children, between 5 and 12 years of age • 31 male and 6 female participants in this study. • Mean age was 8.08 years (standard deviation = 1.95 years). • Race/Ethnicity • Hispanic 59.5% • White, Non-Hispanic 35.1 % • Bi-Racial 5.4 % • Language • English Only 70.3 % • Bilingual (English/Spanish) 29.7 %

29. Data Analysis

30. ASD Participants’ Intellectual and Language Functioning

31. Level of Agreement

32. Level of Agreement, cont. • There was moderate agreement for the identification of Autism Spectrum Disorders (Autism, Asperger’s Disorder, PDD-NOS) between the ADI-R and clinical judgment based on the DSM-IV TR structured interview, • There was also moderate agreement for the identification of an ASD between the ADOS and clinical judgment • The agreement between GARS-2 and the DSM-IV TR clinical interview was fair. • Agreement between clinical judgment and the CARS was slight at best.

33. Sensitivity, Specificity & PPV • Sensitivity-the probability that a child with the disorder will screen positive for that disorder on the assessment • Specificity-the probability that a child without the disorder will screen negative for the disorder on the assessment • Positive Predictive Value (PPV)-the power of an instrument to identify a disorder. Instruments are expected to have high PPV with a known high-risk group • The most clinically useful instruments for identifying the presence of a disorder are those with the lowest false negatives (high sensitivity) as these are cases with the disorder that go unidentified

34. Sensitivity, Specificity & PPV *sensitivity and specificity levels of 80% or higher have been recommended for ASD instruments (Coonrod & Stone, 2005)

35. Sensitivity: • ADOS demonstrated best sensitivity to DSM-IV TR characteristics of ASD (true positives) • ADI-R accurately identified 73.9% of the true ASD cases (true positives) • CARS & GARS-2 accurately identified less than 40% of cases (missed greater than 60% of cases with ASD)

36. Specificity: • ADI-R, ADOS, and GARS-2 accurately identified 78.6% of non-ASD cases (true negatives) • CARS identified children as having ASD when they did not 50% of the time (as good as guessing)

37. DISCUSSION

38. Limitations • Small sample size limits generalizability • Skewed sample • previous or suspected diagnosis of ASD • Many higher-functioning (IQ & language) • Interview limitations

39. Future Directions • Need more studies of ASD instruments to inform the practice of accurately identifying ASDs in school age children • Including newer instruments: Autism Spectrum Rating Scales (ASRS; Goldstein & Naglieri, 2009) • Only measure standardized on large (2,560) normative population

40. Implications for School Psychology Practice • Test selection • ADOS and ADI-R had the best clinical utility in identifying actual cases of ASD • CARS and GARS-II may leave many students with ASD unidentified • Use these instruments within the context of comprehensive evaluations - including assessment of intellectual & language functioning • IQ and language assessments provide a “frame” from which to determine whether the student’s social/communication functioning is developmentally delayed (Klin et al., 2005) • May also inform ASD instrument selection

41. Implications for School Psychology Practice • School personnel training • Instrument administration & interpretation • Changes in diagnostic classifications • Research • Instruments: Reliable, valid, adequately normed • Changing Models : ASD factors

42. Possible Changes to Diagnostic Classification in DSM-V (http://www.dsm5.org/ProposedRevisions/Pages/proposedrevision.aspx?rid=94#) • Single diagnostic category : Autism Spectrum Disorder ASD • Defining the ASD “spectrum” • Three groups of symptoms reduced to two • Explicitly including unusual sensory behaviors • Including a modifier for genetic and medical conditions that may demonstrate as autism • Including a consideration of severity with the diagnosis • Level 1 “Requiring support” • Level 2 “Requiring substantial support” • Level 3 “Requiring very substantial support”

43. Changing Models of ASD (Goldstein, 2011) • DSM-IV-TR : 3-factors • Social • Communication • Unusual/restricted/repetitive behavior • Gillberg’s 2-factor Model (possible direction of DSM-V) • Social/Communication • Unusual/restricted/repetitive behavior • Goldstein’s 3-factor Model • Social/Communication • Unusual/restricted/repetitive behavior • Impaired self-regulation

44. References Allen, R., Robins, D.L., & Decker, S. (2008). Autism spectrum disorders: Neurobiology and current assessment practices. Psychology in the Schools, 45 (10), 905-917. American Psychiatric Association. (2000). Diagnostic and statistical manual of mental disorders (4th ed. text revision). Washington, D.C.: Author. Bryson, S. E., Rogers, S. J., & Fombonne, E. (2003). Autismsprectrumdisorders: Earlydetection, intervention, education, and psychopharmacological management. Canadian Journal of Psychiatry, 48, 506-516. Carrow-Woolfolk, E. (1999). Comprehensive Assessment of Spoken Language (CASL). Circle Pines, MN:  American Guidance Service, Inc. Centers for Disease Control and Prevention. (2006, May 4). Parental Report of Diagnosed Autism in Children Aged 4–17 Years, United States, 2003-2004. Retrieved March 5, 2008, from http://www.cdc.gov/od/oc/media/transcripts/ASDMMWRfactSheet.pdf Coonrod & Stone, 2005 Filipek, P. A., Accardo, P. J., Baranek, G.T., Cook, E. H., Jr., Dawson, F., Gordon, B., et al. (1999). The screening and diagnosis of Autistic Spectrum Disorders. Journal of Autism and Developmental Disorders, 29(6), 439-484. Gilliam, J. E. (2005). Gilliam autism rating scale (2nd ed.; GARS-2). Austin, TX: Pro-Ed Goldstein, S. (2011, January). The changing face of autism: New data, new ideas, and the Autism Spectrum Rating Scale [PowerPoint slides]. Paper presented at the Continuing Education Workshop Series at Nova Southeastern University, Fort Lauderdale, FL. Goldstein, S. & Naglieri, J. A. (2009). Autism Spectrum Rating Scales (ASRS). North Tonawanda, NY: MHS Kaufman, A. & Kaufman, N. (2004). The Kaufman Assessment Battery for Children-Second Edition. Circle Pines, MN: AGS Publishing Klin, A., Saulinier, C., Tsatsanis, K., & Volkmar, F. R. (2005). Clinical evaluation in autism spectrum disorders: Psychological assessment within a transdisciplinary framework. In F.R. Volkmar, R. Paul, A. Klin, & D. J. Cohen (Eds.), Handbook of Autism and Pervasive Developmental Disorders: Vol.2. Assessment, intervention, and policy (3rd ed., pp.772-798). Hoboken, NJ: Wiley.

45. References Lord, C., Rutter, M., Di Lavore, P. C., & Risis, S. (2000). Autism diagnostic observation schedule. Los Angeles, CA: WesternPsychological Services. Noland, R. M. & Gabriels, R. L. (2004). Screening and identifying children with autism spectrum disorders in the public school system: The development of a model process. Journal of Autism and Developmental Disorders, 34, 265-277. Papanikolaou, K., Paliokosta, E., Houliaras, G., Vgenopoulou, S., Giouroukou, E., Pehlivandis, A., Tomaras, V., Tsiantis, I. (2009). Using the autism diagnostic interview-revised and the autism diagnostic observation schedule-generic for the diagnosis of autism spectrum disorders in a Greek sample with a wide range of intellectual abilities. Journal of Autims and Developmental Disorders, 39, 414-420. Rutter, M., Le Couteur, A., & Lord, C. (2003). Autism diagnostic interview-revised (ADI-R). Los Angeles, CA:Western Psychological Services. Schopler, E., & Reichler, J., & Renner, B. (1988). The Childhood Autism Rating Scale (C.A.R.S.). Los Angeles: Western Psychological Services. Sparrow, S. S., Cicchetti, D. V., & Balla, D. A. (2005). Vineland Adaptive BehaviorScales, Second Edition (Vineland–II). Circle Pines, MN: AGS. Ventola, P. E., Kleinman, J., Pandey, J., Barton, M., Allen, S., Green, J., Robins, D., & Fein, D. (2006). Agreement among four diagnostic instruments for Autism Spectrum Disorders in toddlers. Journal of Autism and Developmental Disorders, 36, 839-847. Wilkinson, L. A.(2010). Facilitating the identification of autism spectrum disorders in school-age children. Remedial and Special Education, 31, 350-357. Yeargin-Allsopp, M., Rice, C., Karapurkar, T., Doernberg, N., Boyle, C., & Murphy, C. (January, 2003). Prevalence of Autism in a US metropolitan area. The Journal of the American Medical Association, 289(1), 49-55.