1 / 60

DARA Research and Next Steps

DARA Research and Next Steps. Cara Cahalan-Laitusis & Linda Cook Educational Testing Service. Presentation. Experimental Study of Read Aloud Psychometric Research Research Plans for Year 3 Psychometric analysis of experimental data Tailored Test Design Cognitive labs

urielle-clayton
Download Presentation

DARA Research and Next Steps

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. DARA Research and Next Steps Cara Cahalan-Laitusis & Linda Cook Educational Testing Service

  2. Presentation • Experimental Study of Read Aloud • Psychometric Research • Research Plans for Year 3 • Psychometric analysis of experimental data • Tailored Test Design • Cognitive labs • IEP Decision Making for read aloud

  3. Differential Boost from Read Aloud (Non-disabled vs. RLD) • Is there a Differential Boost from read aloud? • How well do test scores (standard, audio, and fluency) predict variance in teacher ratings of reading comprehension? • Are teachers’ able to predict which students will benefit from read aloud?

  4. Prior Research • No Differential Boost • Kosciokek & Ysseldyke (2000)- Small sample size (n=31) • Meloy, Deville, and Frisbie (2002) – Between subjects design (n=260, 76% non-disabled, randomly assigned to audio or standard) • McKevitt & Elliott (2003)-Small sample size (n=39) • Differential Boost • Crawford and Tindal (2004)-(n=338, 78% non-disabled) • Fletcher, et. al (2006)-Between subjects design (randomly assigned to audio or standard). Sample included 91 Dyslexic (poor decoder) and 91 average decoders

  5. Data Collected • GMRT Forms S and T • Extra Time • Extra Time with Read Aloud via CD • 2 Fluency Measures • WJ Reading Fluency • Test of Silent Word Reading Fluency • 2 Decoding Measures (4th grade only) • WJ Letter Word ID • WJ Word Recognition • Demographic and Survey Data

  6. Sample • 1170 4th Graders • 522 Students with RLD • 648 Students without a disability • 855 8th Graders • 394 Students with RLD • 461 Students without a disability

  7. Design

  8. Means for Grade 4

  9. Means for Grade 8

  10. Scores by RLD and Grade

  11. Is there a Differential Boost from read aloud? Repeated Measures ANOVA and ANCOVA • Dependent Variables: • GMRT “Standard” • GMRT Audio • Independent Variables: • Disability Status (RLD vs. NLD) • Form/Order (STSA, STAS, TSSA, TSAS) • Covariate: Decoding and Fluency Measures

  12. ANOVA Findings • Yes, students with reading-based learning disabilities have larger gains (on average) from read aloud than students without disabilities • Finding consistent at both grades 4 and 8, but boost is larger at grade 4 • Controlling for Decoding and/or Fluency as a covariate did not alter findings

  13. How well do test scores predict reading comprehension? • Multiple regression analyses to determine how much variance in teachers’ rating of reading comprehension (5-point scale) were predicted by three test scores: • Standard • Audio • Fluency

  14. Regression Findings • Audio score does not significantly predict variance in Teacher Ratings of Reading Comprehension (beyond standard and fluency) for Grade 8 RLD • Audio score adds to prediction of reading comprehension (beyond standard and fluency scores) for three groups (NLD grade 4, NLD grade 8, and RLD grade 4), but incremental change is small

  15. Are teachers’ able to predict which students will benefit from read aloud? Analyses: • Analysis of variance in boost by teacher predictions • Cross-tabulations of teacher ratings by degree of boost (more than on SEM, less than one SEM, neither)

  16. Accuracy of Teacher Prediction For this study each student took a reading comprehension test that was read aloud by a CD player and another reading comprehension test that they read to themselves. Which test do you predict the student did better on? Ⓐ Test read aloud by CD player Ⓑ Test the student read to themselves Ⓒ No difference

  17. Findings from Teacher Predictions • ANOVA indicated that on average teachers were able to predict score gain from audio at grade 4 but not grade 8 • At the individual level teachers accurately predicted if a student would benefit from the audio version about 35% of the time and were completely wrong about 5% of the time

  18. DARA Psychometric Research • Purpose of psychometric research: To help us understand how an examinee's disability or the accommodations he or she receives impacts the psychometric properties of a reading test

  19. Results of This Year’s Psychometric Analyses • Psychometric Analyses • Factor analyses • Differential item functioning analyses • Populations • Students with learning disabilities who took the test with and without accommodations • Test • Grade 4 and grade 8 English-language arts (ELA) assessment • Focus • Determine if the test measures the same constructs for • Examinees without disabilities • Examinees with disabilities who took the test with and without accommodations

  20. STAR ELA Grade 4 and Grade 8 Summary Statistics

  21. Factor Analyses of ELA Assessment • Exploratory analyses (separately in each group) • how many factors • Confirmatory (multi-group) • Establish base-line model • Confirm number of factors needed to describe data across all groups

  22. Differential Item Functioning (DIF)Analyses • The purpose of this study was to examine differential item functioning on the same English-Language Arts assessment that was used for the factor analyses • “DIF is a statistical observation that involves matching test takers from different groups on the characteristic measured [by the test] and then looking at performance differences on an item.” (Sireci, 2006)

  23. Method • Mantel-Haenszel Categorization—3 Levels • A  Negligible DIF • B  Slight to Moderate DIF • C  Moderate to Large DIF • Directions of DIF Flags • - Favors reference group • +  Favors focal group

  24. Comparisons Made in the Study

  25. DIF Categories ELA Grade 4 LD Without Accommodations Easy Difficult Favors Students Without Disabilities Favors LD Students

  26. DIF Categories ELA Grade 4 LD With Accommodations (IEP/504) Easy Difficult Favors Students Without Disabilities Favors LD (IEP/504)

  27. DIF Categories ELA Grade 4 LD With Accommodations (Read-Aloud) Easy Difficult Favors Students Without Disabilities Favors LD (Read-Aloud)

  28. DIF Categories ELA Grade 8 LD Without Accommodations Easy Difficult Favors Students Without Disabilities Favors LD Students

  29. DIF Categories ELA Grade 8 LD With Accommodations (IEP/504) Easy Difficult Favors Students Without Disabilities Favors LD (IEP/504)

  30. DIF Categories ELA Grade 8 LD With Accommodations (Read-Aloud) Easy Difficult Favors Students Without Disabilities Favors LD (Read-Aloud)

  31. Interpreting the Results of the DIF Study • Grade 4 • 1 C DIF item, 8 B DIF items • Grade 8 • 1 C DIF item, 6 B DIF items • Majority of flagged items were reading items that favored students who took test with read-aloud accommodation • Consistent with Factor Analysis Results

  32. Next Steps • Psychometric Research • Examination of Tailored Testing • Cognitive Labs • IEP decision making

  33. Psychometric Research

  34. Plans for Next Year’s Psychometric Analyses • Psychometric analyses • Factor Analyses • Differential item functioning analyses • Populations • Students with learning disabilities who took the test with and without an audio accommodation • Test • Gates-McGinitie Reading Test • Focus • Aid in interpretation of results of differential boost study • Increase understanding of impact of disability and audio accommodation on reading test scores

  35. Factor Analyses We Plan to Carry Out • Aid in interpretation of results of differential boost study • Compare factor structures for students without disabilities who took test with and without accommodation • Compare factor structures for students with disabilities who took test with and without accommodation

  36. Factor Analyses We Plan to Carry Out • Increase understanding of impact of disability and accommodation on reading test scores • Compare factor structures of test given to examinees with and without disabilities under standard conditions • Compare factor structure of test given to examinees with disabilities who take test with accommodations and examinees without disabilities who take test without accommodations

  37. Purpose of Doing DIF and DDF Analyses on Data From the Differential Boost Study • Aid in interpretation of results of differential boost study • Increase understanding of impact of disability and accommodation on reading test scores

  38. Possible Comparisons for DIF Analyses

  39. Procedures for Analyzing Data • Differential Item Functioning: Mantel-Haenszel • Differential Distractor Analysis: Standardized Distractor Analysis

  40. Two Staged Tailored Testing

  41. Operational Data

  42. GMRT Data

  43. DARA Tailored Testing Model • Two (or three) stages of testing • Students subtests on stage 2 are determined by performance on routing test administered in stage 1 • Ideally computer administered but can be paper administered • Some parts could be individually administered (e.g., decoding) if only a few students are routed into a decoding measure and this format reduces the number of students receiving individualize testing accommodations (e.g., read aloud by human)

  44. Advantages of Model • Score is more reliable estimate since items are targeted to students ability level • Students may feel less frustrated if they can do some of the items on the routing test • Teacher receives more information on low performing students strengths and weaknesses • Fundamental Skills and Comprehension are not confounded for students with poor fundamental skills (some LD) or poor comprehension (some LD and ELL) • Growth can be more accurately measured in students working significantly below (or above) grade level

  45. Disadvantages of Model • Requires computer administration or teacher scoring of items after stage 1 • Students who are routed to fluency test may be embarrassed • Routing decision is made before test is scaled or standard setting is completed • Design could route more that 2% of students to modified test

  46. Questions for Year 3 • How many items (and of what difficulty) are needed for an accurate routing test? • Can we equate the audio extended and standard extended using the routing test? • What portion of students would be routed to fluency measure and what portion would be routed to decoding?

  47. Questions for Year 3 (continued) • Are the 2 alternate routes highly correlated with the standard administration? • What is the impact audio, fluency, and decoding scores on total test score. • If student is not a fluent reader should the total test score be non-proficient? • Is the routing test accurate for all students? • Do some students do better on hard items? • Do some students having trouble with the first few items on the test?

More Related