Student Characteristics Study

1. Student Characteristics Study October 13, 2006 Martha Thurlow and Ross Moen National Center on Educational Outcomes University of Minnesota www.nceo.info

2. Demands of a Research Paradigm for Validating Accessible Reading Assessment Know it when we see it? PART 1: Why Study Student Characteristics?

3. Attribute X Treatment Interaction Instruction: Trait X Treatment Interaction Aptitude X Treatment Interaction Assessment: Bias reduction studies Accommodations studies Universal Design of Assessment studies Development and validation versus use As pure as possible during development and validation Practical issues become more important during implementation One Classic Research Paradigm

4. Interaction Effect - Absolute Typical Assessment Accessible Assessment

5. Interaction Effect – Relative“Differential Boost” Typical Assessment Accessible Assessment

6. A Key Question:Who is the Target Population? = ? = ?

7. Group (and subgroup) heterogeneity Accommodations should match needs Universal Design of Assessment incorporates specific needs Distinguish proficient from non-proficient Why not just “All students with disabilities” – or even “this disability group”?

8. Interaction Effect – More Detailed Typical Assessment Accessible Assessment

9. Target Student Characteristics – Particular Students with Disabilities Relative Absolute (LAMS) Less Accurately Measured Students Inaccurately Measured Students = (MAMS) More Accurately Measured Students Accurately Measured Students =

10. Student Characteristics Studies Identify LAMS Understand LAMS attributes Test Characteristics Studies Find and use assessment techniques that meet the needs of LAMS Three Key Goals

11. Research Rationale Research Questions Research Procedures PART 2: Pilot Student Characteristics Study Design

12. Desire to identify LAMS efficiently Teachers would be an efficient resource if they can provide at least a first screen They have access to information They complain about testing LAMS But their judgment is questioned Brief supplementary examinations might be might be able to refine teacher data Research Rationale

13. How well can teachers identify LAMS? How well can teachers document their rationale? How well can brief supplemental examination confirm or disconfirm teacher judgments? Research Questions

14. Teachers complete a LAMS nomination questionnaire 4th and 8th grade reading, English/language arts, special education Researchers interview teachers Researchers examine students 4th or 8th grade, native speakers of English, took general state reading assessment, teacher nominated as a LAMS Research Procedures

15. Choose one of four broad reasons (or “other”) for asserting LAMS Estimate degree of measurement inaccuracy Complete an open-ended description of rationale for nomination Nomination Form

16. Fluency limitations obscure comprehension skills Comprehension limitations obscure other reading skills Has strengths outside of what most reading tests cover Responds poorly to standardized testing circumstances or materials The Four Reasons – sliding into “types”

17. CBM, listen to text with immediate retelling CBM, read with immediate retelling Typical test administration (multiple choice questions), think aloud, interview Typical test administration (multiple choice questions) observed Four Brief Examinations Protocols

18. IRB, School Administrators and Timing Small numbers of participants Questionnaire results Interview and examination results Conclusions PART 3: Pilot Student Characteristics Study Results

19. Questionnaire Results 28 total students identified by 7 teachers 5 New York teachers identified 21 students 2 Minnesota teachers identified 7 students 11 females / 15 males / 2 unspecified gender Questionnaire Results

20. Two teachers from Minnesota Six identified students Interview and Examination Results

21. 4th grade Teacher identified as a Type 1 student Teacher interview w/student work: “when she is able to decode, she can answer comprehension questions with an average of 80% accuracy” “comprehension pretty good when something is read to her” [during testing] “when the teacher walks away it is just click, click, click because it is just too hard for her” CBM = 42 (median score); halted reading Oral retelling inconclusive in assessing comprehension skills Nora is most likely a Type 1 student Student Profile: Nora

22. 4th grade / teacher identified as a Type 4 student Teacher interview w/student work: “very fluent reader with an extreme lack of motivation and attention. Unless someone is sitting next to him and making him read a test out loud (with much arguing and resisting), he is able to perform better. When left on his own to complete a test, he finishes very fast and pretty much doesn’t read anything and guesses the whole test.” Mitch responded to “curriculum called ‘The Real Deal.’ It has high interest, real life stories..some with high level vocab. Mitch did very well…he did the best of anyone...He was fluent and motivated.” Given multiple assessments: CBM = 120 (median score); 1 out 5 correct on multiple choice questions; oral retell focused on plot Mitch likes reading; “reading words is easy.” Mitch is most likely a Type 4 student Student Profile: Mitch

23. 4th grade / Teacher identified as a Type 4 student Teacher interview w/student work: “I stepped out during the testing, and told him to keep working. When I got back, he had answered 20 questions in 5 minutes, when we were working together it took him 45 minutes to get through approximately 15. He was just clicking as soon as I stepped out of the room.” (I) “very slow reader. His reading is fairly accurate, but it takes a very long time for him to get through a sentence.” “He is a slow processor-even when you talk with him, it takes him a long time for him to get his thoughts out.”(I) District reading scores at grade level Loves to read; “it is fun to read with more time” CBM = 45 (median score) Oral retelling and MC responses (4 out of 5 correct) support that Tony comprehends well; problem with data collection Tony’s data are inconclusive Student Profile: Tony

24. 7th grade / teacher identified as a Type 4 student/ tested as Type 1 Teacher interview: “True LD student…she’s slow processing things and she likes to think about things…she’ll take stuff home and really work on it, and she’ll think about it and come back the next day and really understand things. So I think it takes her awhile to process information.” Reads Harry Potter and Babysitter’s Club series novels Student interview: “I just read really slow. Nothing else about reading that’s hard.” “I did really bad on reading [test]. I think I went too fast – I just wanted to get done.” CBM = 90 (median score) Megan became frustrated during oral retelling, results not conclusive Megan’s assessment results are inconclusive Student Profile: Megan

25. End of 7th grade / tested with 8th grade materials Teacher identified as Type “other” student Teacher interview: District reading scores below grade level “Kurt rushes through things – completed the district test for reading in 20 minutes.” “He loves to read. He is always reading.” Kurt does not believe that reading tests show his skill level “because I don’t try…rather not take the test.” Kurt likes reading “a little” and finds it a bit difficult; “I don’t really pay attention, if it’s boring I sort of tune out.” 5 out of 7 correct on multiple choice questions Kurt is most likely a Type 4 student Student Profile: Kurt

26. End of 7th grade / tested with 8th grade materials Teacher identified as Type “other” student /tested Type 4 Teacher interview: District reading scores below grade level Marcus has a reading learning disability “he was not interested in doing his best because the standardized test was way too long” Student interview: [testing on computers] “it is easy to choose answer C all the time and just get done with it” “It’s not hard to read but if you do it for too long it can get boring sometimes unless you have a good book.” Scored 3 out of 7 correct on multiple choice assessment Marcus’ data are inconclusive Student Profile: Marcus

27. Teachers seem able to identify LAMS Distinguishing reasons for LAMS is less clear PARA brief examination procedures need refining Very Preliminary Conclusions