Using Think-Aloud Cognitive Labs to Evaluate Test Item Modifications

1. Using Think-Aloud Cognitive Labs to Evaluate Test Item ModificationsAndrew RoachPaper presented as part of “Modified Achievement Tests for Students with Disabilities: Design Strategies and Experimental Results” sessionCCSSO’s National Conference on Student AssessmentJune 2008 Special thanks to Cori Wixson, Tanya Talapatra, and Tamika LaSalle for their assistance in coding the think-aloud videos.

2. Overview • Purpose: To evaluate the influence of test item modifications on students’ problem-solving and test-taking behaviors. • Our study involved three components: 1. Students completed a series of 16 assessment items (8 reading; 8 mathematics). 2. Students were asked to think aloud as they completed or solved these items. 3. We also asked follow-up questions about students’ perceptions of the assessment items. (Johnstone, Bottsford-Miller, & Thompson, 2006; Branch, 2000).

3. Distribution of item modifications X = Item modifications used.

4. Sample Size by Sub-group

5. Overview • We explained the think-aloud procedures, had the students restate their understanding of the process, and modeled thinking aloud on a practice item. • We used a script adapted from a study conducted by Johnstone, Bottsford-Miller, and Thompson (2006). • Students were prompted only when they were silent for 10 consecutive seconds. • If students verbalized infrequently, we reminded them to “keep thinking aloud” or “keep talking.” Otherwise we generally did not give encouragement or support.

6. Results: Use of Visuals • Most SWDs (67%) saw the visuals as being helpful and providing support on reading questions and passages. • 100% of the students without disabilities indicated the pictures made no difference in understanding the reading questions or passages. • Students with (50%) and without disabilities (67%) generally saw the visuals and graphs as being helpful and providing support on math items. • …However, 33% of SWDs indicated that the visuals/graphs were distracting or made it harder to answer the questions.

7. Students’ Comments:Use of Visuals • Student with disability (eligible for AA-MAS): "The one talking about the $100 bills…well it showed me, and I was understanding how it goes with what it was talking about, and I looked at it and it helped me even more.“ • Student with disability (not eligible for AA-MAS): “When people do math, they're working on a sheet and what's the point of looking at a picture. It doesn't really help you. For example, on (questions) #1 and #2, those two pictures were really messing me up.”

8. Results: Directions and Bold Type • Students without disabilities (67%) and SWDs who were eligible for AA-MAS (100%) generally preferred test directions that were explicit, such as “Read…then answer….” • Some students indicated that the less explicit directions (i.e., “Use the passage…”) might encourage test takers to skim rather than read closely. • The majority of students from all groups (78% of the total) felt the use of bold type to identify key terms was helpful in answering the reading items.

9. Results: 3 vs. 4 Answer Choices • SWDs (with one exception) perceived no difference in difficulty between items having 3 or 4 possible answers on reading items. • Conversely, 67% of the students without disabilities identified the 3-answer modification as making the reading items easier. • The results suggest that this modification did not affect either groups’ performance on most reading items (e.g., only one item [“Pesticides”] had a discernable difference in student accuracy between modified and unmodified versions).

10. Original Item Pesticides In the late 1980s, farmers began to use a pesticide to control insects that harmed their cotton crops. This problem was solved. However, an insect group that pollinated the corn crops was also injured. Without pollination the corn kernels did not fully develop. This affected the corn harvest on which the farm families had come to depend. What is not mentioned as one effect of pesticide usage?A. soil contamination B. destruction of pests C. destruction of friendly insects D. crop losses

11. Modified Item Passage revised Pesticides In the late 1980s, farmers began to use a chemical pesticide. It was used to control insects that harmed their cotton crops. This solved one problem, but caused another. An insect group that pollinated the corn crops was also harmed by the pesticide. Without pollination the corn kernels did not fully develop. This decreased the corn harvest. What is NOT mentioned as one effect of using chemical pesticides? A.destruction of the soilB.destruction of pestsC.destruction of friendly insects Question separated from text; “NOT” bolded and capitalized; and the word “chemical” added 3 answer choices; all use word “destruction” Picture added

12. Results: 3 vs. 4 Answer Choices • Students without disabilities (67%) and non-eligible SWDs (67%) generally indicated 3 answer choices made the math items easier. • Some students in these groups appeared to use the possible answer choices to help solve math items, but it was not clear that they used this same strategy in reading. • For the SWDs eligible for AA-MAS, the 3-answer choice modification was less likely to be identified as helpful, but it did seem to make a difference in performance on one particular item (i.e., “scientific notation”).

13. Student Comments:Bold Type & Number of Answer Choices • Student with disability (not eligible): “The bold type made (the answer) easier to find, but it didn’t help to understand the passage.” • Student without disability: "If you didn't get the answer right the first time, you know you only had 3 choices to go back and look at, instead of 4"

14. Results: Format of Analogies • Most students (including 2/3 of SWDs) found the traditional format for the analogy easier (i.e.,“meteor:space::dolphin:_______”). Some students indicated they had been taught analogies using this format and it was familiar to them. • This was supported by the results as SWDs correctly answered all the traditional analogy items. SWDs missed items with a modified analogy format (i.e., “meteor is to space as dolphin is to ___”) 40% of the time.

15. Results: Other Data on Reading Items

16. Results: Other Data on Math Items

17. Take Away Ideas • “Conservative” modifications were used in this study and the effects (on student performance) generally were modest. More “aggressive” modifications might result in more robust effects. • SWDs often appeared unfamiliar with some concepts (e.g., percentages) or incorrectly applied problem solving strategies on math items. In these cases, item modifications are unlikely to provide support or facilitate access. • Reading fluency may be an issue for SWDs. In some cases, SWD’s slower rates of reading resulted in testing sessions that were almost twice as long as their peers. How could (or should) technology be used to address this barrier?

18. Take Away Ideas • Students appeared more likely to guess on the modified items. Could this be attributed to having fewer answer choices? • In most cases, students in each group spent less time and required fewer prompts on the modified items. The difference was most pronounced for the SWDs eligible for AA-MAS. • Conversely, oral reading fluency did not appear to be influenced by the modifications made to reading passages.

19. References • Johnstone, C. J., Bottsford-Miller, N. A., & Thompson, S. J. (2006). Using the think aloud method (cognitive labs) to evaulate test design for students with disabilities and English language learners (Technical Report 44). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Retrieved from the World Wide Web: http://education.umn.edu/NCEO/OnlinePubs/Tech44/. • Branch, J. L. (2000). Investigating the information-seeking processes of adolescents: The value of using think-alouds and think-afters. Library and Information Science Research, 22(4), 371–392.