Computerized Adaptive Testing: What is it and How Does it Work?

1. Computerized Adaptive Testing: What is it and How Does it Work?

2. Goals of this session • Learn about Computerized Adaptive Testing (CAT) • Review Item Response Theory (IRT) • Combining CAT with IRT • Pros and cons of CAT • Answer questions

3. Not to be confused with… Computerized Adaptive Testing: Not as cute, but far fewer hairballs.

4. PART I Introduction to CAT

5. Motivation for Understanding CAT • There are already operational assessments that use CAT • Some believe it will revolutionize classroom testing in the future • Interesting idea that speaks to potential of computers to have new uses in education • Item Response Theory is all over testing now

6. OK, so what is CAT? • A type of assessment where a question is displayed on a monitor • Students use mouse to select answer • Computer chooses next question based on previous responses • Next question is displayed on monitor, or else test ends

7. A graphical representation Questions chosen depend on prior responses

8. Analogy: A Game of 20 Questions • I am thinking of an object. You have 20 “yes-or-no” questions to figure it out. • Would you write out all your questions ahead of time? 1) Is it an animal? 2) Is it a vegetable? 3) Is it blue? 4) Is it red? 5) Is it bigger than a car? 6) Etc.

9. 20 Questions, Continued • Isn’t it more effective to base your next question on previous answers? 1) Is it an animal? NO. 2) Is it a vegetable? YES. 3) Is it commonly found in a salad? YES. 4) Is it green? NO. 5) Would Bugs Bunny eat it? YES.

10. Same principle used in CAT • Computer keeps track of each student’s pattern of responses so far • As test progresses, learn more about individual student • Choose next question (item) to get maximal info about that particular student’s level of ability • Purpose of assessment: Get best possible information about students

11. Some items are more informative than others? Sure! • Some items are easier than others: 2 + 2 vs. 54389 + 34697 • Some items are more relevant than others: 3 + 7 vs. Academy Awards question • Some items are better at discerning proficient students from those who need improvement

12. Which is most informative? • Suppose we have only 2 types of students: “Advanced” and “Beginning” • Use the test to classify each student • Which item below is the best for this purpose?

13. Item 3 is the best • Item 1 is completely useless • Item 2 gives some information • Item 3 is all you need!

14. But wait… • Wouldn’t we choose Item 3 for ALL students? • If so, why customize a test for an individual student? • Answer: For some students, Item A is more informative. For others, Item B is more informative.

15. When is one item more informative than another? • Item A: 2 + 2 • Item B: (34 + 68) / 2 • If you’ve answered many difficult items correctly, Item A is waste of time • If you’ve answered many easy items incorrectly, Item B is too hard • Thus, give Item B to high-performing students, Item A to low-performing students

16. Isn’t that unfair? • It seems like CAT penalizes students for performing well at start • If we give different items to different students, how can we compare their performances? • The above question arises whether we use CAT or not • Item Response Theory to the rescue!

17. Summary of Part I • CAT customizes assessment based on previous responses, as in 20 Questions • Certain items more informative than others • For some students, Item A is more informative; for others, Item B is • When give different items to different students, need way to relate student performances (Item Response Theory)

18. PART II Review of Item Response Theory

19. Item Response Theory (IRT) • Quantifies the relation between examinees and test items • For each item, gives probability of correct response by ability level • Provides a means for describing characteristics of items, estimating ability of examinees • Places examinees on common scale when they have taken different items

20. The IRT Model: One item

21. Different items have different curves

22. Where did those curves come from? • In IRT, ability is denoted by θ • Probability of a correct response is • Each item has its own values of a, b, and c. We know them from field testing • a is the “discrimination”: Related to the slope • bis the “difficulty”: Harder item, higher b • c is the “guessing parameter”: Chance of lucky guess

23. Effect of the a parameter • All curves shown have equal b and c parameters • Larger a increases the slope in the middle

24. Effect of the b parameter • All curves shown have equal a and c parameters • Larger b means harder item

25. Effect of the c parameter • All curves shown have equal a and b parameters • c is the left asymptote

26. Wait a minute • What do you mean by a student with an ability of 1.0? • Does an ability of 0.0 mean that a student has NO ability? • What if my student has a reading ability of -1.2? What in the world does that mean???

27. The ability scale • Ability is on an arbitrary scale that just happens to be centered around 0.0 • We use arbitrary scales all the time: • Fahrenheit • Celsius • Decibels • Nevertheless, need more “user-friendly” reporting: “scaled” scores on conventional scale like 200-300

28. Giving a score for each student • First assign an ability (θ) value to each student (say, -4 to 4) • Student is given the value of θ that is most consistent with his/her responses • The better he/she does on the test, the higher the value of θ that he/she receives • Computer converts the θ score to a scaled score • Report final score!

29. Assigning scores • Set of answers: (C,C,I,C,C,I,I,C,C,C,I,C,C) • We know which items were taken by each student: a, b, c parameters • If Student 1’s items were harder than Student 2’s, take into account through item parameters • Student 1: θ = 1.25, scaled score = 290 • Student 2: θ = 0.65, scaled score = 268 • Can compare students who took different items!!!

30. Summary of Part II • If you didn’t get all that, don’t worry • Just remember: • In IRT, different items have different curves (depending on a, b, c parameters) • IRT allows us to give scores on the same scale, even when students take different items • These features critical in CAT • So how do we choose which items to give?

31. PART III Combining CAT with IRT

32. CAT Reminder • CAT customizes assessment based on previous responses • For some students, Item A is more informative; for others, Item B is • With IRT, it’s OK to give different items to different students

33. Which item would you choose next? PREVIOUS RESPONSES: • 10 + 19 = ? Answered correctly. • 27 + 38 = ? Answered incorrectly. • 12 + 26 = ? Answered incorrectly. POSSIBLE ITEMS TO GIVE NEXT: • 18 + 9 = ? • 13 + 17 = ? • 14 + 20 = ?

34. Item selection to match ability/difficulty • Want to give items appropriate to ability • 2 + 2 is not informative for high-performing students; (34 + 68) / 2 is not informative for low-performing students • Student has taken 10 items, awaits 11th • Classic approach: Give item whose difficulty (b) is closest to current ability estimate (θ)

35. Which item is better for θ = -1.2? Easier item Harder item

36. More complex item selection • Previous method: Match difficulty to ability • This criterion only uses b parameter and θ • Recall that a parameter is related to slope, c is guessing parameter • Shouldn’t we consider those when choosing next item?

37. Another item selection method • Ideal item: High value of a; value of b close to θ; low value of c • “Fisher Information” combines these factors into a single number • Choose item with highest Fisher Info

38. Game: Which item would you choose? • Suppose our current estimate of θ is 0.6

39. Results • If matching ability estimate (0.6) with difficulty, we would give Item 2 • If using Fisher Info, we would give Item 2

40. Round 2 • Suppose our current estimate of θ is 0.7

41. Round 2 Results • If matching ability estimate (0.7) with difficulty, we would give Item 2 • If using Fisher Info, we would give Item 1

42. Summary of Part III • Tailor items to be most informative about individual student’s ability • Do this by combining CAT with IRT • One method: Match difficulty with current estimate of θ • Another method: Take all parameters into account via Fisher Info

43. PART IV Practical Considerations

44. Problem: Content Balance • In operational testing, must balance content (e.g., math test of algebra, geometry, number sense) • What if all your most informative items come from the same content strand? • In practice, dozens of constraints for each CAT: Content, topics, enemies list, etc. • CAT solution: Pick most informative item among those “in play”

45. Problem: Test security • CAT administered on multiple occasions • Person A takes exam, memorizes items, tells Person B. Person B takes exam, benefits from Person A’s information • Different students, different items; however, some items more popular than others • CAT solution: Limit the amount each item can be administered

46. CAT “Pros” • Convenient administration • Immediate scoring • Items maximally informative: Exams just as accurate, with shorter tests • Items at correct level: High-performing students not bored, low-performing students not overwhelmed

47. CAT “Cons” • Limited by technology • Potential bias versus students with less computer experience • Content balance less exact than paper-and-pencil testing • Test security • Expensive

48. Final summary • Introduction to CAT: Benefits of giving different items to different students • Review of IRT • Using IRT to select items in a CAT • Pros and cons of CAT