Measurement Problems within Assessment: Can Rasch Analysis help us?

1. Measurement Problems within Assessment: Can Rasch Analysis help us? Mike Horton Bipin Bhakta Alan Tennant

2. Mental Arithmetic - Test 1 • 4 + 5 = • 2 x 3 = • 18 ÷ 2 = • Arithmetic is one of the ‘3 R’s’. True or False? = • 17 x 13 =

3. Mental Arithmetic - Test 1 - Answers • 4 + 5 = 9 • 2 x 3 = 6 • 18 ÷ 2 = 9 • Arithmetic is one of the ‘3 R’s’. True or False? = True • 17 x 13 = 221

4. Assumptions underpinning test score addition • All questions must be mapping onto the same underlying construct • Unidimensionality • All questions must be unbiased between groups • Item Bias  Differential Item Functioning (DIF) • Raw score is a sufficient statistic

5. Mental Arithmetic - Test 1 – Potential Problems • 4 + 5 = 9 • 2 x 3 = 6 • 18 ÷ 2 = 9 • Arithmetic is one of the ‘3 R’s’. True or False? = True • 17 x 13 = 221

6. Mental Arithmetic - Test 1 – Potential Problems • 4 + 5 = 9 • 2 x 3 = 6 • 18 ÷ 2 = 9 • Arithmetic is one of the ‘3 R’s’. True or False? = True • 17 x 13 = 221 Plus: Item Bias -Gender DIF has been shown to be a particular problem in mathematics exams. (e.g. Scheuneman & Grima, 1997., Lane et al. 1996)

7. Mental Arithmetic - Test 2 • 17 x 13 = • 47 x 64 = • 768 ÷ 16 = • 532 = • 73 =

8. Mental Arithmetic - Test 2 - Answers • 17 x 13 = 221 • 47 x 64 = 3008 • 768 ÷ 16 = 48 • 532 = 2809 • 73 = 343

9. Assumptions of Test Equating (Holland & Dorans, 2006) • Tests measure the same characteristic • Tests measure at the same level of difficulty • Tests measure with the same level of accuracy

10. Requirements of Test Equating (Dorans & Holland, 2000) • The tests should measure the same construct • The measures from the tests should have the same reliability • The function used to equate measures from one test to another should be inversely symmetrical • Examinees should be indifferent about which of the equated test forms will be administered • The function for equating tests should be invariant across subpopulations of examinees

11. Are these elements currently assessed? • Unidimensionality – is assumed on face validity • Cronbach’s alpha • Exam Difficulty Equivalence • Subjective procedures • Classical Test Theory = sample dependent

12. What is Rasch Analysis?

13. Mesa Press, Chicago 1980

14. Rasch Analysis • The Rasch model is a probabilistic unidimensional model • the easier the question the more likely the correct response • the more able the student, the more likely the question will be passed compared to a less able student. • The model assumes that the probability that a student will correctly answer a question is a logistic function of the difference between the student's ability and the difficulty of the question Rasch G. Probabilistic models for some intelligence and attainment tests. Chicago: University of Chicago Press, 1980

15. Assumptions of the Rasch Model • Stochastic Ordering of Items • Unidimensionality • Local Independence of Items

16. What Would We Expect When These People Meet These Items? Hard Easy Least Most Able

17. What Would We Expect When These People Meet These Items? Hard Easy Least Most Able

18. What Would We Expect When These People Meet These Items? Hard Easy Least Most Able

19. What Would We Expect When These People Meet These Items? Easy Hard Least Most Able

20. What Would We Expect When These People Meet These Items? Hard Easy Least Most Able

21. The Guttman Pattern 1 2 3 4 5 6 Total Score 0 0 0 0 0 0 0 1 0 0 0 0 0 1 1 1 0 0 0 0 2 1 1 1 0 0 0 3 1 1 1 1 0 0 4 1 1 1 1 1 0 5 1 1 1 1 1 1 6

22. The Rasch Guttman Pattern 1 2 3 4 5 6 Total Score 0 0 0 0 0 0 0 1 0 0 0 0 0 1 1 1 0 1 0 0 3 1 1 1 0 0 0 3 1 1 0 1 0 0 3 1 1 1 1 1 0 5 1 1 1 1 1 1 6

23. The Probabilistic Rasch Model Probability of a student’s success on an item student 27% 12% 5% 95% 88% 73% -3 -2 -1 0 1 2 3 Difference (in logits) between the ability of the student and the difficulty of the item

24. Rasch Analysis • When data fit the model, generalisability of Item difficulties beyond the specific conditions under which they were observed occurs (specific objectivity) • In other words… • Item Difficulties are not sample dependent as they are in Classical Test Theory

25. What Else Does Rasch Offer us? • When data fit the Rasch Model, the assumptions of summation are met • All questions must be mapping onto the same underlying construct • All questions must be unbiased between groups (DIF) • Raw score is a sufficient statistic • We can then test for other things • Quality of Distractors • It gives us the mathematical basis to compare test scores via equating

26. Limitations of Rasch Analysis • The model tests the internal psychometric properties • The model assumesunidimensionality • The model cannot set standards

27. Summary • The Rasch model offers a unified framework under which all of the assumptions can be tested together • It gives us a lot of information about individual items which can be utilised to ensure that item and test construction is of a high quality • It provides a rigorous mathematical basis for test equating

28. References • Rasch G. Probabilistic models for some intelligence and attainment tests. Chicago: University of Chicago Press, 1980 • Dorans NJ & Holland PW. Population invariance and the equatability of tests: Basic theory and the linear case. Journal of Educational Measurement, 2000: 37; 281-306 • Holland PW & Dorans NJ. Linking and Equating. In RL Brennan (Ed.), Educational Measurement (4th ed., p187-220). Westport, CT: American Council on Education and Praeger Publishers, 2006. • Scheuneman JD & Grima A. Characteristics of quantitative word items associated with differential item functioning for female and black examinees. Applied Measurement in Education, 1997; 299-320. • Lane S, Wang N, Magone, M. Gender-Related Differential Item Functioning on a Middle- School Mathematics Performance Assessment. Educational Measurement, 1996: 15(4); 21-27