The effect of differential item functioning in anchor items on population invariance of equating

1. The effect of differential item functioning in anchor items on population invariance of equating Anne Corinne Huggins University of Florida

2. Introduction • Population invariance of equating • Test score level • Differential item functioning • Item level • Need to connect two facets of invariance • Certain types of DIF might cause equating dependence across forms of a test. • Due to • a differential difficulty, particularly for anchor items • a second factor(s) • Item parameter drift

3. IRT true score equating • A three-step process: • Specify a true score on one form • Find the corresponding IRT ability estimate • Find the IRT ability estimate’s corresponding true score on the other form

4. Place the latent ability scores on each test on the same scale We can use the probability formula to get the relations.

5. Scaling • Mean-sigma • Mean-mean Q: How to estimate A under Rasch model? Here, measurement errors are ignored. If a and b were biased, A and B will be biased.

6. The Stocking and Lord method • Minimize the difference in test characteristic curves (TCCs) for anchor items • The Haebara method • Minimize the difference in item characteristics curves (ICCs)

7. Equating invariance • Compare the whole group to multiple subgroups • Group to overall (compare one subgroup with the whole group) Sigma Q is the unconditional SD of scores in population.

8. The factors that might influence population invariance of IRT true score equating • Differential anchor form DIF • The magnitude of DIF in anchor items • Number of DIF anchor items • Direction of DIF in anchor items • Ability differences between subpopulations

9. Method • Uniform DIF • R, BILOG-MG, SPSS • Design • Nonequivalent groups with anchor test (NEAT) design • Two forms contain 50 dichotomous items, respectively, with 20% of common items • Ability ~ N (0,1) • 4500 participants (3000 in the references group) • 3PL model • 100 replications

10. Manipulations on DIF in anchor items • Magnitude (.3, .6, .9) • % of DIF items (20%, 49%, 60%) • Directionality of DIF • Mean ability differences • Differential anchor form DIF • Null conditions • No DIF in anchor items and no mean ability differences • No DIF in anchor items but had mean ability differences

11. Results—No DIF (RMSD(x))

12. No DIF, Equal mean abilities (RSD(x))

13. No DIF, Unequal mean abilities (RSD(x))

14. Why didn’t DIF show the impact here?

25. Conclusions • Differential anchor form DIF occurs, the magnitude of equating dependence increases. • IPD occurs for different subpopulations. • When differential anchor from DIF occurs: • Direction of DIF in anchor items shows the largest effect on equating dependence. • The magnitude of DIF in anchor items • Number of DIF anchor items • Ability differences between subpopulations

26. Comments • It is a good writing sample. • Errors: • P.9, since the study manipulated the direction of DIF, the statement of magnitude of DIF should be revised properly. • The author intended to design the sample size as the true subpopulations in the US, but the actual sample sizes did not reflect its intends.

27. When DIF or item parameter drift occurs in a specific anchor item, this item should not be used as an anchor. • We usually do not talk about DIF in anchor items because DIF detections should be done before any equating or linking. • When all anchor items are DIF items, the results should be very similar with the ones without DIF anchor items.

28. Future studies • Sample size differences across score levels • Test length, anchor length, ratio of anchor to test length, mixed item formats