A MULTIDIMENSIONAL APPROACH TO THE IDENTIFICATION OF TEST FAIRNESS

1. A MULTIDIMENSIONAL APPROACH TO THE IDENTIFICATION OF TEST FAIRNESS EXPLORATION OF THREE MULTIPLE-CHOICE SSC PAPERS IN PAKISTAN Syed Muhammad Fahad Latifi fahad.latifi@aku.edu Dr. Thomas Christie thomas.christie@aku.edu

2. Perspective of Test Fairness • Substantive / judgmental analysis • Statistical • - Dimensionality of content. • Primary dimension • Secondary dimension ( impact or bias) • - Test item measuring dimension other than primary dimension is produce differential Item functioning (DIF). • - Bundle of Test items measuring dimension other than primary dimension produces differential bundle functioning (DBF).

3. Multidimensionality of test item / bundle of items. • DIF/DBF can be uniform and non-uniform.

4. DIF / DBF can produce interesting explanations . e.g., it may be due to item format characteristics, subject matter related factors and cognitive skills measured on the test. • Males are considered as reference group (majority group) • Females are considered as focal group (minority group)

5. RESULTS

7. Phase –one DIF One item in each subject were found with severe DIF. i.e. only 3.7% of the total item pool.

8. Phase –one DIF (cont.): Three Severe DIF Items

9. Phase One-DIF (cont.):

10. Three Severe DIF actual item text

11. Phase Two-DBF • Analogous to DIF, DBF is conceptualized as several DIF items acting in concert to produce an item bundle favoring matched examinees from one group over another, as judge by bundle score. • The term bundle indicate a set of items organized/ grouped together because they share a common content dimension, cognitive similarity or share a common item structure. • Four organizing principles are suggested in literature. • 1) Test Specification, 2) Content Analysis, 3) Psychological Analysis, and 4) Empirical Analysis. • Test Specification as organizing principle is used in this study.

12. Phase Two-DBF (cont.) :

13. Phase Two-DBF (cont.) :

14. Phase Two-DBF (cont.) : However, DBF is controversial due to amplification and cancellation effect. The small item-level differences, which may go unnoticed, can be magnified when the same difference is evaluated with a bundle, also called DIF-amplification . DIF cancellation is caused when the bundle of items exhibiting DIF against one group while another bundle of items exhibits DIF against the alternate group and therefore each is canceled out. Authors of the 1999 Standards for Educational and Psychological Testing state: “Although DIF procedures may hold some promise for improving test quality, there has been little progress in identifying the causes or substantive themes that characterize items exhibiting DIF. That is, once items on a test have been statistically identified as functioning differently from one examinee group to another, it has been difficult to specify the reasons for the differential performance…" (p. 78).

15. CONCLUSION • The results of this study indicates that there were only three items with Level‑C DIF in the AKU-EB`s SSC May 2011 English, Mathematics and Physics examinations. • DBF is controversial and has limited significance from practitioners’ perspective. Further, to-date , no guidelines exist to interpret the effect size measure for DBF and thus, research is needed to identify and evaluate effect size guidelines for interpreting differential bundle functioning. • Taken together, the present study suggests that the small amount of DIF found does not confound the validity of the interpretation of the examinees’ test scores on SSC examination and likewise test development practices are fair for both male and females.