A procedure for dimensionality analyses of response data from various test designs

1. A procedure for dimensionality analyses of response data from various test designs JinmingZhang William Stout

2. Introduction • Dimension • Dimensional structure of the test (e.g., algebra and geometry) • statistical dimensional structure of response data • Incorporate both Judgments about test content and evidence from statistical analyses • Missing data: CAT & multistage testing • Missing item pair measurement

5. DETECT index

6. Modified DETECT index

7. Example 3 • Stage 1 booklet: 2 dim & Stage 2 booklet 1: 2 dim, Stage 2 booklet 2: 1 dim (additional dim), Stage 2 booklet 3: 1 dim (additional dim). • Bridge item (in Stage 1) • E(1,2) and E(2,3), so E(1,3) • First-stage booklet should • measure all of the constructs/contents the whole test aim to measure, though it is unknown. • classify examinees into different proficiency levels

8. Item 1 and 2 are measuring the same dimension 1. If E(1,2), E(2,3), and E(1,3) • D(P*) is maximized in P* partition if item 1 and 2 are in the same cluster, other than in other P 2. If only E(1,2) and E(2,3), and if item 2 is a bridge item • D(P*) is maximized in P* partition if item 1 and 2 are in the same cluster when item 2 is in the same cluster or not, other than in other P

10. Whether a test has an approximate simple structure or not

11. Discordance • Resulted from • Not a approximate simple structure • Inaccuracy of conditional covariance estimation • Given a partition of items,

13. What made Dd(P*) and PropD(P*) large • Unidimensionality • Violation of approximate simple structure • Inaccuracy of conditional covariance estimation

14. polyDETECT • Obtain the composite theta score: unidimensinoal approximation & simple structure approach. • Use percentiles of composite theta scores as cut-off points in forming AHGs. • Between 25 to 100 in each group • Cross-validation:

15. polyDETECT Evidence of multidimensionality

16. Condition 3 is hold or not? Each sizable cluster contains at least one stage-1 item. Exist at least one sizable cluster that does not contain any Stage-1 items, and all items in such cluster belong to the same booklet. Exist at least one sizable cluster that does not contain any Stage-1 items, and all items in such cluster belong to at least two booklet. Exist at least two sizable cluster that does not contain any Stage-1 items, items in each such cluster come from the same booklet but different clusters belong to different booklets.

17. Dealing with CAT data Estimates of item parameters were obtained before conducting CAT. Why do dimensionality assessment on CAT data? Sparse data set of CAT 100,000 responses are required at least Item selection, item exposure control, content balance to satisfy the condition 3

18. Simulation study M2PL Each booklet has 30 items Dimension: 1, 2, 3 Number of examinees: 750, 1500, 3000, 4000 Theta: MVN(0,sigma), correlation = 0.8 Cut-off points for low-, moderate-, and high-scoring group: <10, 11~18, >18 About 37.82% unestimable item pairs Cross-validation Replication: 100 Composite theta: use unidimensional IRT model

19. Results

20. Real data analyses Missing values are large (55%~71%)

21. Real data analyses • Weak dimensionality (M value) • High PropD(P*) indicates a large amount of spurious information to form partition P* • Confirmatory analyses: D3 >D2 • But DETECT tends to underestimate, so two-cluster partition solution may be preferred.

22. Real data analyses High correlation indicates a weak degree of multidimensionality

23. Concluding remarks Moderate violation of approximate simple structure is still hold