Link between Multilevel Modeling and Item Response Modeling: Multilevel Measurement Modeling

1. Link between Multilevel Modeling and Item Response Modeling: Multilevel Measurement Modeling Akihito Kamata Florida State University September 14, 2004 SAMSI Meeting

2. Item Response Model Basic Rasch Model bi: Difficulty of item i qp: Ability of person p

3. Background • Investigation of item and person characteristics, in relation with latent trait. • Relaxation of “item homogeneity” and “person homogeneity” assumptions.

4. Traditional Approaches • LLTM (Fisher, 1972) • Latent regression (Zwinderman, 1991) etc.

5. Recent Attempts • Generalized Rasch Model • Adams, Wilson, & Wu (1997) • Multilevel modeling • Kamata (1998, 2001) • Cheong & Raudenbush (2000) • Generalized linear mixed modeling • Reijmen et al. (2003) • De Boeck & Wilson (2004) • Generalized latent variable modeling • Skrondal & Rabe-Hasketh (2004)

6. Work in progress… • Special issue on multilevel item response modeling for Journal of Applied Measurement. • Edited book on Multivariate and Mixture Distribution Rasch Models.

7. Generalized Model: x: Item predictors z: Person predictors b: Regression weights for item predictors qp: Regression weights for person predictors

8. Example: Random Effect DIF • Differential item functioning (DIF) is present when examinees from two subpopulations with the same trait level have a different probability of answering the item correctly. • When DIF is present, there is a possibility that the magnitude of the DIF varies across group units, such as schools. • Use hierarchical linear model (HLM) to model random effect DIF.

9. Level 1 model: Level 2 models: Level 3 models:

10. NAEP 4th Grade Science Data • 12 dichotomously scored items • 477 students from 254 schools (1 to 12 students per school) • 162 students received test accommodation; 315 students did not received any test accommodation. • Initial analysis by Mantel-Haenszel procedure identified 4 DIF items (items 1, 6, 7, and 9).

11. Results

12. Issues • Issues on DIF detection • When many items exhibit DIF, DIF detection might be distorted. • Simultaneous DIF detection on all items in the model is not possible.

13. Computational Issues • Underestimation of level-3 variance by PQL. • MCMC and Gauss-Hermite quadrature improve the estimation, but very slow. • Laplace approximation for level-3 variance is one area of possible development.

14. Modeling Issues • 2PL extension has been demonstrated but limited to 2-level modeling. • 3-level modeling has been demonstrated but limited to Rasch. • 3-level modeling with 2PL extension is possible by MCMC and Gauss-Hermite quadrature, but computational speed issue is more serious. • Link to SEM.