1 / 25

Topic 10: Model Comparison

Topic 10: Model Comparison. Introduction. The LCDM is a general CDM that models the logit of the probability of success on an item That is, it employs the logit link When the model is additive, it is equivalent to the Compensatory RUM

corbin
Download Presentation

Topic 10: Model Comparison

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Topic 10:Model Comparison

  2. Introduction • The LCDM is a general CDM that models the logit of the probability of success on an item • That is, it employs the logit link • When the model is additive, it is equivalent to the Compensatory RUM denotes the number of required attribute for item j where is the reduced vector of required attributes

  3. Alternatively, we can model the log of the probability of success on an item • That is, we can employ the log link • When this model is additive, it is equivalent to the Reduced RUM • We can also choose to directly model the probability of success on an item • In doing so, we are using the identity link

  4. The general CDM based on the identity link is called the generalized DINA (G-DINA) model • Its item response function is given by is the intercept is the main effect due to is the interaction effect due to is the interaction effect due to

  5. As the name indicates, the G-DINA model is a generalization of the DINA model • Instead of two groups, the G-DINA model divides the examinees in the different latent classes into groups

  6. 1 0.75 0.5 0.25 0 DINA Model

  7. 1 0.75 0.5 0.25 0 G-DINA Model

  8. As the name indicates, the G-DINA model is a generalization of the DINA model • Instead of two groups, the G-DINA model divides the examinees in the different latent classes into groups • Without the interaction terms, the G-DINA model becomes the Additive-CDM (A-CDM) • The A-CDM is one of several reduced models that can be derived from the saturated G-DINA model

  9. Comparing Reduced and Saturated Models • Using EM algorithm, • Weight matrix W is a diagonal matrix that accounts for the differential sizes of the latent groups • The lth diagonal entry of W is • Design matrix M can be used to obtained the parameter estimates of other models • The lth row of M for the saturated model is

  10. Example when This is the design matrix for the saturated model

  11. Example when This is the design matrix for the A-CDM

  12. Example when This is the design matrix for the DINA model

  13. Example when This is the design matrix for the DINO model

  14. Based on the estimated parameters of the saturated model, the parameters of the reduced model are estimated as • By replacing with or , saturated and reduced models based on other links can also be estimated • The standard errors of the estimates can be obtained using the multivariate delta method

  15. Assuming the Q-matrix has been correctly specified, the saturated model will give the best model-data fit • Statistical tests can be performed to examine whether a reduced model can be used in place of the saturated model • Two tests developed for the G-DINA model framework are the likelihood-ratio (LR) test and the Wald test

  16. The Likelihood Ratio Test • Let where under the null • The statistic is approximately -distributed with

  17. The Wald Test • The Wald statistic is • R is the matrix of restrictions • Under the null hypothesis that

  18. Example of R for A-CDM when

  19. Example of R for A-CDM when

  20. Example of R for A-CDM when

  21. Example of R for A-CDM when Taken as a whole, this R will set

  22. The Wald Test • The Wald statistic is • R is the matrix of restrictions • Under the null hypothesis that • W is asymptotically with

  23. Model Comparison using LR

  24. Discussion • The G-DINA model is an alternative general CDM • The G-DINA model framework allows for reduced model estimation and hypothesis testing • Preliminary results indicate that • the parameters of the reduced models can be accurately estimated • the Type I error rate of Wald test is more accurate than that of the LR test • Testing reduced models at the item level can dramatically extend the flexibility of cognitive diagnosis modeling

  25. This obviates the current requirement of specifying a CDM for an assessment apriori • Multiple CDMs can be used within a single assessment • With respect to the G-DINA model, additional work is needed to better understand its properties • estimation accuracy • Type I error and power of the tests • A Very Important Note: Resources and time need to be invested to properly construct assessments that are truly cognitively diagnostic

More Related