1 / 19

Fit of Ideal-point and Dominance IRT Models to Simulated Data

Fit of Ideal-point and Dominance IRT Models to Simulated Data. Chenwei Liao and Alan D Mead Illinois Institute of Technology. Outline. Background and Objective Hypotheses and Methods Results Discussions. Background. Personality

marycohen
Download Presentation

Fit of Ideal-point and Dominance IRT Models to Simulated Data

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Fit of Ideal-point and Dominance IRT Models to Simulated Data Chenwei Liao and Alan D Mead Illinois Institute of Technology

  2. Outline • Background and Objective • Hypotheses and Methods • Results • Discussions

  3. Background • Personality Used in personnel selection - Incremental validity to predict job performance beyond cognitive ability (Barrick & Mount, 1991; Ones et al, 1993) - Less adverse impact (Feingold, 1994; Hough, 1996; Ones et al, 1993). • Model-data-fit - Need to calibrate personality traits - Use IRT models - Degree of fit depends on data structure

  4. Background (cont.) • Item response processes – thinking of data structure • IRT models and item response processes: 1) Traditional dominance IRT models: - high trait - high probability of endorsing 2) Ideal-point IRT models - similar item & trait – high probability of endorsing

  5. Background (cont.) Dominance Model IRF: - x: Theta (trait level) - y: Probability of endorsing Ideal-point Model IRF: - x: distance between person trait and item extremity - y: Probability of endorsing

  6. Background (cont.) • Chernyshenko et al, (2001) - Traditional dominance IRT models have failed. Suggest to look at item response processes and Ideal-point IRT models • Stark et al. (2006) - Ideal-point IRT models: as good or better fit to personality items than do dominance IRT models • Chernyshenko et al. (2007) - Ideal-point IRT method: more advantageous than dominance IRT and CTT in scale development in terms of model-data-fit

  7. Limitation of previous studiesand objective of current study • Limitation of previous studies - Unknown item response processes! • Objective of current study 1) Investigate model-data-fit by utilizing simulation with known item response processes 2) Test the assumption that the best fit model represents data underlying structure of response processes

  8. Current Study

  9. Models • Dominance: - Samejima’s Graded Response Model (SGRM); • Ideal Point: - General Graded Unfolding Model (GGUM). • Larger sample and longer test were said to be related to a better fit (Hulin et al, 1982; De la Torre et al, 2006).

  10. Hypotheses Generating models • H1: Data generated by an ideal point model will be best fit by an ideal-point model and data generated by a dominance model will be best fit by a dominance model. • H2: The ideal point model will fit the dominance data better than the dominance model will fit the ideal-point data. • H3: The ideal-point model will fit the mixture data better than the dominance model.

  11. Hypotheses (cont.) Sample Sizes • H4: All models will fit better in larger samples. • H5: The GGUM model will fit relatively worse in smaller samples, as compared to simpler, dominance models. Test Lengths • H6: The GGUM model will fit relatively worse for very short tests, as compared to longer tests.

  12. Datasets • Self-Control Scale from the 16PF • Procedure:1) Calibrate 16PF data to get item parameters - SGRM: PARSCALE4.1; GGUM: GGUM2004.2) Generate simulated data: - models: ideal point/dominance/mixed; - sample size: 300, 2000; - test length: 10, 37; - 50 replications;

  13. Model-Data-Fit • Cross validation ratio: each item in each condition • Only singles – simulation study assures unidimensionality assumption • Smaller value – better fit • Frequencies of ratios were tallied into 6 groups: very small (<1), small (1-<2), medium (2-<3), moderately large (3-<4), large (4-<5), very large (>=5).

  14. Results overview

  15. Results

  16. Discussion (1) • “GGUM fits better” - Confirm previous findings. - However, because regardless of the underlying response process, GGUM fits better than SGRM, it does not demonstrate that the response process or IRF/ORF is non-monotone. The previous assumption does not hold true. - Possible reason: Software (PARSCALE & GGUM) manifest models differently • Better fit in small samples, especially for SGRM - Explanation: chi-square is sensitive to sample size

  17. Discussion (2) • Examine similarities of the theta metrics - Negative correlation between theta estimates from GGUM and those from SGRM

  18. Discussion (3) GGUM: - Reverse the estimate - Add a constant in scaling Scaling issue

  19. Thanks!

More Related