1 / 22

Lecture 7: Item Analysis

Lecture 7: Item Analysis. PSY 605. Item Analysis - Overview. What is it? Item-level analysis of psychometric properties Assessment of how each item contributes to the reliability and validity of the scale Method of determining which items to cut/edit When do we do it?

ardith
Download Presentation

Lecture 7: Item Analysis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Lecture 7: Item Analysis PSY 605

  2. Item Analysis - Overview • What is it? • Item-level analysis of psychometric properties • Assessment of how each item contributes to the reliability and validity of the scale • Method of determining which items to cut/edit • When do we do it? • Before and/or during validation study • Ideal: • Study 1 – Pilot Test, Item Analysis, Revisions • Study 2 – Validation Study, (more) Item Analysis, (more) Revisions • Study 3 – (another) Validation Study • Typical: all 3 at once

  3. Qualitative & Quantitative Analyses • Qualitative (old news) • Content adequacy & dimension fit judgments • Done through good scale development process • Quantitative (today’s focus) • Item Difficulty (i.e., Item Endorsement) • Item Discrimination • Item Homogeneity (Impact on Scale Reliability) • Factor Loadings

  4. Overview of Quantitative Item Analyses

  5. Item Difficulty • Item difficulty index (aka item endorsement index) = proportion of test takers responding correctly (or endorsing the item) • Higher value = easier item/ less extreme item • Knowledge tests: based on classical test theory assumptions, aim for difficulty levels around .5 • Attitude/perception tests • Ideal value dependent on purpose • Use p(agree or strongly agree) and/or mean score

  6. Item Discrimination • Extent to which an item can discriminate (i.e., separate) test-takers with high overall test scores from those with low overall test scores • Groups can be top/bottom 10%, top/bottom thirds…

  7. Item Homogeneity • Inter-item correlations • Correlations between all possible item pairs (viewed as matrix) • looking for moderately strong, positive values • negative values may indicate forgotten reverse-coding or bad item • very strong values may indicate unnecessary redundance • Scale Alpha when item removed • What Cronbach’s Alpha would be if the scale did not contain that item (and, thus, was 1 item shorter) • A good item: Alpha drops significantly when item removed • A bad or unnecessary item: Alpha increases or doesn’t change much when item removed

  8. Factor Loadings • A step back – a conceptual look at factor analysis Latent Construct B Latent Construct A Item 1 Item 2 Item 7 Item 3 Item 8 Item 4 Item 5 Item 6

  9. Factor Loadings Latent Construct A Latent Construct B Item 1 Item 3 Item 5 Item 7 Item 6 Item 8 Item 2 Item 4

  10. Factor Loadings – what you hope to see Latent Construct A Latent Construct B .5 .6 .8 .2 .2 .1 Item 1 Item 3 Item 5 Item 7 .9 .3 Item 6 Item 8 Item 2 Item 4

  11. Factor Loadings – what you hope to see Latent Construct A Latent Construct B .9 .7 .1 .2 .5 Item 1 Item 3 Item 5 Item 7 .8 .2 .1 Item 6 Item 8 Item 2 Item 4

  12. Factor Loadings – if you see… Latent Construct A Latent Construct B .9 .7 .1 .2 .5 Item 1 Item 3 Item 5 Item 7 -.7 .2 .1 Item 6 Item 8 Item 2 Item 4

  13. Factor Loadings – if you see… Latent Construct A Latent Construct B .9 .7 .6 .2 .5 Item 1 Item 3 Item 5 Item 7 .2 .2 .1 Item 6 Item 8 Item 2 Item 4

  14. A Full Example: Item Analysis and the Organizational Injustice Survey

  15. Item Endorsement: Item Means & Variances

  16. Item Impact on Scale Reliability:Inter-Item Correlations

  17. Item Impact on Scale Reliability: Alpha when item removed Injustice in Opportunities & Outcomes: α = .913 Injustice in Guidelines & Expectations: α = .904 Injustice in Interpersonal Treatment: α = .913

  18. Factor Analysis: Principal Axis Factor Analysis with Direct Oblimin Rotation

  19. Factor Analysis: Principal Axis Factor Analysis with Direct Oblimin Rotation

  20. Factor Analysis: Principal Axis Factor Analysis with Direct Oblimin Rotation Items 1-7: “Injustice in Opportunities & Outcomes” Items 8-15:”Injustice in Guidelines & Expectations” Items 16-24: “Injustice in Interpersonal Treatment”

  21. Determining Item Edits & Deletions

  22. Final Scale • 6 items cut that violated the majority of the ‘good item’ rules • Final measure: 18 items; 6 items for each of the 3 subscales • Subscale Internal Consistency Reliabilities: • Outcomes & Opportunities: .90 • Guidelines & Expectations: .88 • Interpersonal Treatment: .90 • NOW we collect validation evidence of the final scale, using all the strategies discussed in lectures 5 & 6.

More Related