1 / 21

Validity and Item Analysis

Validity and Item Analysis. Chapter 4. Validity. Concerns what the instrument measures and how well it does that task Not something an instrument has or does not have Informs counselor when it is appropriate to use the instrument and what can be inferred from the results

arwen
Download Presentation

Validity and Item Analysis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Validity and Item Analysis Chapter 4

  2. Validity • Concerns what the instrument measures and how well it does that task • Not something an instrument has or does not have • Informs counselor when it is appropriate to use the instrument and what can be inferred from the results • Reliability is a prerequisite for validity

  3. Movement Away from Traditional Categories of Validity • Traditional types of validity • Content-Related • Criterion-Related • Construct-Related

  4. Evidence Based on Instrument Content • The degree to which the evidence indicates the items, questions, or tasks adequately represent the intended behavior domain. • Central focus is typically on how the instrument’s content was determined • Content-related validation evidence should not be confused with face validity

  5. Evidence Based on the Response Processes • Information processing • Examining the response processes used by different groups • Computer simulated performances

  6. Evidence Based on Internal Structure • Examining internal structure using factor analysis • Examining the internal structure of an assessment with groups (e.g., gender, race or ethnicity)

  7. Evidence Based on Relations to Other Variables • Correlational method • Convergent evidence/Discriminant evidence • Multitrait-multimethod matrix • Prediction or instrument-criterion relationship • Concurrent and predictive validity • Regression • Decision Theory • Validity Generalization

  8. Correlational Method • Correlational method-examine relationship between instrument and other pertinent variables or a criterion • Validity coefficient(s)

  9. Prediction or instrument-criterion relationship • Concurrent validity • Predictive validity

  10. Regression . . 5 . . … … . 4 . . . … Ratings … . … . . . 3 . .. . . .. .. . .. 2 . 1 10 20 30 40 50 60 70 80 90 100 ABC

  11. Regression Line . . 5 . . … … . 4 . . … . Ratings … . Y' = a + bX … . . . 3 . .. . . .. .. . a = intercept b = slope X = score .. 2 . 1 10 20 30 40 50 60 70 80 90 100 ABC

  12. The margin of expected error in the individual’s predicted criterion score as a result of imperfect validity. Standard Error of Estimate

  13. Decision Theory: Expectancy Table ABC Scale Ratings

  14. Expectancy Table (cont.) Misses Hits Hits Misses

  15. Expectancy Table (cont.) False Negatives False Positives

  16. Validity Generalization • A method of combining validation studies to determine if the validity evidence can be generalized • Must be a substantial number of studies • Meta-analysis

  17. Evidence-based consequences of testing • Social consequences of racial/ethnic group differences • Consider social implications when examining the validation evidence

  18. Conclusion on validation evidence • Gradual accumulation of evidence • Counselor must evaluate the information to determine if it appropriate for which client under what circumstance • Validation evidence should also be considered in informal assessments

  19. Item Analysis • Focus is on examining and evaluating each item on an instrument. • Item difficulty: p = # who answered correctly total number • Item discrimination: d = upper % - lower % (or correlational method)

  20. Item Response Theory or Latent Trait Theory • Focus is on each item and on establishing items that measure the individual’s ability or level of a latent trait. • Involves examining the item characteristic function (or characteristic curve) and the calibration of each individual item. • Parameters: • Difficulty • Slope • Basal level

  21. Item Characteristic Curve 1.00 .50 .00 % 4 5 6 7 8 9 Grade

More Related