1 / 22

MEASUREMENT Goal

MEASUREMENT Goal. To develop reliable and valid measures using state-of-the-art measurement models Members: Chang, Berdes, Gehlert, Gibbons, Schrauf, Weiss. Why Item Response Theory?. Item Response Theory (IRT).

sancho
Download Presentation

MEASUREMENT Goal

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. MEASUREMENT Goal • To develop reliable and valid measures using state-of-the-art measurement models • Members: Chang, Berdes, Gehlert, Gibbons, Schrauf, Weiss

  2. Why Item Response Theory?

  3. Item Response Theory (IRT) • A family of mathematical descriptions of what happens when a person meets a test or survey question • Relates characteristics of items (item parameters) and characteristics of persons (person latent traits) to the probability of a correct or rating/categorical response • Models the test-taking behavior at the item level

  4. Item-Person Map Person Latent Trait                            Poor Good Q Q Q Q Q Q Q Q Q Q Q Q Likely (“easy”) Unlikely (“hard”) Q Q Q Q Q Q Q Q Q Q Q Q Item Location Chang & Gehlert (2002).

  5. 1-PL (Rasch) Difficulty (b) 2-PL Difficulty (b) Discriminating (a) 3-PL Difficulty (b) Discriminating (a) Guessing (c) Dichotomous Unidimensional IRT Models

  6. Polytomous IRT Models • Polytomous • 1-PL (threshold) • Partial Credit • Rating Scale • 2-PL (threshold & discriminating) • Nominal • Graded Response • Generalized Partial Credit 2=Yes, Limited a little 1=Yes, Limited a lot 3=No, Not Limited at all 1 2 3 * Vigorous activities, such as running, lifting heavy objects, participating in strenuous sports

  7. Potential Advantages of Using IRT in “Geriatric” Pain Assessment • Refine existing instruments • Evaluate item and scale characteristics • Evaluate different response formats • Detect differential item functioning • Evaluate person fit (clinical diagnosis) • Equate/Link instruments • Establish item banks and brief forms • Develop computerized adaptive testing

  8. Item Banking and CAT A B C D E F new Item Pool (Sets of Questions) IRT  Q Q   Q  Q  Item Bank (Catalogued; Hierarchically Structured) CAT Brief Forms

  9. Principles of Adaptive Testing • IRT pre-calibrated item bank • Initial item selection • Test scoring method • Item selection during test administration • Stopping rules

  10. Item Bank • Set of carefully IRT-calibrated questions • Items covers entire latent trait continuum • Items represent differing amounts of trait • Items represent differing amounts of information • Basis for tailored/adaptive testing • Items can be selected to maximize precision and retain clinical relevance

  11. Item Banking is Inter-disciplinary • Psychometricians • Information scientists • Clinicians/healthcare providers • Outcomes researchers • Content experts • …

  12. Approaches to Develop Item Banks • Top-Down Approach • Bottom-Up Approach

  13. Development and Maintenance of an Item Bank • How to best calibrate existing items? • Model selection • Whose item parameters to use? • Standardization? • Generic vs. disease-specific • Item parameter drift • Anchor or Re-calibrate? • How to write and best test new items?

  14. Adaptive Test • An adaptive test is a tailored, individualized measure which involves selecting a set of test items for each individual that best measures the psychological characteristics of that person (Weiss, 1985) • Weiss DJ. Adaptive testing by computer. J Consult Clin Psychol. Dec 1985;53(6):774-789.

  15. Why Computerized Adaptive Testing? • Adaptive testing selects questions based on previous responses • Tailored item and test difficulties • Eliminates floor and ceiling effects • Require fewer questions to arrive at an accurate estimate • Automate question administration, data recording, scoring, and prompt reporting • Allows for immediate feedback

  16. CAT Algorithm Administer Item of Median Difficulty (or Screening Item) Score Item Estimate Latent Trait (Theta) Termination Criterion Satisfied Choose and Administer Next Item with Maximum Information No Yes Stop

  17. Increase of Accuracy of Ability or Latent Trait Estimation in CAT For each item added to the test, the width of the interval decreases. Item 1-5 Item 1-4 Item 1-3 Item 1-2 Item 1 Ability ()

  18. Potential Problems with CAT in Pain and Health Outcomes Measurement • Context effects • Unbalanced content • Time frame • Response categories • Multidimensionality

  19. What kind of short form? • Question 1 • 0 I do not feel sad. • 1 I feel sad • 2 I am sad all the time and I can’t snap out of it. • 3 I am so sad or unhappy that I can’t stand it. Are you basically satisfied with your life? True/False

  20. Item production Item statistics Item exposure Maintaining a valid bank of items for test construction Fairness Delivery options Effects of modes of administration Cost-benefit considerations MORE Research Still Needed for Effective CAT Implementation

  21. Infrastructure of a National Geriatric Pain Item Bank Subscriber Public Individual Researchers Pharm. Industries Non-profit Institutions Government Agencies Customized Information Retrieval; CAT; (automated) Brief Form National “Central” Item Bank Collector Analyzer Builder Retriever Consortium Approval IRT Analyses Item Parameters

  22. An Integrated Solution for Pain and Outcomes Assessments Chang, C.-H., & Yang, D. (2003, April 15). Patient-Reported Outcomes Information Technology: The PROsITTM System. ISPOR CONNECTIONS, 9(2), 5-6.

More Related