Identifying and Selecting Measures for Health Disparities Research

1. Identifying and Selecting Measures for Health Disparities Research Anita L. Stewart, Ph.D. University of California, San Francisco Clinical Research with Diverse Communities EPI 222, Spring May 8, 2007

2. Organization of Class 5 • How to select measures for your own studies of diverse populations • Measurement issues in research with diverse populations

3. Organization of Class 5 • How to select measures for your own studies of diverse populations • Measurement issues in research with diverse populations

4. Selecting Measures for Your Own Study: The Problem • You are beginning a study • You know the concepts (variables) of interest • Question: Which measure of ________ should I use? • A popular measure • One that a colleague used successfully • Create your own

5. Inappropriate Measures can Result in: • Conceptual inadequacy • Measuring wrong concept for your study • Poor data quality (e.g. missing data) • Poor variability • Poor reliability and validity • Inability to detect true associations • e.g., no measured change in outcome when change occurred

6. Two Types of Considerations in Selecting Measures • Contextual - factors unrelated to specific measurement tools • Characteristics of target population • Goals of research • Practical constraints • Psychometric - properties of measures in your context

7. Basic Steps in Selecting Appropriate Measures 1. Specify context 2. Define concept for your study 3. Locate potential measures for consideration 4. Review potential measures for: a) conceptual match to your definition b) adequate psychometric properties in your target group 5. Pretest potential measures in your target group 6. Choose best ones based on pretest results OR 7. Adapt if necessary to address problems

8. 1. Specify Context • Research question and how concept fits research (outcome, predictor, covariate) • Nature of target population (health, age, SES, race/ethnicity, literacy) • Practical constraints (time, personnel, budget, respondent burden)

9. Step 2: Define Each Concept ForYour Study • Define each concept from your perspective, taking into account your.. • study questions • target population • For outcomes: • describe how the intervention or independent variables might affect it • describe specific types of changes you expect

10. Step 3. Locate Potential Measures • Identify candidate measures for all domains or concepts in your framework • For health outcomes: • generic or condition-specific profiles of multiple domains OR measures of single domains • Redundancy OK for now • Do NOT develop your own questions unless it is absolutely necessary

11. Locating Measures • Compendia • Web • Organizations • National surveys • Large research studies • Many other sources

12. Locating Measures: Compendia • Compendia of measures (reviewed) • Books that compile various measures and review their characteristics (e.g., McDowell and Newell) • Web sources of Compendia • HaPI-Health and Psychosocial Instruments

13. Locating Measures: Organizations • RAND publishes measures, scoring manuals, and lists citations • Measures of quality of care, patient satisfaction • Measures of health-related quality of life • Generic and disease specific • All Medical Outcomes Study measures • www.rand.org/health/ • Go to “surveys and tools”

14. Inter-University Consortium for Political and Social Research • http://www.icpsr.umich.edu/ • Maintains archive of social science data • Membership-based organization – over 500 member colleges/universities • UCSF is a member • Can search website using keywords to locate studies, data, and questionnaires

15. MacArthur Research Network on Socioeconomic Status and Health • Has measures in several domains: • Psychosocial • Social and physical environment • Socioeconomic status (SES) • SES across the lifecourse • MacArthur Network on SES and Health • www.macses.ucsf.edu/research/overview.htm

16. Anxiety Coping Depression Discrimination Hostility Optimism/pessimism Personal control Psychological stress Purpose in life Self-esteem Social support Vitality and vigor Examples of Psychosocial Measures http://www.macses.ucsf.edu/Research/wgps.htm

17. Individual level Economic status Occupation Education Sociodemographic characteristics Neighborhood level Residential segregation Physical environment Stress in environment Availability of healthy foods Examples of Environmental Measures http://www.macses.ucsf.edu/Research/Social%20Environment/chapters.html

18. Ottawa Health Research Institute: Goals • Explore ways to help patients make "tough" healthcare decisions (multiple options, uncertain outcomes, benefits/harms that people value differently) • Measure and understand decision support needs of people and patients, particularly disadvantaged groups, and their providers • e.g., decisional conflict, decision self-efficacy, stages of decision making www.decisionaid.ohra.ca/eval.html

19. Locating Measures: National and State Surveys • National Center for Health Statistics • Surveys and data collection systems • NHIS, NHANES, HHANES, BRFSS, CHIS, MEPS • Entire surveys can be downloaded • Seldom includes measurement information

20. National Center for Health Statistics • National Health Interview Survey • National Health and Nutrition Examination Survey • National Health Care Survey • Ambulatory health care data (NAMCS) • National Home and Hospice Care Survey • National Survey of Family Growth • National Maternal and Infant Health Survey • Longitudinal Studies of Aging (LSOA) • www.cdc.gov/nchs/express.htm

21. Locating Measures: National Agencies • Agency for Healthcare Research and Quality (AHRQ) • National Quality Measures Clearinghouse • Consumer Assessment of Health Plans Survey (CAHPS) • National Healthcare Quality Report • National Healthcare Disparities Report

22. Locating Measures: National Agencies • National Institutes of Health • NCI has a measurement initiative • Health Information National Trends Survey (HINTS) • Working group compiled cancer screening questions, identified best ones, conducted extensive pretesting, cognitive interviewing • Measures are on the NCI web site

23. Locating Measures: National Agencies • US Dept. of Veterans Affairs, National Center for PTSD • Has assessment section of web site • Variety of instruments to measure stress and trauma exposure www.ncptsd.va.gov/ncmain/assessment/

24. Locating Measures: Large Studies and Centers • Large research studies on similar topic • Health ABC, CARDIA, WHI, EPESE • Projects/Centers • Toolkit to measure end of life care (TIME) • Stanford Patient Education Research Center • Michigan Diabetes Research and Training Center • Resource Centers for Minority Aging Research

25. Locating Measures: Bibliographic Searches • Published measurement articles • Medline Searches • MESH headings or keywords: • health status indicators • outcome assessment (health care) • psychometrics (methods) • questionnaires

26. Locating Measures: Finding Authors of Measures • Published research using measure you are interested in • Unpublished measures often described in methods • Authors may provide measures

27. Step 4: Review Potential Measures for • Conceptual appropriateness & relevance • in your study • in target group • Psychometric adequacy in target group or groups • Practicality • Acceptability • To respondents and interviewers

28. Conceptual Relevance • Example: you are interested in reports of perceived discrimination in the health care setting • In reviewing measures of discrimination, most are about • Discrimination over the lifecourse • Discrimination in various life settings (work, school) • Not relevant for your purpose

29. Concept Depicted as a Measurement Model • Measurement model: • the dimensional structure of a measure • how the items related to the construct • Can be depicted as a list or visually

30. Measurement Models • Physical Functioning (4 items) • Psychological Distress (7 items)

31. Measurement Model (List format) • Physical Functioning defined in terms of: • Walking • Climbing stairs • Bending • Reaching

32. Measurement Model (Visual format) Physical Functioning Reaching Climbing Stairs Bending Walking

33. Measurement Model (List format) • Psychological distress • Depression • Sad • Lost interest • Can’t get going • Anxiety • Restless • Nervous

34. Measurement Model (Visual format) Psychological Distress Depression Anxiety Sad Lost interest Can’t get going Restless Nervous

35. Psychometric Adequacy for Your Study • In samples similar to yours: • good variability • low percent of missing data • good reliability • good validity • As an outcome for your planned intervention • responsive, sensitive to change in similar population • able to detect expected magnitude of change

36. Good Variability • All (or nearly all) scale levels are represented • Distribution approximates bell-shaped normal • No floor or ceiling effects • Scores “bunched” at either end

37. Reliability • Extent to which an observed score is free of random error • Population-specific; reliability increases with: • sample size • variability in scores (dispersion) • a person’s level on the scale

38. Reliability Coefficient • Typically ranges from .00 - 1.00 • Higher scores indicate better reliability • Types of reliability tests • Internal-consistency • Test-retest • Inter-rater • Intra-rater

39. Internal Consistency Reliability: Cronbach’s Alpha • Requires multiple items measuring same construct • Extent to which items measure same construct (same latent variable) • It is a function of: • Number of items • Average correlation among items • Variability in your sample

40. Minimum Standardsfor Internal Consistency Reliability • For group comparisons (e.g., regression, correlational analyses) • .70 or above is minimum • .80 is optimal • above .90 is unnecessary • For individual assessment (e.g., treatment decisions) • .90 or above (.95) is preferred JC Nunnally, Psychometric Theory, McGraw-Hill, 1994

41. Validity • Does a measure (or instrument) measure what it is supposed to measure? • And…Does a measure NOT measure what it is NOT supposed to measure?

42. Validation of Measures is an Iterative, Lengthy Process • Validity is not a property of the measure • validity is a property of a measure for particular purpose and sample • validation studies for one purpose and sample may not serve another purpose or sample • Accumulation of evidence: • Different samples • Longitudinal designs

43. Three Major Forms of Measurement Validity • Content • Criterion • Construct

44. Construct Validity Basics A process of answering the following questions: • What is the hypothesis? • What are the results? • Do the results support (confirm) the hypothesis?

45. Construct Validity: NOTE • Sometimes the hypothesis is that the measure will NOT be correlated with certain other measures, or will be less correlated with some than with others • THUS, observing a low or non-significant correlation can confirm construct validity

46. Limited Data on Psychometric Properties of Many Measures • Not easy to find this information • Many studies do not report any psychometric properties • Assume the properties from original study carry over

47. Limited Data on Psychometric Properties of Many Measures (cont.) • Especially in diverse populations: • Few studies test measures across diverse groups • Even when diverse groups are included in research • sample sizes usually too small to conduct measurement studies by subgroups

48. Review Measures for Practicality • Method of administration appropriate for your study • Costs of administration within study resources • Scoring rules clearly documented • Measure available at cost you can afford • You are allowed to adapt it if necessary • Translations available if needed

49. Practical - Scoring • Know ahead of time how to score items • Count of “correct” answers? Summated scale? Weighted? • Are scoring instructions or computer scoring programs available? • Can scoring programs be purchased from developers? • Do you have a scoring codebook?

50. Review Measures for Availability of Translations if Needed • If you need the questionnaire in another language, are there translations available? • Official (published and tested) • Unofficial (by some other researcher) • If not, you have to conduct translations • Use state-of-the-art methods