Self-Reported Utilization of Health Care Services: Improving Measurement and Accuracy

1. Self-Reported Utilization of Health Care Services: Improving Measurement and Accuracy Aman Bhandari Todd H. Wagner

2. Collecting Health Care Utilization • Costly and time consuming • No gold standard method • Administrative data are incomplete / inaccurate • Limited benefits • Out-of-plan or out-of-pocket utilization • Capitated health plans

3. Using Self Report • Primary data collection • Complement to administrative data • Substitute for administrative data (more attractive given HIPAA) • Secondary data analyses • NHIS • MEPS • AHEAD/HRS

4. Study Motivation • Widespread use of self-reported utilization • Use in VA CSP trials • Under what conditions is it accurate? • Can you improve accuracy?

5. Study Goals • Review literature • Health services • Psychology • “Develop” a cognitive model for understanding self-report • Create “standards,” if possible

6. Literature Review • Databases • BIOSIS from 1969-2003 • The Cochrane Library • Current Contents from 1993-2003 • Medline from 1966-2003 • PsycINFO from 1872-2003 • Web of Science from 1945-2003

7. Search Criteria • Keywords included: • “interviews,” “questionnaires,” “recall,” “self assessment,” “self-report,” or “survey design.” • and “utilization” or “health care utilization” • Reviewed abstracts to identify articles • Reviewed bibliographies for additional citations

8. Inclusion Criteria • Articles or book chapters • Self report compared to • archival data • computerized medical record • financial record • medical chart abstraction • or other administrative record

9. Exclusion Criteria • Non-English articles • Diagnostic tests • Medication use • Preventive care dosing and name recognition

10. Articles Reviewed

11. What is Self Report? • Cognitive process of recalling information • Ample opportunity for distortion and error (Khilstrom et. al 2000) • Implied assumption: self-report not valid when people lack the cognitive capabilities • How do you define and measure capabilities

12. Conceptual Model

13. Fixed Attributes • Process influenced by illnesses or disabilities (e.g., dementia or mental retardation) • Older age is consistently correlated with poorer recall accuracy (spurious correlation) • Older adults more likely to under-report.

14. Modifiable Attributes • Recall timeframe • Type of utilization • Utilization frequency • Questionnaire design • Mode of data collection

15. Recall Timeframe • Longer recall times result in worse accuracy • Longer timeframes lead to • Telescoping • Memory decay

16. Timeframe Note: Agreement based on authors’ definitions

17. Type of Utilization • Stigma • Social devaluation; includes embarrassment or humiliation • Under reporting due to “forgetting” • Salience: • Stands out in memory for being unusual (e.g., hospitalization) • Higher salience = higher accuracy.

18. Utilization Frequency • Under-reporting is exacerbated with increased utilization • As the number of visits increase, people forget some

19. Questionnaire Design “How many times have you seen a physician in the past 6 months?” • What is a time? What about multiple times on same day? • What is a physician? Does a nurse count? • Is “seen” literal? What about a phone consultation with prescription? • What about care for someone else?

20. Examples • During the past 12 months, how many times have you seen a doctor or other health care professional about your own health at a doctor's office, a clinic, or some other place? Do not include times you were hospitalized overnight, visits to hospital emergency rooms, home visits, or telephone calls. (NHIS) • In the last 12 months (not counting times you went to an emergency room), how many times did you go to a doctor’s office or clinic to get care for yourself? (MEPS)

21. Design: wording • Recall order • Chronological: go back a year and think forward • Reverse chronological: supposition: later events are the easiest to recall and helps recall previous events • Free recall • Data are inconclusive; unclear whether this varies by gender or culture

22. Design: motivation • Motivation to participate ≠ motivation to complete a survey

23. Data Collection • Modes: mail, telephone, Internet, and in-person data • No study has compared all four • Probing with memory aids can help improve accuracy • Stigma is important

24. Recommendations 1: Are respondents able to self-report • Consider age and cognitive capacity • 14 is lower limit • Use cognitive screening tool, such as MMSE

25. Timeframe, Frequency and Type of Utilization • Avoid recall timeframes greater than 12 months • Shorter recall may be necessary for • Office visits (low salience) • Frequent users • Consider two-timeframe method (i.e., 6-2)

26. Questionnaire Design • No standards exist (possible?) • Pretest: Dillman (2000) • Placement in questionnaire might matter

27. Mode of Data Collection • Other factors: timeline and budget • Phone, in person and some Internet surveys allow for memory aids • For example, landmark events

28. Response Scale • Use counts • Include “your best estimate is fine” • Avoid categories • 0, 1-2, 3-5, 6+ • Categories can introduce biases and error in the statistical analysis

29. Costs • Self-reported costs are assumed poor • Imputing costs from self-reports can introduce biases • Analyze visits, not just costs

30. Summary • No gold standard for measuring utilization • Self-report standards do not exist; may not be possible • Carefully design instrument • 5 modifiable attributes • Statistical analysis • Pretest