Consumers’ Appraisal of Anecdotal Accounts of Patient Experience

1. Consumers’ Appraisal of AnecdotalAccounts of Patient Experience AHRQ Annual Conference Steven C. Martino, PhD September 10, 2012

2. Customer Reviews of Products and Services are Omnipresent Online • Amazon.com first began allowing customers to post reviews of products in 1995 • Now almost all retail websites include them • 78% of internet users (58% of adults) have researched a product or service online, and 32% have posted a comment or review (Pew Internet & American Life Survey, 2011) • Reviews are a trusted source of information on product or service quality • Current retail industry focus is on increasing the relevance of online customer reviews

6. Narrative Accounts of Patient Experience are Increasingly Available Online • Dozens of commercial websites provide narrative accounts of patients’ experiences with their doctors (sometimes accompanied by ratings) • e.g., Vitals.com, HealthGrades, Yelp, Angie’s List, and Consumer Reports • RateMDs currently hosts narrative reviews of at least 1 in 6 practicing doctors in the U.S. (Gao et al., 2012) • Currently, the only equivalent in the public sector is NHS Choices (British National Health Service)

7. Currently, the Audience for these Sites is Small • 80% of internet users (59% of adults) have looked online for health information • 16% of internet users have consulted online rankings or reviews of doctors or other providers (4% have posted such reviews) • 15% have consulted rankings or reviews of hospitals or other medical facilities (3% have posted such reviews) Data source: Pew Internet & American Life Survey conducted Aug-Sep 2010

10. Promises and Limitations of Patient Narratives • Patient narratives often cover the same domains as standardized surveys of patient experience but in ways that can be more vivid, concrete, and engaging • Yet, the sample of respondents is often not representative (biased) and small (poor signal, low reliability) • On RateMDs, the average number of reviews per physician is three (Lagu et al., 2010) • If presented with standardized data, patient narratives could make unfamiliar, difficult-to-understand quality measures more salient and evaluable • Yet, anecdotal data may overwhelm more formal statistical information

11. Extracting Meaning from Narratives is aComplex Cognitive Task • Enduring Characteristics of the Doctor • Technical ability • Interpersonal skill • Enduring Characteristics of the Environment • Health system guidelines • Case load and mix Doctor’s Behavior • Unique Aspects of the Situation • Individual patient • Circumstantial context

12. Extracting Meaning from Narratives is aComplex Cognitive Task • Enduring Characteristics of the Doctor • Technical ability • Interpersonal skill Consumers try to infer this from this • Enduring Characteristics of the Environment • Health system guidelines • Case load and mix Doctor’s Behavior • Unique Aspects of the Situation • Individual patient • Circumstantial context

13. Cognitive Strategies that People Use to Discern Among Multiple Possible Causes of Behavior . . . • Typically, people look for • Consistency in behavior across time and situations: Has the doctor behaved a certain way repeatedly across time and situations? • Consensusin people’s experiences with the person: Do other patients report getting similar treatment from the doctor? • Distinctiveness of behavior: Is this doctor the only doctor to have behaved this way toward the patient?

14. . . . Are Not Necessarily Applicable in the Case ofPatient Narratives • Consumers often do not have the information needed to answer questions about consistency, consensus, and distinctiveness. • Usually limited to a small number of narratives • Do not get to “observe” the doctor’s behavior over multiple occasions with the same patient or get the patient’s view of other doctors • Narratives rarely mention situational factors that may have facilitated or constrained behavior; instead, they strongly imply dispositional causes

15. Consumers’ Values, Disposition, and Past Experiences May Affect Their Appraisal • Consumers differ in how much they value data on patient experience and the importance they ascribe to various facets of patient experience and doctor quality • Consumers with different decision-making styles may use narrative accounts differently • Consumers have different levels of experience with healthcare quality data

16. Overview of Research Questions • What cues do people use to judge the value and authenticity of patient narratives? • How salient (memorable) are patient narratives relative to standardized measures of healthcare quality? • How trustworthy and useful do consumers perceive narratives to be? What are bases for skepticism? • Do people’s preferred decision-making styles, health status, or past exposure to healthcare quality data affect their attention to and appraisal of patient narratives?

17. The SelectMD Website: Basic Design • A fictitious public reporting website custom built for this and related studies • Designed to be consistent with current public reports in terms of content, format, and functionality • Presents three types of data • Standardized patient experience measures (CAHPS) • Clinical process and outcome measures (HEDIS) • Patient narratives (reviews) • Hidden tracking system to monitor: • Click patterns • Time spent on each page

18. The SelectMD Website: Creating Realistic Patient Narratives • Modeled on actual narratives collected from RateMDs • Matched to actual narratives based on length, cognitive complexity, and clarity of writing • Pilot testing to compare actual (RateMD) narratives to a large set of fictitious ones constructed for this study • Matched to reported CAHPS scores: • Focused on areas of experience covered by CAHPS • Higher scores associated with more positive comments (but imperfectly) • Comments varied in emotional tone for all doctors

24. Study Participants Pretesting of Patient Narratives • Three rounds of testing, n = 8-9 per round • Convenience sample of internet users with recent healthcare experience, mix of demographic characteristics Online Experiment • Random sample drawn from Knowledge Networks online research panel • Representative national sample (>60,000 households) • Limited to households with computer-based internet access (89% of total panel), aged 25-64 years • Of 1,757 invitations, over 48% accepted (n = 849)

25. Experimental Design:Six Study Arms

26. Experimental Design:Six Study Arms

27. Study Measures • Questions asked of pretest participants • Emotional valence of narratives • Perceived informativeness & authenticity of narratives • Questions asked prior to MD Choice • Prior exposure to public reports on plans, hospitals, and doctors • Health status • Questions asked after MD Choice • Recall of measures • Perceived usefulness and trustworthiness of measures • Decision making styles

28. Qualities of Narratives Associated with Their Perceived Usefulness and Authenticity • Strongly worded narratives were perceived as less authentic (p = .001) but more informative (p < .001) than mildly worded narratives • Narratives about negative experiences were perceived as less authentic (p < .001) than narratives about positive experiences • Length of narratives was positively related to perceived authenticity, r (129) = 0.25, p = .005, and positively related to perceived informativeness, r (129) = 0.30, p = .001

29. Recall of Patient Narratives • Of Ss who were shown patient narratives on the SelectMD site, 86% recalled seeing them • Of Ss shown narratives plus CAHPS measures • 58% recalled both • 32% recalled narratives only • 4% recalled CAHPS measures only • 7% recalled neither • Of Ss shown narratives plus HEDIS measures • 29% recalled both • 53% recalled narratives only • 3% recalled HEDIS measures only • 15% recalled neither

30. Recall of Patient Narratives (cont.) • Ss low in avoidant decision-making were more likely to recall patient narratives than Ss high in avoidant decision-making (89% vs. 84%; p = .07) • Ss high in intuitive decision-making were more likely to recall patient narratives than Ss low in intuitive decision-making (88% vs. 83%; p = .10) • Ss high in intuitive decision-making and low on rational decision-making were especially likely (94%) to recall patient narratives

31. Recall of Patient Narratives (cont.) • 81% of Ss who recalled the narrative data provided open-ended data on what they remembered specifically

32. Perceived Usefulness of Patient Narratives • Among Ss who recalled seeing the narratives, 44% said that they were very useful for decision-making, 44% said somewhat useful, 9% said only a little useful, and 3% said not at all useful • Unexpectedly, Ss high in intuitive decision-making judged the narratives as less useful than did Ss low in intuitive decision-making (p = .06) • Ss with recent serious or chronic health conditions saw the narratives as less useful than Ss without those conditions (p = .01) • Past exposure to healthcare quality data was not related to perceived usefulness of the narratives (p = .34)

33. Trustworthiness of Patient Narratives • Ss perceptions of the usefulness and trustworthiness of narratives were strongly related, r = 0.59, N = 588, p <.001 • Among Ss who recalled seeing the narratives, 18% said that they were very trustworthy, 65% said somewhat trustworthy, 15% said only a little trustworthy, and 2% said not at all trustworthy • Decision-making style and prior exposure to healthcare quality data were unrelated to how trustworthy narratives were perceived to be • Ss with recent serious or chronic health conditions trusted narratives less than Ss without those conditions (p = .04)

34. Reasons Given for Mistrust of Patient Narratives * Given by 101 subjects who judged narratives to be “only a little” or “not at all” trustworthy

35. Attention Given to Patient Narratives • Overall, Ss (N = 593) read reviews of 17.1% (SD = 24.4) doctors (includes those who did not drill down to reviews) • Ss who drilled down to the reviews (N = 157) read the reviews of 30.2% (SD = 25.6) of doctors • Ss high in avoidant decision-making read reviews of a smaller percentage (11.2%) of doctors than Ss low in avoidant decision-making (14.9%; p = 0.10)

36. Attention Given to Patient Narratives (cont.) • Ss low in analytic decision-making and low in intuitive decision-making read reviews of a smaller percentage (11.7%) of doctors than Ss with other combinations of these two styles (who read the reviews of 17-20% of doctors; F = 3.2, p = .02) • Ss with prior exposure to healthcare quality information read reviews of a larger percentage of (20.3%) of doctors than Ss without prior exposure (15.1%; p = .01) • Ss with recent serious or chronic health conditions read reviews of a larger percentage of doctors (19%) than did Ss without such conditions (16%; p =.09)

37. Conclusions • People use cues in narratives to determine their value and relevance • When presented together, patient narratives appear to be more memorable (salient) than standardized data on quality • Most people trust patient narratives and find them useful for decision-making • A minority recognizes the limitation of patient narratives as a source of data on data on doctor quality • Positive and negative accounts of patient experience are recalled equally; no domain of patient experience is more salient than others • Attention to and evaluation of patient narratives depends on people’s decision-making tendencies, health status, and prior experience with healthcare quality data

38. Implications • It may be possible to increase healthcare consumers’ engagement in and use of reports of standardized quality data by incorporating patient narratives • Need to elicit patient narratives in a way that makes them most useful to consumers (requires attention to issues of reliability and validity) • Need to present narratives in a way that clarifies their value (as illustrative rather than representative), relevance (by building trust in the methods of elicitation and reporting) and limitations