Presenting Health Statistics on the Web

1. Presenting Health Statistics on the Web • Bill Killam, MA CHFP

2. Acknowledgments • We would like to thank Dr. Holly Massett of the National Cancer Institute’s Office of Market Research and Evaluation, our contracting officer, for funding this project. • We would also like to thank Dr. Paul Han, also of the National Cancer Institute, for his desire to explore the effects of uncertainty and randomness on perception, which allowed us to pursue these topics.

3. Project Background • The National Cancer Institute (NCI) in its effort to disseminate information about cancer, uses both observed data and models to develop statistical data about cancer. • NCI has developed three mathematical models for estimating a persons risk for specific cancers including breast, melanoma, and colorectal cancer. • The colorectal cancer risk assessment tool (CCRAT) came under scrutiny by the press.

4. Our Goal • The project was initiated to address a concern about the CCRAT that was unrelated to its output. (An issue related to the scope of the tool’s data set.) • However, given the opportunity for some redesign, we were able to address other design concerns including issues of data entry and output display. • One of the NCI leads was specifically interested in how uncertainty and randomness affected peoples’ understanding and acceptance of statistical estimates.

5. Guidelines • Nine guidelines for presenting health related statistics (specifically for risk) had previously been identified: • Provide absolute and comparative risk information. • Specify the duration of risk. • Provide risk reduction strategies. • Address numeracy issues by providing risk both percentage and frequency formats. • Describe the risk using words and numbers. • Include a visual display of risk. • Acknowledge that the risk estimate contains an element of uncertainty. • Compare cancer risk to the risk of other hazards. • Frame the risk in positive and negative terms.

6. The Original Deign The original design of the CCRAT met only two of the 9 guidelines. It showed absolute and relative risk and it indicated the duration of the risk. (Risk reduction info was also available, but on a separate site and not integrated into the tool.)

7. The “Easy” Stuff • Addressing some of the guidelines was not a significant design issue. • Text changes in the current output page could address numeracy issues by describing risk using words and numbers. • Text changes in the current output page could also address numeracy issues by presenting risk in both percentage and frequency formats.

8. Guidelines That Were Not Addressed • Other guidelines could also be addressed with text changes in the output page: • Framing the risk in both positive and negative terms. • Comparing cancer risk to the risk of other hazards. • However, these two guidelines were considered less critical and there was no agreement in the need to follow them, so it was decided that these guidelines would not be accommodated.

9. The Not-So-Easy To Address Guidelines • The guideline to provide a visual display of risk has been previously researched. And there were some examples already online. • Icon arrays have been recommended by several studies, particularly those with human icons (Ancker et al, 2011 ; Garcia-Retamero et al, 2010). • Bar chart graphs had been recommended as more useful under certain conditions (McCaffery et al, 2012) and less useful under other conditions (Zikmund-Fisher et al, 2008).

10. Uncertainty • The value of reflecting uncertainty in calculations has also been researched (Han et al, 2006; Nadav-Greenberg & Joslyn, 2009). • In addition, some research showed that overly precise estimates, with or without uncertainty levels, may be considered less believable or credible and are less available for recall (Brasse et al, 202, Han, P. K., 2009, Lipkus, 2007, Witteman et al, 2011, Zikmund-Fisher, 2013). • However, specific recommendations for how to address uncertainty in visual displays are a bit less clear.

11. Uncertainty (concluded) • This approach had tested well and was used in another tool, but the users of this tool are physicians who generally have higher numeracy scores and have been exposed to concepts of confidence intervals and statistical error. We had done some prior work exploring a visual design for showing confidence intervals as floating bars on an otherwise standard bar chart (by replacing the error bars typical of a bar chart with a rectangle and removing the background).

12. Alternatives Evaluated We tried several approaches to add uncertainty to icon arrays. Some were determined to be completely unusable. We did test one concept with users where the lower CI value remain constant and the CI range dynamically filled in and was removed. Though the array was understood after an explanation was given, no one in testing picked it up on its meaning on their own. (There was also an interesting misinterpretation that may be present in all icon arrays.)

13. Alternatives Evaluated (cont.) We tried similar approaches to add uncertainty to bar graphs. Participants had more difficulty understanding uncertainty in this concept than in the icon array – even after an explanation was provided.

14. Alternatives Evaluated (concluded) Finally we tried an alternate approach similar to the bar graph shown to physicians. This approach tested very well, particularly when we added a visual indicator of decaying confidence. However, it was particularly interesting that the placement of the anchor points on the display was critical for participants to obtain a correct interpretation.

15. Risk Output Display The final output display for risk level was arranged vertically. The average was added onto the graph in a different style to help distinguish it. The scale was set to a constant value of 0 to 100% regardless of the risk level to allow the same design to be used and compared across all risk types (and to avoid the infamous “Gee Whiz” graph – an issue we had observed in a number of other projects and other designs). To address issues of screen size, a magnifier was used to view the risk detail while still maintaining its context on the scale.

16. Ensuring Attention to the Details Because of the large amount of information on the display, animation was used to direct the user’s attention to each data element. First, the scale was drawn, followed by the appearance of the average risk. The person’s calculated risk was then added. Finally, the magnifier appeared showing the risk in a more readable display.

17. Supporting Text for Risk Text was created to support the graphic. The text represented risk in two text formats – percentage and frequency out of 100. All calculated values were rounded to an integer level. A relative reference was made to the average risk. Different versions of the text for explaining uncertainty were evaluated. “Statistical calculations are not exact. We can only say that we are 95% sure the actual percentage of people who will get the cancer is somewhere in the range shown above.” “Calculations for the number of people who will get the cancer are not exact. However, the best estimate is that the value is somewhere in the range shown above. This also means that there is a possibility that the value is higher or lower than the range shown.” “Estimates are not exact. Your risk for developing colorectal cancer during your lifetime is most likely in the range of X%-X% (or somewhere between X and X out of every 100 people), but may be higher or lower. This means your risk is [higher/lower/the same as] than the average risk for all white males over the age of 55 - which is approximately X%.”

18. The Combined Risk Display Testing demonstrated that the text and the visual display were completely redundant. In other words, participants appeared able to understand all of the data from either the text or the graph. Note: We added bolding to the text to support skimming.

19. Randomness (Chance) • The issue of randomness was a specific interest of one of the stakeholders of the project. • The incorrect interpretation of the icon array showing uncertainly suggested observers may not be conceptualizing the randomness element of the calculated risk. • Data on how to display of the concept of randomness was the least well-researched issue.

20. Conceptualizing Randomness To address the concept of randomness, we returned to the icon array. We developed an animated icon array to show the concept of randomness. In this array, a random number of icons matching the calculated risk value are displayed. Then, after a short delay, the first set of icons fade out and a different random set are displayed. The uncertainty of the risk value is represented as well – the number of icons displayed any given moment is a random number within the confidence interval.

21. Supporting Randomness Text Several versions of the text for explaining randomness were also evaluated. This proved to be one of the most difficult aspects of this project. “The value shown describes what happens to a group of people: It does not tell what will happen to a specific individual.” “The calculation above is true for a large group of people: It cannot be applied to a specific individual. Think about tossing a coin. Statistics tells us that out of 100 times a coin is tossed it will land "heads" about 50 times. However, if you looked at one example of the coin from that group of tosses, it would be impossible to predict which way that particular toss had landed.” “We can calculate that a certain number out of 100 similar people will get a type of cancer, but we cannot tell which of the 100 people will get the cancer.” “Though we estimate that somewhere between 5 and 13 out of every 100 people will get [xxx] cancer, we can't tell for any one person if they will get it or not.”

22. The Combined Randomness Display The selected text was added below the graph. However, unlike the display of the risk value, the text and the visual display describing randomness performed differently. Participants initially looked at the icon array and did not understand it. Then they read the text and the icon array made sense. We considered removing the icon array but participants emphatically stated it was valuable and didn’t mind having to read the text to understand it.

23. The Final Layout In the final layout, users were able to select different time frames and see what factors determined their risk. Finally, risk mitigation factors were added to the primary display. Selecting an item on the right side produced a modified version of the original estimate with the original estimate shown as a shadow display for direct comparison.

24. Risks Below 1% Addressing low levels of risk was an exception to some of our own design guidelines. We displayed risk as a decimal value on the risk display, but described it in terms of rate out of 1000 instead of rate out of 100. This meant that risk values below 1% were simply lowered on the scale. The magnifier still provided the data at a readable level while maintaining a conceptual view of absolute risk.

25. Randomness Display at Values below 1% However, we modified the randomness display to be an array of 1000.

26. Summary • The final version was implemented and went live almost exactly as designed and evaluated. • The NCI elected to remove the risk mitigation factors from the display since the statisticians were not comfortable predicting the data based on changes in behavior. The risk mitigation factors were replaced with basic information on what increases or decreases a person’s risk.

27. Summary (concluded) • Of the three model-based risk calculators currently available from the NCI, this design is currently being used only for the colorectal cancer risk assessment tool (CCRAT). • However, additional research is ongoing to evaluate the effects of randomness and uncertainty using the CCRAT design.

28. Bibliography • Akl EA, Oxman AD, Herrin J. (2011). Using alternative statistical formats for presenting risks and risk reductions. Cochrane Database Syst Rev. 2011(3):CD006776. • Brasse G., Cosmides L., Tooby J. (1998). Individuation, counting, and statistical inference: the role of frequency and whole-object representations in judgment under uncertainty. Journal of Experimental Psychology. 127(1):3–21. • Edwards A, Elwyn G, Gwyn R. General practice registrar responses to the use of different risk communication tools in simulated consultations: a focus group study. BMJ Sep 18;319(7212):749-752 [FREE Full text] [Medline: 10488001] • Fagerlin A, Ubel P.A., Smith D.M., Zikmund-Fisher BJ. (2007). Making numbers matter: present and future research in risk communication. Am J Health Behavior; 31(Suppl 1):S47 S56. [Medline: 17931136] • Forrow L., Taylor W.C., Arnold R.M. (1992). Absolutely relative: how research results are summarized can affect treatment decisions. Am J Med, Feb; 92(2):121-124. [Medline: 1543193] [doi: 10.1016/0002-9343(92)90100-P] • Garcia-Retamero, R., gale sic, M. & Gigerenzer, G. (2010). Do Icon Arrays Help Reduce Denominator Neglect? Medical Decision Making. 30:672-684. • Han, P. K., Lehman, T. C., Massett, H., & Freeman, A. N., (2011). Communication of uncertainty regarding individualized cancer risk estimates: effects and influential factors. Medical Decision Making. 31(2):354-366. • Han P.K., Klein W.M., Killam B., Lehman T., Massett H., Freedman AN. Representing randomness in the communication of individualized cancer risk estimates: Effects on cancer risk perceptions, worry, and subjective uncertainty about risk. Patient Education and Counseling. Mar 3 2011. • Han, P. K., Moser, R. P., & Klein, W. M. (2006). Perceived ambiguity about cancer prevention recommendations: relationship to perceptions of cancer preventability, risk, and worry. Journal of Health Communication, 11 (Suppl. 1), 51-69. • Kahneman, D., & Tversky, A. (1982). Variants of uncertainty. Cognition, 11, 143-157.

29. Bibliography (concluded) • Lipkus I.M. (2007). Numeric, verbal, and visual formats of conveying health risks: suggested best practices and future recommendations. Medical Decision Making. 27(5):696–713. • Lipkus I.M. (2007). Numeric, verbal, and visual formats of conveying health risks: suggested best practices and future recommendations. Medical Decision Making. Sep-Oct 2007; 27(5):696-713. • Lipkus I.M., Hollands J.G. (1999). The visual communication of risk. Journal of the National Cancer Institute Monograph. (25):149–163. • McCaffery, K. J., Dixon, A., Hayen, A., Jansen, J., Smith, S., & Simpson, J. M. (2012). The Influence of Graphic Display Format on the Interpretations of Quantitative Risk Information among Adults with Lower Education and Literacy: A Randomized Experimental Study. Medical Decision Making. 27(32):532-544. • Trevena L, Zikmund-Fisher B, Edwards A, Gaissmaier W, Galesic M, Han P, King J, Lawson M, Linder S, Lipkus I, (2012). Ozanne E, Peters E, Timmermans D, Woloshin S. 2012. Presenting Probabilities. In Volk R & Llewellyn-Thomas H (eds). Update of the International Patient Decision Aids Standards (IPDAS) Collaboration’s Background Document. Chapter C. http://ipdas.ohri.ca/resources.html • Visschers V.H., Meertens R.M., Passchier W.W., de Vries N.N. (2009). Probability information in risk communication: a review of the research literature. Risk Anal. Feb; 29(2):267-287. • Waters, E. A., Sullivan, H. W., Nelson, W., & Hesse, B. W. (2009). What is my cancer risk? How internet-based cancer risk assessment tools communicate individualized risk estimates to the public: Content analysis. Journal of Medical Internet Research, 11(3), e33. • Waters E.A, Weinstein N.D., Colditz G.A., Emmons K. (2006) Formats for improving risk communication in medical tradeoff decisions. J Health Communications. Mar;11(2):167-182. [Medline: 16537286] [doi: 10.1080/10810730500526695]. • Wills CE, Holmes-Rovner M. (2003). Patient comprehension of information for shared treatment decision making: state of the art and future directions. Patient Educ Couns. Jul;50(3):285-290. [Medline: 12900101] [doi: 10.1016/S0738-3991(03)00051-X] • Witteman, H. O., Zikmund-Fisher, B. J., Waters, E. A., Gavaruzzi, T., & Fagerlin, A. (2011). Risk Estimates From an Online Risk Calculator Are More Believable and Recalled Better When Expressed as Integers. Journal of Medical Internet Research. 13(3):e54 • Yamagishi K. (1997). When a 12.86% mortality risk is more dangerous than a 24.14%: implications for risk communication. Applied Cognitive Psychology. 11(6):495–506. • Zickerman -Fisher, B. J., Fagerlin, A., & Ubel, P. A. (2010). Improving understanding of adjuvant therapy options by using simpler risk graphics. Cancer. 113(12):3382-90.