CSTE 2013 Conference, June 2013 Ava Amini, David Birnbaum, Bernard Black & David Hyman

1. Public Reporting of Hospital Infection Rates:Ranking the States on Credibility and User Friendliness CSTE2013 Conference, June2013 Ava Amini, David Birnbaum, Bernard Black & David Hyman

2. In 1999, the Institute of Medicine published To Err is Human, a review of the frequency of adverse outcomes. By the time a Healthcare Infection Control Practices Advisory Committee (HICPAC) guidance document was published (McKibben et al., 2005), four states had enacted legislation requiring public disclosure of healthcare-associated infection rates. By 2006, eight states had mandated reporting of healthcare-associated infection rates. The state-law mandated list kept growing. Federal initiatives effectively made the mandate universal. 2009 - HHS Requirement of State Action Plans 2011 - CMS IPPS rules HAI was unfamiliar territory for state health departments. History of Mandatory Public Reporting in the United States

3. Healthcare Facility Reporting to CMS via NHSN: Current and Proposed HHS Requirements

4. Improving our understanding of what works, for whom, in what settings Does the site meet the users needs? Does the information influence perceptions? What are the information seeking behaviors of different kinds of users? What do the higher performing places know or do that the lower performing hospitals don’t (a key feature being engagement of physicians)? How will people even know the information service is there? Information Delivery Knowledge Translation Group Dynamics and Organizational Development Informatics Communication of Scientific Information Who are the users? What kinds of people use this type of information source and for what purposes? HAI Program Sociology/Medical Anthropology Quality Sciences Does visiting HAI websites: influence expectations, influence level of trust, and/or promote Marketing Psychology Systems Engineering Risk Communication Does the process drive itself to be continuous, or are external motivators needed? Patient Expectations and Patient-Provider Communication Action to Improve communication with staff? Health Service Research Do HAI rates improve over time and do they improve more in areas with versus without reporting? Hospital Epidemiology How does communication skills of staff reinforce or reduce patient satisfaction relative to initial expectations and perceived or actual adverse events? How will we set systems to detect impact, and know what motivated change? Can we automate signal detection?

5. Key Unanswered Questions • What kinds of people use this type of information source and for what purposes? • What are the information seeking behaviors of different kinds of users? • How well does the site meet the users needs? • Elizabeth Borycki & André Kushniruk’s team • Bernie Black & David Hyman’s team

6. What Constitutes User Friendliness? • In General • Nielsen J. Usability engineering. New York: Academic Press; 1993. • Kushniruk AW, Patel VL. Cognitive and usability engineering methods for the evaluation of clinical information systems. Journal of Biomedical Informatics 2004;37:56-76. • Cleveland WS. Visualizing Data. Summit, NJ: Hobart Press; 1993. • Wheildon C. Type & layout: Are you communicating or just making shapes. Mentone, Australia: Worsley Press, 2005. • Krug S. Don’t make me think: A common sense approach to web usability, 2nd edition. Berkeley, CA: New Riders; 2008. • HAI-Specific • Mazor KM, Dodd KS. A qualitative study of consumers’ views on public reporting of health care-associated infections. American Journal of Medical Quality; 2009;24(5):412-418. • Mazor KM, Dodd KS, Kunches L. Communicating hospital infection data to the public: A study of consumer responses and preferences. American Journal of Medical Quality 2009;24(2):108-115.

7. What Contributes to Credibility?

8. First-draft Scoring Tool (fall 2012) User Friendliness Scoring Credibility Scoring 1. Make it easy to verify the accuracy of the information on your site. You can build web site credibility by providing third-party support (citations, references, source material) for information you present, especially if you link to this evidence. Even if people don't follow these links, you've shown confidence in your material. 2. Show that there’s a real organization behind your site. Showing that your web site is for a legitimate organization will boost the site's credibility. The easiest way to do this is by listing a physical address. Other features can also help, such as posting a photo of your offices or listing a membership with the chamber of commerce. 3. Highlight the expertise in your organization and in the content and services you provide. Do you have experts on your team? Are your contributors or service providers authorities? Be sure to give their credentials. Are you affiliated with a respected organization? Make that clear. Conversely, don't link to outside sites that are not credible. Your site becomes less credible by association. 4. Show that honest & trustworthy people stand behind your site… 5. Make it easy to contact you… 6. Design your site to make it look professional… 7. Make your site easy to use… 8. Update site often (at least show that its been reviewed recently)… 9. Avoid promotional content (adds, offers)… 10. Avoid errors of all types… typographical… broken links… 1. Free access? 2. Is consumer path to basic reports obvious? 3. Ease of finding data explanations? 4. Ease of website use for hospital comparison? 5. Introduction to the topic (what HAIs are… types…) 6. Is data meaning explained? 7. Explanation helps consumers integrate information from multiple indicators? 8. Uses numbers as well as graphs or symbols to convey numeric or statistical information? 9. Consumer can locate specific hospitals, sensible comparisons? 10. Format is simple & brief, not too much technical language? 11. Understandable & useful to different groups (average consumers, physicians & sophisticated consumers, infection control professionals, insurers, researchers)?

9. First-draft Scoring Tool (fall 2012) User Friendliness Scoring Credibility Scoring Make it easy to verify the accuracy of the information on your site. You can build web site credibility by providing third-party support (citations, references, source material) for information you present, especially if you link to this evidence. Even if people don't follow these links, you've shown confidence in your material. (1 = bad to 5 = excellent) 1 = No citations, references, or source material. 2 = No references within actual report, but website includes one to two links. 3 = Report or website include three or more references. 4 = Actual report includes list of several references, but does not have links to any of them. 5 = Actual report includes list of references, with links to one or more of them. Show that there’s a real organization behind your site. Showing that your web site is for a legitimate organization will boost the site's credibility. The easiest way to do this is by listing a physical address. Other features can also help, such as posting a photo of your offices or listing a membership with the chamber of commerce. (0 = no; 1 = yes) 0 = No 1 = Yes Highlight the expertise in your organization and in the content and services you provide. Do you have experts on your team? Are your contributors or service providers authorities? Be sure to give their credentials. Are you affiliated with a respected organization? Make that clear. Conversely, don't link to outside sites that are not credible. Your site becomes less credible by association. (1 = bad to 5 = excellent) 1 = No names of individuals. 2 = Difficult to find list of actual individuals; credentials may or may not be included. 3 = Report or website includes list of HAI Advisory Committee members, but all credentials are not included; somewhat difficult to locate. 4 = Report or website includes list of HAI Advisory Committee members, but all credentials are not included; relatively easy to locate. 5 = Report or website includes list of HAI Advisory Committee members and members’ credentials; easy to locate. Etc. Free access? 0 = No 1 = Yes Is consumer path to basic reports obvious? 1 = HAI report or website is not first search result. Very difficult to find actual report. 2 = HAI report or website is not first search result. Requires multiple clicks (at least 3) throughout website to locate actual report. Not necessarily clear or obvious path. 3 = HAI report or website is not first search result. May take more than one (but less than three clicks) to access report once on HAI website. Alternatively, may not necessarily be first search result, but within first five AND when click on link, user goes directly to HAI report page. 4 = First search result to state HAI website. It takes more than one additional click to access actual report, but steps to find report are clear. 5 = First search result directly to state HAI report or to HAI website, which contains clear and immediately-identifiable link Ease of finding data explanations? 1 = Little to no explanation, either within report itself or on state HAI website. 2 - Explanation not within actual report but on website. Some definitions provided, but little to nothing explaining results, significance, implications. 3 - Explanation within actual report or relatively easy to find on state HAI website. More than merely definitions; some background, explanation, significance of data. 4 - Explanation of infection and infection rates and statistical significance within actual report. 5 = Explanation within actual report. Provides explanation regarding calculation, interpretation, results. and analysis. Also, explanation is located prior to each specific chart or report instead provides references to charts in headers or otherwise makes it clear for user to understand which data the explanation is referencing. Etc.

10. First Draft Scoring Tool (cont.)

11. Credibility vs. Usability ScoresAugust 2012

12. What We’ve Learned About HAI Reporting • HAI Reporting is widespread but quality of report format varies widely. • Larger states do not necessarily do better than smaller states (e.g. NH vs. FL). • Rankings can change dramatically if states redesign their website (e.g. Washington) • Redesign of website or content is not always an improvement

13. Impact of Redesign

14. Next Steps: Refining the Scoring Tools • Separate “user friendliness into “usability” and “content” Does a site contain useful information vs. ease of finding content Dividing line often not clear • Scoring initially done by one graduate student; project now working on inter-rater reliability • rcontent = 0.87 rcredibility = 0.98 rusability = low (2 students) • Still a work in progress

15. Science of Public Reporting • Credibility score & User Friendliness score are positively correlated • Pearson coefficient 0.58 • Many states are not applying most of what already is known in the research literature. Nor learning from each other about best practices? • We don’t know whether interactive format, PDF static report format, or offering both options is more effective to convey HAI rate information

16. Select Publications Birnbaum D, Cummings MJ, Guyton K, Schlotter J, Kushniruk A. Designing Public Web Information Systems with Quality in Mind: Public Reporting of Hospital Performance Data. CLINICAL GOVERNANCE 2010;15(4):272-278. Bell S, Benneyan J, Best A, Birnbaum D, Borycki EM, Gallagher TH, Goeschel C, Jarvis B, Kushniruk AW, Mazor KM, Pronovost P, Sheps S. Mandatory Public Reporting: Build It and Who Will Come? STUD HEALTH TECHNOL INFORM 2011;164:346-52 (from Information Technology & Communications in Health 2011 international conference). Amini A, Birnbaum DW, Black B, Hyman DA. Public Reporting of Hospital Infection Rates: Ranking the States on Credibility and User Friendliness. STUD HEALTH TECHNOL INFORM 2013;183:87-92. DOI: 10.3233/978-1-61499-203-5-87. Hyman DA, Black BS. Public Reporting of Hospital Infection Rates: Not All Change is Progress. Northwestern University Law School Law and Economics Research Paper No. 12-21; Northwestern University Institute for Policy Research Working Paper 13-07; University of Illinois Law, Behavior & Social Sciences Paper No. LE 13-18 draft available from Social Science Research Network electronic library at http://ssrn.com/abstract= 2219510. Birnbaum D. Unraveling a Web of Confusion. CLINICAL GOVERNANCE 2013;(in press)

17. Senior Partners in our Universities Council • Sigall Bell, MD • Assistant Professor of Medicine, Harvard Medical School • Jim Benneyan, PhD • Professor, Northeastern University College of Engineering; Director, Center for Health Organization Transformation; Executive Director, New England VA Healthcare Systems Engineering Partnership • Allan Best, PhD • Senior Scientist, Centre for Clinical Epidemiology and Evaluation, Vancouver Coastal Health Research Institute; Clinical Professor, School of Population and Public Health, University of British Columbia; Managing Director, InSource. • Bernard Black, MS, JD • Chabraja Professor, Northwestern University Law School and Kellogg School of Management • David Birnbaum, PhD, MPH • Adjunct Professor, University of British Columbia School of Population and Public Health, and UBC School of Nursing; Adjunct Professor, University of Victoria School of Health Information Science; Principal, Applied Epidemiology; Manager, Washington State Dept. of Health Healthcare Associated Infections Program. • Elizabeth Borycki, RN, MN, PhD • Assistant Professor, University of Victoria School of Health Information Science • Thomas H. Gallagher, MD • Associate Professor of Medicine, University of Washington • Chris Goeschel, RN, MPA, MPS, ScD • Director of Patient Safety & Quality Initiatives, Manager of Operations, Johns Hopkins University Quality & Safety Research Group; Clinical Instructor, Johns Hopkins School of Nursing; Associate faculty, Johns Hopkins Bloomberg School of Public Health • Bill Jarvis, MD • Jason and Jarvis Associates • André Kushniruk, PhD • Professor, University of Victoria School of Health Information Science • Kathy Mazor, EdD • Associate Professor, University of Massachusetts Medical School • Peter Pronovost, MD, PhD, FCCM • Professor, Johns Hopkins Schools of Medicine, Nursing, and Bloomberg School of Public Health; Director, Johns Hopkins Quality & Safety Research Group • Sam Sheps, MD, MSc, FRCPC(C) • Professor, University of British Columbia School of Population and Public Health; Director, Western Regional Training Centre for Health Services Research