Rating Health Information Websites

1. Consumer Reports WebWatch Trust or Consequence Berkeley, California June 9, 2005 Rating Health Information Websites Peter G. Goldschmidt, President Health Improvement Institute Bethesda, Maryland

2. Presentation 1. Introduction • HII-CRW partnership • Independent ratings • Project/ratings philosophy • Types of health websites 2. Methods 3. Results 4. Conclusions

3. 1A HII-CRW Partnership Health Improvement Institute • 1990, founded; non-profit 501(c)3 education/research charitable organization • 1991, established Aesculapius Awards for excellence in health communication • 1997, held workshop on quality of health information on the Internet • 2003, entered into partnership with CU/CRW Purpose/objectives regarding health websites • To enable consumers to be more effective users • To provide consumer with independent ratings Medical Library Association is actively supporting project

4. 1B Independent Ratings Many past/present activities • Criteria • American Medical Association • Mitretek Systems • Codes of conduct • eHealth Code of Ethics • HI-ETHICS • Trustmarks/seals • Health on the Net Foundation (HONcode) • URAC accreditation Growing literature on criteria, quality of health websites

5. 1C Project/Ratings Philosophy Health information websites must be • Transparent regarding • Identity, contact information • Ownership • Privacy (or its invasion) • Easy to use regarding • Finding/navigating contents on website • Comprehending contents (clarity of writing, reading level, etc) • Printing contents of interest • Serving consumers with special needs • Meticulous in distinguishing advertising/selling from contents • Editorially adequate, for example, disclose & be appropriate regarding • Providing authors' credentials • Selecting/grading & editing contents • Resulting content • Accurate, complete, objective, balanced • Well-organized • coherent/clear; not muddled • Referenced • Current • Useful to consumers

6. 1D Types of Health Websites • Health communication • Health information • Decision support tool • Health ratings • Health information resource • Health website search engine • Health advice on-line • Behavior modification • Behavior self-help • Disease management • On-line product sales/marketing • On-line pharmacy • On-line store • Health product marketing • Health care organization • Health plan enrollment/transaction • Health care provider • Producer of health resources • Public health program • Other health care organization To start, focus is "health information websites"

7. 2 Health Website Rating Methods Enter into partnership HII-CRW Select websites to be rated Select additional websites to be rated Develop health website rating instrument Rate health websites Refine methods Rate additional websites Develop concepts/ approach Analyze feedback Rerate rated websites Credential/ select raters Add/update ratings website displays Display ratings Design ratings website

8. 2A Ratings Concepts/Strategy 3 levels for health information websites • Transparency/accountability • Editorial adequacy • Information reliability — completeness/accuracy of what is stated in any medium for any audience — excluded from present project because • Website may contain information on very many subjects • To rate validity of health information for given subject requires panel of qualified medical/research experts • Ultimately, assessments of reliability/validity of health information reflect state of medical science

9. 2B Select Websites To start • Define health information websites • Identify 100 most-visited "health" websites (A. C. Nielson) • Select top-20 "health information websites" To continue • Rate additional top-100 websites • Rate health websites suggested by consumers Periodically, rerate rated websites

10. 2C Develop Ratings Instrument • Developed general instrument applicable to rating all types of websites (CRW principles) • Adapted CRW instrument to meet project purposes • Created "Part I" after evaluation by HII volunteers of adapted CRW instrument (& compiled generic/specific criteria) • Pretested/revised Part I Consumer Reports WebWatch Health Improvement Institute • Analyzed/assessed criteria sets intended to evaluate health websites • Compiled list of (generic & specific) criteria • Asked HII volunteers to evaluate criteria • Created "Part II" to focus on specific criteria • Piloted/refined Part II

11. 2C Contents of Ratings Instrument I Website transparency/accountability II Health information editorial policies III HONcode compliance IV Raters' feedback

12. 2C HWRI, Part I, CRW Principles Developed/applied by CRW staff • Identity • Advertising & sponsorships • Ease of use • Corrections & currency • Privacy

13. 2C HWRI, Part II, HII principles Developed by HII; applied by volunteers A. Criteria/descriptors • Characteristics of website contents • Accessibility of contents to consumers • Editorial policies/procedures • Authors of articles/contents • Articles • Summary scores • Optional • Raters could also complete Part-I (to facilitate construction of evaluative narratives) B. Evaluative narratives • Purpose/scope & intended audience • Characteristics of website • Accessibility • Editorial policies/procedures • Contents • Website's greatest strengths • Website's greatest weaknesses • Utility of website to consumers

14. 2D Credential/Select Raters Process • Call for raters • Request applicants present their credentials • Submit applicants' credentials to raters credentialing committee • Credential health website raters • Select credentialed raters for panel • Type of organization in which employed currently • Role • Professional background • Region of country Credentialing criteria • Health professional • At least 5 years continuous relevant experience • Currently active in health field • Sufficient qualifications/experience to evaluate health websites • No apparent disqualifying event • After initial credentialing, satisfactory performance as rater

15. 2E Rate Websites • CRW staff applied HWRI, Part I • Panel of HII volunteers; each applied HWRI • Raters signed HII "Policy on Conflict of Interest“ • If conflict of interest, website reassigned • HII summarized individual panel members' scores/assessments to produce coordinated ratings • HII/CRW integrated ratings for website display • CRW created ratings website displays

16. 2F Create Ratings Website • Purpose/objectives • To display independent health website ratings in consumer-friendly way • To solicit feedback • Scope —website contains • Introductory/explanatory material • Description of methods • Disclosures/disclaimers • Ratings page for each rated website • Ratings include • Website's stated purpose • HII-CRW description of website • Global/attribute scores • Greatest strengths/weaknesses • Noteworthy items

17. 3 Results • Rating health websites • Rated websites • Median attribute ratings • Distribution of ratings • Excellent top-20 health websites • Example of ratings webpage Ratings are accessible on www.healthratings.org

18. 3A Rating Health Websites Process • 20 most-visited health websites • Rated for • Transparency, 2 CRW raters • Editorial policies, HII raters • 14 HII raters (3-member panels) • 2 health practitioners • 5 health information experts • 2 health education specialists • 5 media, production & related • Resultant ratings • Excellent • Very good • Good • Fair • Poor Attributes • Identity • Advertising & sponsorship • Ease of use • Corrections & currency • Privacy • Design • Coverage • Accessibility (navigation/reading level) • Contents • Overall rating

19. 3B Rated Websites (20 most-visited) • webmd.com • nih.gov • health.yahoo.com • about.com/health • mayoclinic.com • medicinenet.com • emedicine.com • drugs.com • intelihealth.com • pfizer.com • realage.com • kidshealth.org • rxlist.com • qualityhealth.com • healthology.com • health.ivillage.com • medscape.com • heartcenteronline.com • healthboards.com • healthsquare.com

20. 3C Median Attribute Ratings • Identity - Excellent • Advertising & sponsorship - Excellent • Ease of use - Good • Corrections & currency - Fair • Privacy – Excellent • Design – Good • Coverage – Very good • Accessibility – Very good • Contents – Very good

21. 3D Distribution of Ratings Top 20 Health Websites Excellent (30%) Very Good (25%) (40%) Good Fair (6%) Poor (0%)

22. 3E Excellent Top-20 Health Websites • emedicine.net • kidshealth.org • mayoclinic.com • medscape.com • nih.gov • webmd.com Ratings are accessible on www.healthratings.org

23. 3F Website Ratings Page Example

24. 4 Conclusions • Designing, producing & updating excellent health information websites is costly, complex, challenging • Rating websites is equally challenging, but desirable & feasible • 6 of 20 most visited health websites, were rated “excellent” overall • Quality of information is limited by state of medical science

25. 4A Conclusions: Needed improvements • Generally, rated websites need to improve • Contents - descriptions of editorial policies; also policies & procedures • Describe how select topics, search/grade information, develop contents, assure quality of articles/contents • Name authors/reviewers • Provide authors/reviewers’ credentials; must be appropriate to contents; disclose financial/other interests • State date last reviewed/updated • Refer to sources of facts/citations • Indicate criteria for linking to other websites • Design & Ease of use • Accessibility - especially for consumers with special needs • Currency & corrections

26. 4B Feedback • HII/CRW welcome feedback on • Ratings’ utility to consumers • Suggestions for improving ratings website, process, criteria, etc. • Health websites to be rated • Volunteer to rate health websites!