Scope, Completeness, and Accuracy of Drug Information in Wikipedia Clauson et al.

1. Scope, Completeness, and Accuracy of Drug Information in WikipediaClauson et al. Presented by Adam Martin on September 27, 201o

2. Background Scope, Completeness, and Accuracy of Drug Information in Wikipedia. Kevin A Clauson, Hyla H Polen, Maged N KamelBoulos, and Joan H Dzenowagis. Annals of Pharmacotherapy, Dec 2008; 42: 1814 - 1821. • Clauson is Associate Professor in the Pharmacy Practice Department at Nova Southeastern University, College of Pharmacy. • Interested in informatics: • “Pharmacists' duty to warn in the age of social media” • “Readability of patient and health care professional targeted dietary supplement leaflets used for diabetes and chronic fatigue syndrome” • “Performance of online drug information databases as clinical decision support tools in infectious disease medication management” • Annals of Pharmacotherapy impact factor: 2.45

3. Relationship to Other Literature • Health Information (Quality and Seeking Behavior) • Giles J. Internet encyclopaedias go head to head. Nature 2005;438:900-1. • “The average science entry in Wikipedia contained around four inaccuracies; Britannica, about three.” • Spate of Wikipedia articles in JAMIA: • Seeking Health Information Online: Does Wikipedia Matter? • “Based on its search engine ranking and page view statistics, the English Wikipedia is a prominent source of online health information compared to the other online health information providers studied.” • Wikipedia and osteosarcoma: a trustworthy patients' information? • “The quality of osteosarcoma-related information found in the English Wikipedia is good but inferior to the patient information provided by the NCI”

4. Motivating the Study • “Consumers search the Internet using suboptimal techniques”(1814) • “Wikipedia is a frequently accessed resource by consumers seeking health information on the Web”(1814) • “When information about drugs is in written form, it can impact patients’ attitudes and behaviors”(1815) • “(1) many articles are written anonymously, (2) many contributors are students still studying the very topics about which they write, (3) some Wikipedia authors self-identify as ‘pre-hospital care workers,’ and (4) anyone can purposefully add misinformation”(1815)

5. Purpose of the Study • “To evaluate the scope, completeness, and accuracy of Wikipedia for content that one-third of all Internet health-seekers search for: information on medications” (1815)

6. On Method • Comparator Database: • “had to be a freely accessible general drug information database that had been previously evaluated and found to be broad in both scope and depth of drug information that was gathered from authoritative references”(1815) and non-wiki. • Medscape Drug Reference was selected as comparator. • Categories and Questions: • Eight Categories: “administration, adverse drug events, contraindications, dosage, drug interactions, indications, mechanism of action, and use in pregnancy and lactation”(1815) • Each with ten questions covering topics like “inpatient and outpatient care, non–Food and Drug Administration (FDA)-approved indications, novel medications, and recently approved dosage changes”(1816) • “Answers were verified with gold standard and authoritative references used in general practice in the US, not including the comparator database”(1816)

7. More on Method • Evaluation: • Scope: “If an answer was present, a score of 1 was assigned; conversely, if an answer was absent, a score of zero was given and it was also classified as an error of omission.”(1816) • Accuracy: “Factually erroneous answers (ie, those that conflicted with the answer key) were also counted and characterized.”(1816) • Completeness: “Completeness was assessed using a 3-point scale, with 3 being the most complete and 1 being the leas complete.”(1816)

8. Still More on Method • Authors also evaluated previous previous Wikipedia entries. • Compared “current” to the entries of 90 days prior. • Only evaluated scope and completeness. Why not accuracy?

9. Results: Scope • “Wikipedia was able to answer significantly fewer (40.0%) of the total number of drug information questions compared with MDR (82.5%; p < 0.001).”(1816) • “Wikipedia was able to answer more questions in the indication category (60.0%) than was MDR (50.0%)”.(1816) • It “tied MDR in the scope score for the mechanism of action category (80.0%)”.(1816) • “Wikipedia was unable to answer any questions on dosage, versus the MDR score of 90.0%”.(1816)

10. Results: Completeness • “Answers in Wikipedia were significantly less complete compared with those of MDR (p < 0.001).”(1817) • “The answers that Wikipedia was able to provide were only 76.0% complete, while MDR was able to provide answers that were 95.5% complete.”(1817) • “Wikipedia did not outscore MDR in completeness for any of the drug information categories.”(1817)

11. Results: Accuracy • “No factually inaccurate answers were provided by Wikipedia, whereas 4 answers in MDR conflicted with the answer key.”(1817) • In MDR, the “two errors were due to lack of timely updates for newly approved FDA indications, and 2 errors occurred because the database provided conflicting information in different sections of the monograph.”(1817) • “Fourteen errors of omission were recorded for MDR and 48 errors of omission were noted with Wikipedia.”(1817) • Ommission breakdown: contraindications = 7, pregnancy and lactation = 7, drug interactions = 6, adverse drug reactions = 5

12. Results: Current vs. Prior Wiki Entries • “Scope scores between current and prior Wikipedia evaluations were 40.0% and 33.8%, respectively”(1817) at p = .024. • “Prior page version overall completeness scored 76.5%, while current versions scored 76.0%”(1817) but p = .77.

13. Discussion • “While Wikipedia provided factually accurate drug information, it was incomplete, much more likely to contain errors of omission, and thus, of more limited scope than the information available in MDR.”(1817) • Wikis “may be perceived to be edited by altruistic ‘peers’ who improve the content with each edit. This trust may spill over to health and drug information, where it might be misplaced, given that commercial considerations can drive content and increase the risk of abuse.”(1818) • “An important drawback of user-edited sites is that free and anonymous editing potentially allows amateur or erroneous entries, fraudulent material, or conflict of interest (whether by inclusion or omission) to go unnoticed or unchallenged. In particular, the increasing use of nonvalidated resources is thought to pose a risk for consumers”(1818)

14. Limitations and Reflections • “They didn't really give a reason WHY the change in scope MAY have increased during the 90 days time, or if it would be expected to continue.” – LK • “Why was the comparison database needed instead of just comparing Wikipedia to the gold standard references?” – MS • “This article seems to say we should denigrate wikipedia as a source. I would argue instead we should be educating people on how to search and recognize authoritative and appropriate sources”. – BH • The study concerns information seeking behavior but doesn’t take into account actual consumer questions or behaviors (e.g., ability to negotiate various interfaces).

15. Limitations and Reflections, Cont’d • “It is disturbing that due to search engine manipulation, wiki may be one of the very first "hits" a consumer gets when searching, so publicizing sites like Pub Drug, RX WIKI, etc. should have more emphasis for the general public to navigate to these sites rather than a general non-health credentialled site like wiki. The issue if that with the number of intranet users geometrically increasing, it is like letting people drive without a license, and then trying to teach them the rules of the road - how can you possibly catch up at this point? There does need to be some input by health professionals and organizations like AMA, or American Academy of Family physicians, or American Nurses Association…” – LK • “My biggest problem is that wikipedia is a reference source, often pointing you to good sources of information. So, when they evaluated the answers did they read all the linked materials provided in that entry? Do we know if average consumers do this or not?” – BH • “Previously, physicians and pharmacists have been considered the "keepers of the information", now the explosion of information all over the web, has changed that role. Who is responsible for providing the literacy? Proliferation of many authoritative sites often makes it difficult too, which one to choose?” - LK