Addressing Users’ Healthcare Needs through Personal Health Messages

1. Addressing Users’ Healthcare Needs through Personal Health Messages Presenter : Jason H.D. Cho Department of Computer Science, University of Illinois at Urbana-Champaign, Urbana, IL

2. Health Informatics and Data Science • Healthcare is becoming an emerging area. • Lots of data readily available! • Medical web forums, which we have used in this paper, typically spans millions of posts. • Traditionally, health informatics utilize Electronic Medical Records. • In 2006, less than 10% of the hospitals used EMR. By 2009, almost 50% of the hospitals started using EMR.

3. EMR and Medical Web ForumsWhy bother with medical web forums? • Electronic Medical Records traditionally used in health informatics. • Privacy issues. • Data not readily available. • In this talk, I’ll talk about how we can utilize personal health messages, or in our case, medical web forums, can be used to address similar problems that EMRs do. • I’ll talk about two works, each from different perspective: • Macro and Micro.

4. Addressing Users’ health needs • Macro Perspective • Learn what vast majority of users are saying.

5. Addressing Users’ health needs • Macro Perspective • Learn what vast majority of users are saying.

6. Addressing Users’ health needs • Macro Perspective • Learn what vast majority of users are saying.

7. Addressing Users’ health needs • Macro Perspective • Learn what vast majority of users are saying. Chee, 2011

8. Addressing Users’ health needs • Micro Perspective • Users would like to conduct their own research.

9. Addressing Users’ health needs • Micro Perspective • Users would like to conduct their own research.

10. In this talk… • I’ll present two works that address both the macro and micro perspective. • Macro Perspective : Comparative Effectiveness Research • Micro Perspective : Case Retrieval System (Under submission)

11. Aggregating Personal Health Messages for Scalable Comparative Effectiveness Research Jason H.D. Cho1,4, Vera Q.Z Liao1,4, Yunliang Jiang1,4,5, Bruce R. Schatz1,2,3,4 1Department of Computer Science, 2Institute of Genomic Biology, 3Department of Medical Information Science, 4University of Illinois at Urbana-Champaign, Urbana, IL 5Twitter, Inc., San Francisco, CA

12. Comparative Effectiveness Research • Generation and synthesis of evidence that compares the benefits and harms of alternative methods to prevent, diagnose, treat, and monitor clinical conditions or to improve the delivery of care. • The American Recovery and Reinvestment Act of 2009 (ARRA) allotted $1.1 billion to support CER. • Existing Approaches : • Randomized trials – precise, but expensive to conduct, generally not scalable. • Research reviews – scalable, but only utilize works done in existing literature. • Our approach: • Low cost, can generate hypotheses quickly, scalable • MedHelp (1 million health messages), Yahoo! Answers (10 million health messages), HealthBoards (1 million health messages)

13. General Technique & Terminologies Attitude of Context : sgn(Positive – Negative) = -1 So negative attitude Treatment Sentence Context

14. Our Approach • We have determined users’ sentiment towards treatment is a good indicator of effectiveness. • Users’ sentiment towards treatment : Summation of context attitude the user makes towards treatment of interest • Preference between the two treatments is defined if more people have more net positive sentiment towards a treatment than the other. • We introduce three different approaches to determine effectiveness based on users’ sentiments.

15. Individual Effectiveness Comparison Study Chemotherapy : + Hormonal Therapy : - Chemotherapy : - Hormonal Therapy : + Chemotherapy is preferred over hormonal therapy. • Compare authors who explicitly compare two treatments. • This approach is more precise since the person is directly comparing two treatments against each other. • However, not many patients compare two treatments directly. We can relax the definition of effectiveness. • The new approach should be consistent with individual effectiveness comparison study.

16. Population Effectiveness Comparison Study Chemotherapy : - Chemotherapy : + Hormonal Therapy : - Hormonal Therapy : + Chemotherapy is preferred over hormonal therapy • Compare groups of people who prefer treatment over those who do not. • This approach allows leveraging bigger pool of population cohort. • Both individual comparison and population comparison gave similar preference results on experiments we ran. • Allows us to run population effectiveness comparison in lieu of individual effectiveness comparison! Increases size of cohort pool by order of magnitude.

17. Demographics Effectiveness Comparison Beta Blocker : - Beta Blocker : + Beta Blocker : + Beta Blocker : - Older people prefer beta blockers than younger people do. • Different demographics may react differently to a given treatment. • We conducted population effectiveness comparison study on each demographic groups of interest. • Two types of comparison : • Cross-group Comparison : Compares against two different demographic groups on one treatment. • Within-group Comparison : Compares against two treatments on one demographics group. • Q : How do we extract patient’s demographics? Young Old

18. Demographic Extraction • Approach 1 : Utilize users’ Profile • What if user did not list demographic information? • We implemented rule-learning demographic extraction algorithm to solve this problem.

19. Demographic Extraction I am 30 years old, ... …, a 30 year old … He is 30 years old. Day 30 for me. I am 30 years old, ... …, a 30 year old … He is 30 years old. Day 30 for me. We introduce rule-learning algorithm to extract age. 1. Extract all phrases that match users’ profile page demographic information and mentions in health messages. 2. Run frequent sequence pattern mining algorithm (PrefixSpan) to mine frequent patterns. 3. Remove low precision frequent patterns

20. Demographic Extraction Performance Evaluation # Inferred & has Age # Inferred & no age Precision # Users Users w Age # Inferred Breast Cancer Heart Disease Our approach effectively removed most of the inferred age that is irrelevant compared to the baseline approach. This approach doubled the number of people with demographic information.

21. Our Findings • We used MedHelp forums as our data source, and selected forum categories based on diseases of interest. • We chose diseases and treatments to conduct experiments from Institute of Medicine’s 100 CER priority list. • test to determine preference significance. • Many of the findings were consistent with existing medical literature, such as those from Cochrane Reviews, Agency for Healthcare Research Quality (AHRQ) and New England Journal of Medicine. • We show some of the results that were statistically significant. • On population effectiveness comparison study, 50% of our findings were consistent with existing literature. The rest, we weren’t able to find literature that verified our claim.

22. How big is the cohort pool? • Population Effectiveness Comparison : • Generally each treatments had thousands of patients. • For breast cancer : Radiation (2,393), Chemotherapy (2,878), Hormonal Therapy (1,680) – approximately 7,000 patients • For heart disease : Anticoagulants (2,162), Inhibitor (2,422), Blocker (7,257), Device (2,457) – almost 15,000 patients

23. Population Effectiveness Comparison New England Journal of Medicine :Patients who had radiation therapy showed lower post-treatment side effects than those who had hormonal treatments. Cochrane Review : Chemotherapy is advantageous over hormonal therapy in reducing tumor response rate

24. Population Effectiveness Comparison New England Journal of Medicine: - Warfarin is at least as effective as beta blockers, but are often times more cost-effective. • New England Journal of Medicine : • Patients using devices (pacemakers, ICDs) often take Warfarin (anticoagulant).

25. Demographic Effectiveness Comparison Agency for Health Research and Quality :ACE inhibitors reduce composite efficacy endpoints similarly in males and females. Archives of Internal Medicine :Younger people have trends of being more impacted by cognitive impairment than older people.

26. Conclusion • We introduced how CER hypotheses can be generated using health messages. • We introduced how preference as measured by sentiment can be a good indicator of treatment effectiveness. • We also introduced high precision demographic extraction algorithm to broaden the cohort pool. • Personal health messages are scalable. MedHelp was used as our data source, but other forums can be aggregated to further broaden the cohort pool. • The results from our algorithm was consistent with existing medical literature.

27. Future Works • Investigate on signals that can be a good indicator of effectiveness (depth). • Entity relation semantics extraction to analyze relation between treatment and its aspects (effectiveness, side effects, etc’s) • Shallow Information Extraction approach can be utilized to determine whether subsection of forum text is about symptoms or treatments. • Merge multiple sources to leverage bigger cohort pool (breadth). • Other medical web forums, such as WebMD, HealthBoards. • Social networks and micro-blogs such as Facebook, Twitter and other sources.

28. Resolving Healthcare Forum Posts via Similar Thread Retrieval Jason H.D. Cho1,4, Parikshit Sondhi1,4, Chengxiang Zhai1,4, Bruce R. Schatz1,2,3,4 * Slides Courtesy of ParikshitSondhi 1Department of Computer Science, 2Institute of Genomic Biology, 3Department of Medical Information Science, 4University of Illinois at Urbana-Champaign, Urbana, IL

29. Case Retrieval Task • Users may often want to conduct research by themselves. • They may be curious about what disease they have, or which medications they may take. • Macro-tasks cannot take care of this, since it assumes users already know what they want already.

30. Query Characteristics • Queries meant for human experts not automated systems • Simple non-technical language • Presence of emotional statements

31. Document Characteristics

32. Envisioned Response • The following threads discuss similar problems: • Doritos Allergy Very Severe and New • Certain Foods + Beer = Flushing and Head Pounding…Help! • Peanut/Food Allergies • ……………………

33. Method Overview • Baseline Weighing • First Post BM-25 • Thread BM-25 • Semantic Weighing • Medical term extraction • Shallow Information Extraction • Post Weighing • Monotonic Weighing • Parabolic Weighing • Forum Category Weighing • Uniform Weighing (FCUW) • Feedback Weighing (FCFW)

34. Shallow Information Extraction I am severly allergic to some product that is found in both Tostitos and Doritos, as well as random other types of chips. I know the solution is "don't eat chips" but what could the product be?I don't want to accidentally consume it. When I eat this, I get very bad stomach cramps and it ruins the rest of my day/night - the only solution is to go to sleep so I can't feel it.Help! Any ideas on this? Sondhi, 2010 Physical Examination (PE) Disease, Symptoms Medication (MED) Treatment, Prevention Background (BKG) Neither PE nor MED

35. Medical Entity Extraction Applied ADEPT toolkit (MacLean and Heer 2013) High precision but low recall

36. Post Weighing Not all posts are equally representative Sondhi, 2013

37. Post Weighing : gives the weight of post i in a thread with K posts

38. Monotonic Post Weighing Relative Post Weight for K=10 Post Position i

39. Parabolic Post Weighing

40. Post Weighing Methods Evaluation

41. Forum Categories

42. Forum Category Weighing Ratio of current forum ID amongst retrieved documents Randomly selecting forum ID • Relevance feedback based on top k retrieved documents. • Forum Category Uniform Weighing (FCUW) : Weighs top-k forum categories equally. • Forum Category Feedback Weighing (FCFW) : Weighs forum categories based on how frequently they appeared on retrieved documents.

43. State of the Art Baseline Baseline BM-25 formula: c(w,t):Count of word win thread t c(w,q):Count of word win query q FPBM-25: Consider only the content of first post to represent the thread document TBM-25: Consider content of entire thread to represent the thread document

44. ShallowEx: Relevance Scoring Modified Query Count Word count in PE sentences Word count in MED sentences Word count in BKG sentences Give higher importance to PE and MED sentences

45. MedicalEx: Relevance Scoring Modified query frequency Count of occurrences labeled as med entity Count of occurrences not labeled as med entity

46. Post Weighing: Relevance Scoring Modified Thread Frequency Post Weight Post Frequency

47. Forum Category Weighing Scoring Weights for forum category weighing New Score Forum Category Feedback Weighing

48. Method Summary • Baseline Weighing • First Post BM-25 • Thread BM-25 • Semantic Weighing • Medical term extraction • Shallow Information Extraction • Post Weighing • Monotonic Weighing • Parabolic Weighing • Forum Category Weighing • Uniform Weighing (FCUW) • Feedback Weighing (FCFW)

49. Evaluation via Pooling • 350K threads and 20 queries from HealthBoards • 2 judges first judged 100 query-thread pairs • 88% agreement (κ=0.76) • 730 total judged query-thread pairs • 324 relevant • 406 irrelevant

50. Results: Semantic Methods Shallow extraction is better than medical entity extraction