Twitter Catches The Flu: Detecting Influenza Epidemics using Twitter

1. Twitter Catches The Flu: Detecting Influenza Epidemics using Twitter Eiji ARAMAKI * Sachiko MASKAWA * Mizuki MORITA** * The University of Tokyo ** National Institute of Biomedical Innovation EMNLP2011

2. Why we developed this system? Let me show you several existing systems

3. Centers for Disease Control and Prevention (CDC)

4. Infection Disease Surveillance Center (IDSC)

5. European Influenza Surveillance Network (EISN)

6. Why each country has each surveillance system? • Influenza epidemics are a major public health concern, because it causes tens of millions of illnesses each year. • To reduce the victims, the early detection of influenza epidemics is a national mission in every country. • BUT: These surveillance systems basically rely on hospital reports (written manually).

7. Two Problems & Recent Approach • (1) Small Scale • For example, IDSC gathers influenza patient data from 5,000 clinics. But It does not cover all cities (especially local cities). • (2) Time Delay (Time lag) • For example, the data gathering process typically has a 1–2 week reporting lag • To deal with these problems • Recently,various approaches that directly capture people’s behavior are proposed

8. Recent Approach • using Phone Call data • Espino et al. (2003) used data of a telephone triage service, a public service, to give an advice to users via telephone. They reported the number of telephone calls that correlates with influenza epidemics. • using Drug sale data • Magruder (2003) used the amount drug sales. Among various approaches…

9. The State-of-the-ArtWeb based Approach • Ginsberg et al. (Nature 2009) used Google web search queries that correlate with an influenza epidemic, such as “flu”, “fever”. • Polgreen et al. (2008) used a Yahoo! query log. • Hulth et al. (2009) used a query log of a Switzerland web search engine.

10. This Study • Web search query is a extremely large scale and real-time data resource. • BUT: the query data is closed (not freely available), which is available only for several companies, such as Google, Yahoo, or Microsoft. → This study examines Twitter data, which is widely available.

11. OUTLINE • Background • Objective • Method • Experiment • Discussion • Conclusion Detailed Task Definition

12. Simple Word Frequency in Twitter“Cold”, “Fever” & “influenza” Winter Summer Actual influenza curve is more smooth Simple Word Frequency contains various noises Because….

13. A word “influenza” does not always indicate an influenza patient Positive Influenza Tweet Negative Influenza Tweet

14. Two types of Influenza Tweets • Negative influenza tweet indicates an influenza patient • Negative influenza tweet includes mention of “influenza”, but does not indicate that an influenza patient is present • Not only the general news, but also various phenomena generate Negative influenza tweet… Positive Influenza Tweet Negative Influenza Tweet

15. Various Negative Influenza Tweet (1/2) • Prevention • You need to get a influenza shot sometime soon. • Modality (just suspition) • @Johnmight be suffering from influenza • Question • Did you catch the influenza ?

16. Various Negative Influenza Tweet (2/2) • Influenza of Cat or Dog • Today, I couldn't go home late. My cat caught the influenza... • Influenza of TV Character • Inthe last episode of that TV Series, Ritsu-chan caught the flu

17. Research Questions • In total, half of Influenza related tweets are negative, motivating an automatic filtering. • RQ1: Could a NLP system filter out the negative influenza tweet? • RQ2: Could this filtering contributes to the surveillance accuracy?

18. OUTLINE • Background • Method • Experiment • Discussion • Conclusion

19. Basic Idea: Binary Classification • We regard this task as a binary classification task , such as a spam mail filtering input (2) What kind of Feature? (3) What kind of Machine Learning Method? Training Corpus (1) What kind of Corpus? Negative Positive

20. Corpus (5k Sentences with Labels) See proceeding for detailed Average Annotator Agreement Ratio = 0.85

21. What kind of Feature? Twitter contains many ungrammatical expressions • Surrounding Words (BOW, no stemming, no POS) I think the influenza is going around L3 L2 L1 R2 R1 R3 • Among various settings, Window size = 6 achieved the highest accuracy

22. What kind of Machine Learning Method? • Among various settings, SVMachieved the feasible accuracy

23. OUTLINE • Background • Method • Experiment • Discussion • Objective

24. Twitter Data (2008-2010) • First month is used for training corpus • We divides the other data into 4 seasons • Twitter API sometimes changes the spec, leading to dropout periods. Season I Season II Season III Season IV

25. Method Comparison & Evaluation • (1) TWEET-SVM (The proposed method) • (2) TWEET-RAW • Based on simple word frequency of “influenza” • (3) GOOGLE [Ginsberg 2009] • Based on Google web-search query • The previous estimation data is available at the Google Flu Trend website. • (4) DRUG-SALE [Magruder 2003] • Evaluation is based on • Average Correlation with GOLD_STANDARD DATA that is the real number of the influenza patients reported by Infection Disease Surveillance Center (IDSC)

26. Result: Correlation Ratio +SVM Bold indicates the correlation > statistical significance level. In most seasons, the proposed method achieved the higher correlation than simple word freq-based method, demonstrating the advantage of the SVM based filtering

27. Result: Correlation Ratio +SVM Bold indicates the correlation > statistical significance level. Except for Season II, the proposed method achieved almost the same accuracy to GOOGLE.

28. Why Twitter suffers from Season II? Because it includes Pandemic! Suggesting Twitter might be biased by News Media WHO says Pandemic In 1999 Jul (Season II).

29. Season I Relative number TWEET-SVM ≒ GOOGLE

30. Season II Relative number TWEET-SVM << GOOGLE

31. OUTLINE • Background • Method • Experiment • Discussion • Conclusion Extra Experiment

32. Frequent Question • Could an Influenza Patient REALLY use a Twitter or Google Search? • That seems to be un-natural situation! I’d like to sleep ... Due to that, we modified the system assuming as follows: People use Twitter or Google at the first sign of the influenza

33. Implemented by usingInfectious Model [Kermack1927] (≒ Markov model) • S-to-I transition is observed by Twitter / Google • 38% of Influenza people recover a day UNDER FLU 0.62 AFTER FLU BEFORE FLU Catch the flu Recover S I R 0.38 Infectious Recover Susceptible

34. BUT: It ALSO improves Google based Approach • This model improves correlation of BOTH Twitter& GOOGLE. • This result suggests that there is a room of collaboration between medical study and web/NLP study

35. OUTLINE • Background • Method • Experiment • Discussion • Conclusion

36. Answer to Research Questions • This study proposed a new influenza surveillance system using Twitter • RQ1: Could a system filter out the negative influenza? • Yes. But NOT Perfect • RQ2: Could this accuracy contribute to the surveillance performance? • YES. It increases the correlation(except for pandemic period). • We could achieve the almost same accuracy to GOOGLE using freely available data.

37. Conclusion • Still now, more than 100 (sometime over 1,000) people die from influenza in Japan • We hope that this study might help people

38. Thank youNLP could save a life! • Eiji ARAMAKI Ph.D. • University of Tokyo • http://mednlp.jp