1 / 63

Summarizing Threads of Email Conversations: Using QA Pairs Detection to Improve Extractive Summaries

Summarizing Threads of Email Conversations: Using QA Pairs Detection to Improve Extractive Summaries. Lokesh Shrestha. Reasons for Summarizing Email. Email has become a primary means of business and personal communication.

melora
Download Presentation

Summarizing Threads of Email Conversations: Using QA Pairs Detection to Improve Extractive Summaries

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Summarizing Threads of Email Conversations: Using QA Pairs Detection to Improve Extractive Summaries Lokesh Shrestha

  2. Reasons for Summarizing Email • Email has become a primary means of business and personal communication. • Conversations take place and decisions are made entirely through email. • Given the high volume of email each individual accumulates, how can we efficiently retrieve information from our email archives?

  3. Summarizing Email vs. Summarizing Newswire • Email has interactive structure • Email can have informal language • Email does not have different, independent documents about same topic (not “multi-document summarization”)

  4. Contributions • Email specific features can be used for machine learning based extractive summarization of email threads • A novel approach to question-answer pair detection • Integration of QA pair sentences with extractive sentences improve summaries.

  5. Overview • Related Work • Corpus • Approach 1: Sentence Extraction • Approach 2: Question-Answer Pairs Detection • Approach 3: Integration • Outlook Email Client • Conclusion

  6. Related Work • Summarizing individual emails • Derek Lam, Steven L. Rohall, Chris Schmandt, and Mia K. Stern. 2002 • Sentence extraction • Smaranda Muresan, Evelyne Tzoukermann, and Judith Klavans. 2001. • Key phrase extraction • Summarizing discussion lists • Ani Nenkova and Amit Bagga. 2003. • Sentence extraction • Paula Newman and John Blitzer. 2003. • Thread topic clustering and sentence extraction. • Summarizing speech dialogues • Klaus Zechner. 2002. • Sentence Extraction and QA pairs

  7. Overview • Related Work • Corpus • Approach 1: Sentence Extraction • Approach 2: Question-Answer Pairs Detection • Approach 3: Integration • Outlook Email Client • Conclusion

  8. Corpus • Columbia ACM chapter executive board mailing list • Approximately 10 regular participants • ~300 Threads, ~1000 Messages • Threads include: scheduling and planning of meetings and events, question and answer, general discussion and chat. • Annotated by human annotators: • Hand-written summary • Categorization of threads and messages • Highlighting important information (such as question-answer pairs)

  9. Sample Hand-Written Summary for Thread Annotator 1 Summary: Alexander McCaughly asks the group if he can reschedule his C-session for Wednesday night. Raju Gupta tells McCaughly that he is able to reschedule his C-session. Reema Ramachandran reminds McCaughly that he scheduled an MS Office Session for November 14, and she asks McCaughly to confirm that he can be at that session.

  10. Overview • Related Work • Corpus • Approach 1: Sentence Extraction • Approach 2: Question-Answer Pairs Detection • Approach 3: Integration • Outlook Email Client • Conclusion

  11. Sentence Extraction Machine learning approach to extractive summarization of email threads • Creating Training Data • Learn extractive rules • Use rules to generate summary

  12. Sentence Extraction: Creating Training Data Using human generated summaries to create a model extractive summary • Compare thread sentences with human summary sentences using SimFinder • Given a summary size, select highly ranked sentences • Represent each sentence with a vector of features and the class

  13. Guys, I can't come tonight. Can I reschedule my C session for Wednesday night, 11/8, at 8:00? If that's cool with you guys, please reserve me a room. Sure we can, but that's the day after Election Day. Are you sure you want to do it then? alex, a reminder that your scheduled to do an MSOffice session on Nov. 14, at 7pm in 252Mudd. --please confirm that you can do that session/posters Confirmed. Intro to MS Office, then there will be three more where we'll work on the individual programs for full sessions Alexander McCaughly asks the group if he can reschedule his C-session for Wednesday night. Raju Gupta tells McCaughly that he is able to reschedule his C-session. Reema Ramachandran reminds McCaughly that he scheduled on MS Office Session for November 14, and she asks McCaughly to confirm that he can be at that session. SimFinder in Action

  14. Guys, I can't come tonight. Can I reschedule my C session for Wednesday night, 11/8, at 8:00? If that's cool with you guys, please reserve me a room. Sure we can, but that's the day after Election Day. Are you sure you want to do it then? alex, a reminder that your scheduled to do an MSOffice session on Nov. 14, at 7pm in 252Mudd. --please confirm that you can do that session/posters Confirmed. Intro to MS Office, then there will be three more where we'll work on the individual programs for full sessions Alexander McCaughly asks the group if he can reschedule his C-session for Wednesday night. Raju Gupta tells McCaughlythat he is able to reschedule his C-session. Reema Ramachandran reminds McCaughlythat he scheduled on MS Office Session for November 14, and she asks McCaughly to confirm that he can be at that session. SimFinder in Action SimFinder: 0.0038

  15. Guys, I can't come tonight. Can I reschedule my C session for Wednesday night, 11/8, at 8:00? If that's cool with you guys, please reserve me a room. Sure we can, but that's the day after Election Day. Are you sure you want to do it then? alex, a reminder that your scheduled to do an MSOffice session on Nov. 14, at 7pm in 252Mudd. --please confirm that you can do that session/posters Confirmed. Intro to MS Office, then there will be three more where we'll work on the individual programs for full sessions Alexander McCaughly asks the group if he can reschedule his C-session for Wednesday night. Raju Gupta tells McCaughly that he is able to reschedule his C-session. Reema Ramachandran reminds McCaughly that he scheduled on MS Office Session for November 14, and she asks McCaughly to confirm that he can be at that session. SimFinder in Action SimFinder: 0.0028

  16. Guys, I can't come tonight. Can I reschedule my C session for Wednesday night, 11/8, at 8:00? If that's cool with you guys, please reserve me a room. Sure we can, but that's the day after Election Day. Are you sure you want to do it then? alex, a reminder that your scheduled to do an MSOffice session on Nov. 14, at 7pm in 252Mudd. --please confirm that you can do that session/posters Confirmed. Intro to MS Office, then there will be three more where we'll work on the individual programs for full sessions Alexander McCaughly asks the group if he can reschedule his C-session for Wednesday night. Raju Gupta tells McCaughly that he is able to reschedule his C-session. Reema Ramachandran reminds McCaughly that he scheduled on MS Office Session for November 14, and she asks McCaughly to confirm that he can be at that session. SimFinder in Action SimFinder: 0.0028

  17. Guys, I can't come tonight. Can I reschedule my C session for Wednesday night, 11/8, at 8:00? If that's cool with you guys, please reserve me a room. Sure we can, but that's the day after Election Day. Are you sure you want to do it then? alex, a reminder that your scheduled to do an MSOffice session on Nov. 14, at 7pm in 252Mudd. --please confirm that you can do that session/posters Confirmed. Intro to MS Office, then there will be three more where we'll work on the individual programs for full sessions Alexander McCaughly asks the group if he can reschedule his C-session for Wednesday night. Raju Gupta tells McCaughly that he is able to reschedule his C-session. Reema Ramachandran reminds McCaughly that he scheduled on MS Office Session for November 14, and she asks McCaughly to confirm that he can be at that session. SimFinder in Action SimFinder: 0.0028

  18. Guys, I can't come tonight. Can I reschedule my C session for Wednesday night, 11/8, at 8:00? If that's cool with you guys, please reserve me a room. Sure we can, but that's the day after Election Day. Are you sure you want to do it then? alex, a reminder that your scheduled to do an MSOffice session on Nov. 14, at 7pm in 252Mudd. --please confirm that you can do that session/posters Confirmed. Intro to MS Office, then there will be three more where we'll work on the individual programs for full sessions Alexander McCaughly asks the group if he can reschedule his C-session for Wednesday night. Raju Gupta tells McCaughlythat he is able to reschedule his C-session. Reema Ramachandran reminds McCaughlythat he scheduled on MS Office Session for November 14, and she asks McCaughly to confirm that he can be at that session. SimFinder in Action SimFinder: 0.983

  19. Guys, I can't come tonight. Can I reschedule my C session for Wednesday night, 11/8, at 8:00? If that's cool with you guys, please reserve me a room. Sure we can, but that's the day after Election Day. Are you sure you want to do it then? alex, a reminder that your scheduled to do an MSOffice session on Nov. 14, at 7pm in 252Mudd. --please confirm that you can do that session/posters Confirmed. Intro to MS Office, then there will be three more where we'll work on the individual programs for full sessions Alexander McCaughly asks the group if he can reschedule his C-session for Wednesday night. Raju Gupta tells McCaughly that he is able to reschedule his C-session. Reema Ramachandran reminds McCaughly that he scheduled on MS Office Session for November 14, and she asks McCaughly to confirm that he can be at that session. SimFinder in Action SimFinder: 0.563

  20. Guys, I can't come tonight. Can I reschedule my C session for Wednesday night, 11/8, at 8:00? If that's cool with you guys, please reserve me a room. Sure we can, but that's the day after Election Day. Are you sure you want to do it then? dan, a reminder that your scheduled to do an MSOffice session on Nov. 14, at 7pm in 252Mudd. --please confirm that you can do that session/posters Confirmed. Intro to MS Office, then there will be three more where we'll work on the individual programs for full sessions Daniel Kestin asks the group if he can reschedule his C-session for Wednesday night. Janak Parekh tells Medina that he is able to reschedule his C-session. Christy Lauridsen reminds Medina that he scheduled on MS Office Session for November 14, and she asks Kestin to confirm that he can be at that session. SimFinder in Action SimFinder: 0.0038 SimFinder: 0.983 SimFinder: 0.0038 SimFinder: 0.0038 SimFinder: 0.0038 SimFinder: 0.752 SimFinder: 0.221 SimFinder: 0.368

  21. Determining Summary Size Determine the summary size the human summarizers used • Create gold-standard data manually • Select about 10% of ACM threads • gold-standard threads • Manually classify sentences in gold-standard threads • positive if content reflected in human summary • negative otherwise • Compare SimFinder derived classifications at various summary sizes with gold-standard classifications

  22. Determining Summary Size • Results • Use 45% • Verifies the use of SimFinder

  23. Guys, I can't come tonight. Can I reschedule my C session for Wednesday night, 11/8, at 8:00? If that's cool with you guys, please reserve me a room. Sure we can, but that's the day after Election Day. Are you sure you want to do it then? alex, a reminder that your scheduled to do an MSOffice session on Nov. 14, at 7pm in 252Mudd. --please confirm that you can do that session/posters Confirmed. Intro to MS Office, then there will be three more where we'll work on the individual programs for full sessions Alexander McCaughly asks the group if he can reschedule his C-session for Wednesday night. Raju Gupta tells McCaughly that he is able to reschedule his C-session. Reema Ramachandran reminds McCaughly that he scheduled on MS Office Session for November 14, and she asks McCaughly to confirm that he can be at that session Result: Sentences Marked as in Summary/not in Summary N Y N N N Y N Y

  24. Sentence Features: Thread as a document • Length: number of words in sentence • TF-IDF scores: highest, sum and mean • Centroid similarity • Subject similarity • Relative position in thread • Is question?

  25. Sentence Features:Email-Specific Features • Number of responses to the email. • Number of recipients of email • Has sender names: does the sentence contain the name of the senders of messages in the thread? • Email contains forwarded message? • Features derived from quoted material

  26. Learn extractive rules: Results • Using full feature set, 5-fold cross-validation with Ripper • Baseline scores are obtained with random classification

  27. Sample Ruleset: Nice Rules • IF centroid_sim_local  0.32 AND thread_line_num  4 AND isQuestion = 1 AND tfidfavg  0.21 AND tfidfavg  0.30 THEN Y. • IF centroid_sim  0.72 AND numOfRecipients  8 THEN Y. • IF centroid_sim_local  0.31 AND thread_line_num  4 AND tfidfmax  0.61 AND m_rel_pos  0.36 AND t_rel_pos  0.18 THEN Y. • IF centroid_sim_local  0.31 AND centroid_sim  0.76 AND centroid_sim  0.79 AND tfidfavg  0.19 THEN Y. • IF subject_sim  0.33 AND tfidfsum  2.84 AND tfidfsum  2.64AND tfidfmax  0.68 THEN Y. • ELSE N

  28. Automatically Generated Sample Summary Regarding "meeting tonight...", on Oct 30, 2000, Alexander Max McCaughly wrote: Can I reschedule my C session for Wednesday night, 11/8, at 8:00? Responding to this on Oct 30, 2000, Raju J Gupta wrote: Are you sure you want to do it then? Responding to this on Oct 30, 2000, Reema Ramachandran wrote: alex, a reminder that your scheduled to do an MSOffice session on Nov. 14, at 7pm in 252Mudd.

  29. Overview • Summarizing Email • Corpus Development • Approach 1: Sentence Extraction • Approach 2: Question-Answer Pairs Detection • Approach 3: Integration • Outlook Email Client • Conclusion

  30. The Problem • Question-answer exchanges common in email • Multiple questions in one thread; in one message • Multiple, possibly contradictory, answers to a single question • If a summary has question, and answer is in thread, summary should have the answer

  31. Questions in Email Summaries Complete summary from our rule-based sentence extractor: Regarding "acm home/bjarney", on Apr 9, 2001, Muriel Danslop wrote: Two things: Can someone be responsible for the press releases for Stroustrup? Responding to this on Apr 10, 2001, Theresa Feng wrote: I think Phil, who is probably a better writer than most of us, is writing up something for dang and Dave to send out to various ACM chapters.Phil, we can just use that as our "press release", right? In another subthread, on Apr 12, 2001, Kevin Danquoit wrote: Are you sending out upcoming events for this week?

  32. Approach • Same machine learning as before: Supervised rule induction based • Ripper (Cohen, ’96) • Same email corpus as before • ACM Corpus

  33. Detection of Questions Detecting questions is non-trivial • Informal use of question mark Use question mark in cases other than questions - to denote uncertainty, to make a suggestion. • I am on with Monday - perhaps some time in the afternoon or evening? • I suggest 7pm? • If it's better for ppl we could also have shorter lunch meetings (mon,tues,thurs)? • Overlook using a question mark after posing a question • Who can we get in touch with at your organization regarding these services. The work we present here is based on the detection of interrogative questions – inverted subject-verb order.

  34. Detection of Questions • Training Corpus - Speech • Switchboard corpus annotated with DAMSL tags. • 5000 positive examples, 5000 negative examples • negative examples - "statement-opinion" and "statement-non-opinion". • positive examples - "yes-no-question", "Wh-question", and "rhetorical-question" • Test Corpus - Email • manually extracted from the ACM corpus • 300 positive examples, 300 negative examples.

  35. Detection of Questions • Features • POS tags for the first five terms • POS tags for the last five terms • length of the utterance • most discriminating POS-bigrams

  36. Detection of Questions Results • Recall low because: Questions in ACM corpus start with a declarative clause • So, if you're available, do you want to come? • if you don't mind, could you post this to the class bboard? • Results without declarative clause:

  37. Detection of Answers • Detection difficult • Multiple topics discussed in parallel • Those that begin with a single topic may spin off different ones • Use of reply back function to answer a question asked earlier in the thread. We show how various features derived from the structure of email threads can improve upon lexical similarity between message segments

  38. Detection of Answers ACM Corpus • Annotators were asked to • Highlight and link Question and Answer pairs. • Annotator 1: 200 Threads, 81 QA Threads • Annotator 2: 138 Threads, 62 QA Threads • Inter-Annotator Agreement (Kappa statistic) • Question Detection: 0.68 • Answer Detection (given question): 0.81

  39. Detection of Answers Methods • Use human annotated data to generate training data • Textual Unit: • use message segments rather than individual sentences to reduce lexical gap between questions and candidate answers • Learn a classifier that predicts if a subsequent segment to a question segment answers it • Represent each question and candidate answer segment by a feature vector

  40. Detection of Answers Features Used • Standard: word counts, word overlap (Cosine, Euclidean) • Based on thread structure: • is candidate answer the first • number of emails between the question and the answer segments • the number of emails in the thread before the question segment • Based on other candidate answer segments • is candidate the most similar • relative position of the candidate among other candidates • number of other candidates

  41. Detection of Answers Experiments and Results • 5 fold cross validation using Ripper (Cohen, 96)

  42. Detection of Answers Experiments and Results • 5 fold cross validation using Ripper (Cohen, 96)

  43. Detection of Answers Experiments and Results • 5 fold cross validation using Ripper (Cohen, 96)

  44. Overview • Summarizing Email • Corpus Development • Approach 1: Sentence Extraction • Approach 2: Question-Answer Pairs Detection • Approach 3: Integration • Outlook Email Client • Conclusion

  45. Integrating extractive summaries with QA pairs: Approaches • Use QA pairs as features • Add corresponding answers to extracted questions and corresponding questions to extracted answers • Add extractive sentences to QA pairs • Use all QA pairs detected as basis for summary • Use machine learning technique to identify QA pairs to be included in summary

  46. Integrating extractive summaries with QA pairs: First Approach • Use QA pairs as features • Each sentence in the thread is represented by a feature vector • Relative position of the sentence in email and thread • TFIDF weights • Is question? • . • . • . • Is answer?

  47. Integrating extractive summaries with QA pairs: First Approach • Use QA pairs as features • Number of rules learned with this augmented set of features: 1397 • Number of rules that include the answer feature: 54 • Maximum number of rules that any feature is included in: 160

  48. Integrating extractive summaries with QA pairs: Second Approach • Add corresponding answers to extracted questions • Alex -- since you're in OS, what do you think? Do you think students will be working on the 15th? • I'm in OS, and yeah, I'm pretty sure people will be working on the weekend of a week before. • Add corresponding questions to extracted answers • Sure we can, but that's the day after Election Day. • Can I reschedule my C session for Wednesday night, 11/8, at 8:00?

  49. Integrating extractive summaries with QA pairs: Third Approach • Augment QA pair sentences with extractive sentences • Automatically detect QA segment pairs in a thread • Select the question sentence from each question segment • Select an answer sentence from each answer segment • Add extractive sentences if they do are not in any automatically detect QA segment pairs

  50. Integrating extractive summaries with QA pairs: Third Approach • Example Summary: Adding questions

More Related