Email Data Cleaning (KDD’05)

1. Email Data Cleaning(KDD’05) Jie Tang1, Hang Li2, Yunbo Cao2, Zhaohui Tang3 1 Tsinghua University 2 Microsoft Research Asia 3 Microsoft Corporation

2. Outline • Motivation and Problem Description • Related Work • Our Approach • Implementation • Experimental Results • Summary

3. Motivation • Email is one of the most common modes of communication • Text mining applications on emails • Email classification • Email summarization • Term extraction from email • …

4. Term Extraction From: SY <sandeep....@gmail.com> - Find messages by this author Date: Mon, 4 Apr 2005 11:29:28 +0530 Subject: Re: ..How to do addition?? Hi Ranger, Your design of Matrix class is not good. what are you doing with two matrices in a single class?make class Matrix as follows import java.io.*; classMatrix { public static int AnumberOfRows; public static int AnumberOfColumns; private int matrixA[][]; public void inputArray() throws IOException { InputStreamReader input = new InputStreamReader(System.in); BufferedReader keyboardInput = new BufferedReader(input) } -- Sandeep Yadav Tel: 011-243600808 Homepage: http://www.it.com/~Sandeep/ On Apr 3, 2005 5:33 PM, ranger <asiri....@gmail.com> wrote: > Hi... I want to perform the addtion in my Matrix class. I got the program to > enter 2 Matricx and diaplay them. Hear is the code of the Matrix class and > TestMatrix class. I'm glad If anyone can let me know how to do the addition.....Tnx Extra line break Missing space Extra space Missing period . Case errors Hi Ranger, Your design of Matrix class is not good. What are you doing with two matrices in a single class? Make class Matrix as follows:

5. Outline • Motivation and Problem Description • Related Work • Our Approach • Implementation • Experimental Results • Summary

6. Related Work -- Data Mining • Email Cleaning • Several products have the feature of email cleaning by using rules • E.g. eClean (2000), WinPure ListCleaner Pro (2004) • Information Extraction from Email • Extracting contact information, etc • E.g. Kristjansson and Culotta (2004), Culotta, Bekkerman, and McCallum (2004), Viola (2005) • Web Page Cleaning • Removing banner ads, decoration pictures • E.g. Yi and Liu (2003), Lin and Ho (2002) • Tabular Data Cleaning • Detecting and removing duplicate information • E.g. Hernández and Stolfo (1998), Rahm and Do (2000), SQL Server 2005

7. Related Work -- Language Processing • Sentence Boundary Detection • Palmer and Hearst (1997) • Case Restoration • Lita and Ittycheriah (2003) • Mikheev (2002) • Spelling Error Correction • Golding and Roth (I996)

8. Outline • Motivation and Problem Description • Related Work • Our Approach • Implementation • Experimental Results • Summary

9. Our Approach -- Cascaded Approach Cleaning= non-text block filtering + text normalization • Non-text block filtering • - Quotation detection • - Header detection • - Signature detection • - Program code detection • Text normalization • - Paragraph normalization • * Extra line break detection • - Sentence normalization • * Missing period detection • - Word normalization • * Case restoration

10. Cascaded Approach From: SY <sandeep....@gmail.com> - Find messages by this author Date: Mon, 4 Apr 2005 11:29:28 +0530 Subject: Re: ..How to do addition?? Hi Ranger, Your design of Matrix class is not good. what are you doing with two matrices in a single class?make class Matrix as follows import java.io.*; class Matrix { public static int AnumberOfRows; public static int AnumberOfColumns; private int matrixA[][]; public void inputArray() throws IOException { InputStreamReader input = new InputStreamReader(System.in); BufferedReader keyboardInput = new BufferedReader(input) } -- Sandeep Yadav Tel: 011-243600808 Homepage: http://www.it.com/~Sandeep/ On Apr 3, 2005 5:33 PM, ranger <asiri....@gmail.com> wrote: > Hi... I want to perform the addtion in my Matrix class. I got the program to > enter 2 Matricx and diaplay them. Hear is the code of the Matrix class and > TestMatrix class. I'm glad If anyone can let me know how to do the addition.. Hi Ranger, Your design of Matrix class is not good. what are you doing with two matrices in a single class?make class Matrix as follows Hi Ranger, Your design of Matrix class is not good. What are you doing with two matrices in a single class? Make class Matrix as follows. Hi Ranger, Your design of Matrix class is not good. what are you doing with two matrices in a single class? make class Matrix as follows. Quotation Detection Header Detection Signature Detection Program Code Detection Extra line break Detection Missing Period and Missing Space Detection Extra Space Detection In a particular text mining application, we can retain some of the blocks Case Restoration

11. Outline • Motivation and Problem Description • Related Work • Our Approach • Implementation • Experimental Results • Summary

12. Technical Issues • Non-text filtering • Quotation detection • Header detection • Signature detection • Program code detection • Text normalization • Extra line break detection • Sentence normalization • Case restoration

13. Non-text Filtering Using SVMs Header detection Signature detection Program code detection

14. Special Pattern Features Positive Word Features Position Feature Ending Character Features Contains one type of special patterns: email, date, number, URL, percentage, etc. Begins with: “From:”, “Re:”, “In article”, etc. Contains: “original message”, “Fwd:”, etc. Ends with: “wrote:”, “said:”, etc. Is the first line? Ends with: colon, semicolon, quotation mark, question mark, exclamation mark, etc. Features Used in Header Detection

15. Header Detection From: SY <sandeep....@gmail.com> - Find messages by this author Date: Mon, 4 Apr 2005 11:29:28 +0530 Subject: Re: ..How to do addition?? Hi Ranger, Your design of Matrix class is not good. what are you doing with two matrices in a single class?make class Matrix as follows import java.io.*; class Matrix { public static int AnumberOfRows; public static int AnumberOfColumns; private int matrixA[][]; public void inputArray() throws IOException { InputStreamReader input = new InputStreamReader(System.in); BufferedReader keyboardInput = new BufferedReader(input) } -- Sandeep Yadav Tel: 011-243600808 Homepage: http://www.it.com/~Sandeep/ On Apr 3, 2005 5:33 PM, ranger <asiri....@gmail.com> wrote: > Hi... I want to perform the addtion in my Matrix class. I got the program to > enter 2 Matricx and diaplay them. Hear is the code of the Matrix class and > TestMatrix class. I'm glad If anyone can let me know how to do the addition.....Tnx Two SVM models are employed to respectively identify the start line and end line.

16. Automatic Feature Generation - Input: An annotated email dataset. - Output: Discovered features. - Algorithm: Step 1: Preprocessing. This step first processes emails by using hard rules. it replaces several special patterns by a tag. For example, an email address “joke@hotmail.com” is to be replaced by a tag <email>. Step 2: Learning patterns. This step take the header lines as positive samples and the other lines as negative samples. It employs the pattern learning tool to discovering the patterns. An example of the discovered patterns is: “<begin> Date: <week> <date> <time> <end>”. Step 3: Generating features. This step generates features according to the learned patterns by using heuristic rules. For the above example, the corresponding feature can be: “^\s*Date: <week> <date> <time>\s*$”. The feature represents whether or not the current line contains the pattern. - Feature definition is tedious. - Can we automate the feature generation?

17. Example Features Used in Signature Detection

18. Example Features Used in Program Code Detection

19. Extra Line Break Detection Using SVMs

20. Case Features Whether the current line ends with a word in lower case letters and whether or not the next line starts with a word in lower case letters Features Used in Extra Line Break Detection

21. Extra Line Break Detection Hi Ranger, Your design of Matrix class is not good. what are you doing with two matrices in a single class?make class Matrix as follows One SVM model is employed to identify whether a line break is an extra one or not.

22. Case restoration • tri-gram + sentence level decoding Backoff scheme:

24. Outline • Motivation and Problem Description • Related Work • Our Approach • Implementation • Experimental Results • Summary

25. Datasets in Experiments 0.15 0.38 0.17 5565 4229(0.760) 3256(0.585) • 73.2% contain extra line breaks • 85.4% need sentence normalization • 47.1% contain case errors • Only 1.6% are absolutely clean

26. Cleaning Results -- 5-fold Cross Validation • Baseline methods • Header detection (eClean2000) • Signature detection (rule based) • Extra line break detection baseline (eClean2000) • For case restoration: • Our method can reach 98.15% in terms of accuracy • The accuracy of Trucasing is about 97.7%

27. Automatic Features vs. Manual Features

28. Term Extraction Using Email Cleaning BR J2EE

29. How Cleaning Processing Helps Term Extraction +6.4% +41% +74.2% BR

30. How Cleaning Processing Helps Term Extraction (cont.) +2.3% +24.7% +42.4% J2EE

31. Outline • Motivation and Problem Description • Related Work • Our Approach • Implementation • Experimental Results • Summary

32. Summary • Formalized email data cleaning as non-text filtering and text normalization • Conducted email cleaning in ‘cascaded’ approach • Used SVM models for header, signature, program code, and extra line break detection • Our approach significantly outperforms baseline methods • When applied to term extraction, significant improvement on extraction accuracy can be obtained

33. Thanks!