1 / 15

文本挖掘简介

文本挖掘简介. 邹权 博士，助理教授. Outline. Introduction TF-IDF Similarity. Introduction. Why ？ Text mining ≈ Web mining How ？ Classification or Clustering Retrieval. 文本分类一般过程. 预处理 将文档集表示成易于计算机处理的形式 特征表示与选择、降维 根据适宜的权重计算方法表示文档中各项的重要性 学习建模 构建分类器. 文本分类预处理. 去标点、多余空格、数字（可选） 大小写统一

gay
Download Presentation

文本挖掘简介

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. 文本挖掘简介 邹权 博士，助理教授

  2. Outline • Introduction • TF-IDF • Similarity

  3. Introduction • Why？ • Text mining ≈ Web mining • How？ • Classification or Clustering • Retrieval

  4. 文本分类一般过程 • 预处理 • 将文档集表示成易于计算机处理的形式 • 特征表示与选择、降维 • 根据适宜的权重计算方法表示文档中各项的重要性 • 学习建模 • 构建分类器

  5. 文本分类预处理 • 去标点、多余空格、数字（可选） • 大小写统一 • 去停用词（stop words） • 没有实际含义的词，比如and, you, have等等 • 统一词根 • PorterStemmer • 分词 • 英文？中文

  6. 特征表示 • 向量空间模型 • 以词项为特征组成高维特征向量 • TF/IDF得到权值

  7. TF-IDF • TF（Term Frequency） • 表示词项频率 • IDF（Inverse Document Frequency） • 逆文档频率 • TF*IDF值

  8. Similarity Applications Many Web-mining problems can be expressed as finding “similar” sets: Plagiarism/Mirror Pages/Articles from the Same Source/Duplication Remove Collaborative Filtering as a Similar-Sets Problem Recommend to users items that were liked by other users who have exhibited smilar tastes 8

  9. Measurement • Edit distance • Short text, words • For personal text • Jaccard distance • Long text, ignoring the word similarity • For government text

  10. Real-world Data is Rather Dirty！ Kenneth DeJong Kenneth Dejong PK http://academic.research.microsoft.com/Author/2037349.aspx http://academic.research.microsoft.com/Author/3054641.aspx Microsoft Academic Search 2014/10/14 Trie-Join @ VLDB2010 10/38

  11. Real-world Data is Rather Dirty！ DBLP Complete Search Argyrios Zymnis Argyris Zymnis relaxed Typo in “author” Typo in “title” related 2014/10/14 Trie-Join @ VLDB2010 11/38

  12. Similarity Joins R • The similarity join is an essential operation for data integration and cleaning • Perform a similarity join on Name attribute (find all record pairs whose Name attributes are similar) • Output: (2037349, 3054641), … 2014/10/14 Trie-Join @ VLDB2010 12/38

  13. Near Duplicate Data On one end, a winded Pete Sampras tried to summon enough energy to give the New York fans another memorable win to talk about it on the subway ride home. On the other side, Roger Federer wore a sly grin like he knew age was about to catch up to the former world No. 1 - the man who owns the record of 14 Grand Slams he wants. By JAY COHEN, AP Sports Writer Mar 11, 4:23 am EDT 03/11/2008 | 11:28 AM

  14. Similarity Join • Tokenize: • Each record is a set of tokens from a finite universe. • Suppose each record is a single text document • x = “yes as soon as possible” • y = “as soon as possible please” • x = {A, B, C, D, E} • y = {B, C, D, E, F}

  15. 参考文献 Chuan Xiao, Wei Wang, Xuemin Lin, Jeffrey Xu Yu. Efficient Similarity Joins for Near Duplicate Detection. WWW 2008. Guoliang Li, Dong Deng, Jiannan Wang, Jianhua Feng. Pass-Join: A Partition based Method for Similarity Joins. VLDB2012.

More Related