1 / 19

PAT Tree and Chinese Keyword Extraction

PAT Tree and Chinese Keyword Extraction. Chien Chin Chen Department of Information Management National Taiwan University. PAT Tree (1/8). A PAT tree is a data structure used to index text collections. It has been proven effective in text processing and search * .

enochd
Download Presentation

PAT Tree and Chinese Keyword Extraction

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. PAT Tree and Chinese Keyword Extraction Chien Chin Chen Department of Information Management National Taiwan University

  2. PAT Tree (1/8) • A PAT tree is a data structure used to index text collections. • It has been proven effective in text processing and search*. • The PAT tree based index method treat the entire text as an array of characters and construct a PAT tree from a set of sistrings. • A sistring(semi-infinite string) is a subsequence of characters from this array, taken from a given starting point but going on as necessary to the right. *G.H. Gonnet, R.A. Baeza-Yates, and T. Snider, New Indices for Text: PAT Trees and PAT Arrays, Information Retrieval – Data Structures and Algorithms, Prentice Hall, 1992.

  3. PAT Tree (2/8) • Example of text and sistrings: Text Once upon a time, in a far away land … sistring1Once upon a time … sistring2nce upon a time … sistring3ce upon a time … … sistring8on a time … Each sistring is uniquely identified by its starting positing.

  4. PAT Tree (3/8) • Basically, each sistring is represented as a bit sequence. • And PAT tree is constructed by inserting sistrings one by one. • An internal node in PAT tree has an indication of which bit of the sistring is to be used for branching. • A zero it will cause a branch to the left subtree. • A one bit will cause a branch to the right subtree. • Hence PAT trees are binary trees. • The external nodes in PAT tree are sistrings.

  5. PAT Tree (4/8) • A recursive method for PAT tree construction. Insert(root, sistring) if root is an external node identify the bit position that branches the two sistrings. create a parent node with the branch position and attach the two sistring as its left and right subtrees. else // root has left and right subtrees let x be the bit position of the root if the (x-1) prefixes of sistring and a (any) external node of root are the same if the x bit of sistring is 0 Insert(root->lefttsubtree, sistring) else Insert(root->rightsubtree, sistring) else create a parent node with the branch position and attach the sistring and root as its left and right subtrees.

  6. PAT Tree (5/8) Text: 00010011… Position: 12345678… External node (sistring), labeled by its position Internal node, labeled by bit position An internal node must have lift and right subtrees. 3 2 1 1 sistring1: 00010011… sistring2: 0010011… sistring3: 010011… sistring4: 10011… sistring5: 0011… 1 3 1 2 3 2 4 2 4 1 2 3 3 3 3 Every subtree has all the sistrings with the same prefix!! 1 2 1 4 For instance, sistrings 1, 2, and 3 all have ‘0’ prefix. 2 5

  7. PAT Tree (6/8) • Implementation details: • So far we have assumed that every sistring in the text need to be constructed. • However, if we are interested in word, then only those sistrings that are at the beginning of words are necessary. Text: 台灣的總統大選… Bit stream: 000100100011… Position: 159…

  8. PAT Tree (7/8) • Prefix search on PAT tree: • Notice that every subtree of the PAT tree has all the sistrings with a given prefix. • Prefix search in a PAT tree consists of searching the prefix in the tree up to the point where we exhaust the prefix or up to the point where we reach an external node. • At this point we need to verify whether we could have skipped bits. • We compare a (any) sistring in the subtree with the searching prefix. exhaust the prefix 00010011 Prefix search: 3 reach an external node 00010001 7 The top 10 bits of these sistrings are identical. But they can be 10010011.. rather than the searching prefix 11 If the prefixes match, we can easily identify the locations of the searching information in terms of the sistring positions.

  9. PAT Tree (8/8) • The most frequent search: • The number of leaf nodes indicates the frequency of a prefix in a text collection. If we construct the PAT tree in terms of Chinese documents, a frequent subtree may represent a Chinese keyword!! 1 9 9 HOWEVER, substrings of keywords, such as“統府”, are also significant. 13 總統今天… 統府秘書長… 統府前… The top 8 bits may represent the word “總統” – 3 times 總統府秘書長… 總統府前… The top 12 bits may represent the word “總統府” – 2 times

  10. Lee-Feng Chien, “PAT-Tree-Based Keyword Extraction for Chinese Information Retrieval,” in Proceedings of the 20th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 50-58, 1997.

  11. PAT Tree on Chinese Keyword Extraction (1/2) • Difficulties of Chinese keyword extraction: • No explicit word boundaries in written sentences. • Too many unknown words, such as names, with no effective extraction rules. • A keyword … supposedly … will occur at least several times in a text collection. • The propose method tries to identify significant lexical patterns(SLP) from a set of relevant (Chinese) documents. • Lexical patterns  Chinese keywords. • For example, “資訊檢索” can be a SLP for a set of IR-related documents. • Significant frequent and complete in semantic.

  12. PAT Tree on Chinese Keyword Extraction (2/2) • The proposed method first constructs a PAT tree for the text collection. • When applying PAT tree to Chinese keyword extraction, instead of recording the sistrings at document level, we record them at sentence level. • We use punctuation marks such as “.” and “,” as delimiters to determine a sentence boundary. • By using the frequency information of prefix, the PAT tree then produces a lot of possible lexical patterns. • However, many patterns should be filtered out. • For instance, if “關鍵字抽取” is frequent, then “關鍵字抽” and “鍵字抽取” are frequent as well. However, they are incomplete in semantics an lack of representatives.

  13. Significance Estimation (1/4) • According to the findings in our experiments, most of the significant lexical patterns have strong association between its composed and overlapped substrings. • For a frequent lexical pattern C=c1c2c2…cn, its two longest composed substrings, which are A=c1c2…cn-1 and B=c2c3…cn, respectively, must be frequent as well. • But if A and B are highly dependent to C, then A and B should not be complete patterns. • For example, C=“關鍵字抽取”, A=“關鍵字抽”, B=“鍵字抽取”.

  14. Significance Estimation (2/4) High SEc indicates that A and B are incomplete because they often occur together with C in the text collection. Proportion of C in A union B A B C

  15. Significance Estimation (3/4) • For a high SE pattern, we can filter out its frequent but incomplete substrings. • However, high SE does not indicate that the pattern is complete. • For example, In a text collection, the word “奧林匹克” occurs 6 times, definitely, C=“奧林匹”, A=“奧林”, and C=“林匹”, must occurs no less than 6 times. • Suppose that C, A, and B occur exactly 6 times. • Then SEC = 6 / (6+6-6) = 1.

  16. Significance Estimation (4/4) • On the other hand, low SEdoes not mean a insignificant pattern. • For example, C=“關鍵字” occur 10 times, A=“關鍵” occur 25 times, and B=“鍵字” occur 10 times. • Then SEC = 10 / (10+25-10) = 0.4.

  17. Pattern Filtering Algorithm (1/2) • For each lexical pattern a in the PAT tree, where ahas not been marked as a non-SLP and fa >= THf(a frequency threshold). • If a is a terminal node, then a is determined as a candidate of SLP. • Let c be one of a’s successor. And b be the another longest substring of c. If SEc >= THse(SE threshold), a nd b are determined non-candidates of SLP and will have a non-SLP mark on the PAT tree. • If fc >= THf, then c should be check recursively. • If SEc < THse, but fa >> fb, then a is determine as a candidate of SLP. • However, b and c are uncertain. • If fc >= THf, then c should be check recursively. c=“鍵詞抽”-10 times, a=“鍵詞”– 40 times, b=“詞抽”– 10 times. Example in the paper c=“關鍵詞抽”– 10 times, a=“關鍵詞”– 40 times, b=“鍵詞抽”– 10 times. But … problematic…

  18. Pattern Filtering Algorithm (2/2) • The algorithm will be started from the longest patterns with frequency larger than the threshold and then backtraced to their ancestors. • In our findings of the experiments, the above filtering algorithm is especially useful in extracting SLP like names, locations, and technical terms. • There do exist weaknesses with the filtering algorithm. • A refined procedure, using a lexicon and another set of candidates from a general-domain PAT tree, is employed to remove SLP candidates.

  19. Resources & References • CKIP: • Chinese Knowledge Information Processing at Academia Sinica, Taiwan. • Is continuing building a Chinese word lexicon with rigid syntactic information. • http://ckipsvr.iis.sinica.edu.tw/ • PAT tree for information Extraction: • Chia-Hui Chang and Shao-Chen Lui, “IEPAD: information extraction based on pattern discovery,” in Proceedings of the 10th international conference on World Wide Web, pp. 681-688, 2001.

More Related