Learning to Cluster Web Search Results.

1. Learning to Cluster Web Search Results. Hua-jun zeng Qi-cai He Zheng Chen Wei-Ying Ma Jinwen Ma

2. Contents • Motivation • Algorithm -Salient phrases extraction -Learning to rank salient phrases •Experiments •Conclusion

3. Motivation • Organizing Web search results into clusters facilitates user’s quick browsing through search results. • Traditional clustering techniques don’t generate clusters with highly readable namessalient phrase

4. Motivation(cont’d)

5. Problem formalization and algorithm • Algorithm is composed of the four steps: 1.Search result fetching 2.Document parsing and phrase property calculation 3.Salient phrase ranking 4.Post-processing

6. Salient Phrases extraction • Phrase Frequency/Inverted Document Frequency( TFIDF) • Phrase Length • Intra-Cluster Similarity • Cluster Entropy • Phrase Independence

7. TFIDF w=current phrase D(w)=the set of documents that contains w N=the number of total documents f(w)=frequency caclulation

8. Phrase Length • Generally, a longer name is preferred for users’ browsing

9. Intra-Cluster Similarity(ICS) •First,convert documents into vectors •For each candidate cluster, we then calculate its centroid as: •ICS is calculated as the average cosine similarity between the documents and the centroid

10. Cluster Entropy

11. Phrase Independence • a phrase is independent when the entropy of its context is high (I,e, the left and right contexts are random enough).we use a IND to measure the independence of phrases.

12. The IND,value for right context could be calculated similarly • The final IND value is the average of those two

13. Learning to rank salient phrases. • Regression tries to determine the relationship between two random variables X=(x1,x2,..,xp)and y x= (TFIDF,LEN,ICS,CE,IND) • Linear Regression

14. Experiments • Each query,200 returned documents from search engines. • Extract all n-grams from the documents where n<=3. • Use SVM_Light(2) to do support vector regression.

15. Experiments(cont’d) • Evaluation Measure -precision(P) at top N results to measure the performance:

16. Experiments(cont’d) • Training Data Collection

17. Experimental Results • Property Comparison.

18. Experimental Results (cont’d) • Learning Methods Comparison

19. Experimental Results(cont’d) • Learning Methods Comparison. -the coefficients of one of the linear regression models as follows: Y=-0.427 +0.146 X TFIDF +0.241 X LEN -0.022 X ICS +0.065 X CE +0.266 X IND

20. Experimental Results • In put Document Number

21. Experimental Results(cont’d) • Coverage and Overlap

22. Conclusion • Reformalizes the search result clustering problem as a supervised salient phrase ranking problem. • Several properties and regression models are proposed to calculate salience score for salient phrase