Modeling term relevancies in information retrieval using Graph Laplacian Kernels

1. Modeling term relevancies in information retrievalusing Graph Laplacian Kernels Shuguang Wang Joint work with SaeedAmizadehand MilosHauskrecht

2. A Problem in Document Retrieval • There is a ‘gap’ between search queries and documents. Query: car

3. A Problem in Document Retrieval • There is a ‘gap’ between search queries and documents. Query: car Google.com Bing.com Yahoo.com …

4. A Problem in Document Retrieval • There is a ‘gap’ between search queries and documents. Query: car Google.com Bing.com Yahoo.com …

5. A Problem in Document Retrieval • There is a ‘gap’ between search queries and documents. Query: car Google.com Bing.com Yahoo.com … Good enough?

6. A Problem in Document Retrieval • What about the documents about automobiles, BMW, Benz, …? • There are various expressions for a same entities. • One solution is to expand the original user queries with some ‘relevant’ terms.

7. Traditional Query Expansion Methods • Human and/or computer generated thesauri • Zhou et al., SIGIR 2007 proposed to expand query with MeSH concepts. • Human Relevance feedback • Implicit feedback from human such as tracking eye movement (Buscher et al., SIGIR 2009). • User click information (Yin et al., ECIR 2009) • Automatic query expansion • Pseudo Relevance Feedback first proposed in (Xu and Croft, SIGIR 1996). • Use top ‘n’ documents from the initial search as the implicit feedback and select ‘relevant’ terms from these ‘n’ documents.

8. Traditional Query Expansion Methods • Human and/or computer generated thesauri • Zhou et al., SIGIR 2007 proposed to expand query with MeSH concepts. • Human Relevance feedback • Implicit feedback from human such as tracking eye movement (Buscher et al., SIGIR 2009). • User click information (Yin et al., ECIR 2009) • Automatic query expansion • Pseudo Relevance Feedback first proposed in (Xu and Croft, SIGIR 1996). • Use top ‘n’ documents from the initial search as the implicit feedback and select ‘relevant’ terms from these ‘n’ documents. • Analyze query flow graph in (Bordino et al., SIGIR 2010) Expensive, and time consuming

9. Traditional Query Expansion Methods • Human and/or computer generated thesauri • Zhou et al., SIGIR 2007 proposed to expand query with MeSH concepts. • Human Relevance feedback • Implicit feedback from human such as tracking eye movement (Buscher et al., SIGIR 2009). • User click information (Yin et al., ECIR 2009) • Automatic query expansion • Pseudo Relevance Feedback first proposed in (Xu and Croft, SIGIR 1996). • Use top ‘n’ documents from the initial search as the implicit feedback and select ‘relevant’ terms from these ‘n’ documents. • Analyze query flow graph in (Bordino et al., SIGIR 2010) Expensive, and time consuming Human Input

10. Traditional Query Expansion Methods • Human and/or computer generated thesauri • Zhou et al., SIGIR 2007 proposed to expand query with MeSH concepts. • Human Relevance feedback • Implicit feedback from human such as tracking eye movement (Buscher et al., SIGIR 2009). • User click information (Yin et al., ECIR 2009) • Automatic query expansion • Pseudo Relevance Feedback first proposed in (Xu and Croft, SIGIR 1996). • Use top ‘n’ documents from the initial search as the implicit feedback and select ‘relevant’ terms from these ‘n’ documents. • Analyze query flow graph in (Bordino et al., SIGIR 2010) Expensive, and time consuming Human Input

11. A Different View • What we really need here is a way to estimate term-term relevance. • Problem of finding expansion terms for user queries  Problem of finding ‘relevant’ terms given a similarity metric. • How to derive a term-term similarity metric?

12. Term-Term Similarity • Hypothesis: the metric ‘d’ should be smooth, i.e., d(t1) ~ d(t2) if ‘t1’ and ‘t2’are similar/relevant. • Why not graph Laplacian kernels?! • We can easily have smoothness property. • We can also define distance metrics with it.

13. Define Affinity Graph • Nodes are terms • Edges are co-occurrences • Weights of the edges are the number of documents terms co-occur

14. Graph Laplacian Kernels • General Form Definition: • Resistance: • Diffusion: • P-step Random Walk: • … Recall:

15. Graph Laplacian Kernels • General Form Definition: • Resistance: • Diffusion: • P-step Random Walk: • … How to choose hyper parameters? Recall:

16. Graph Laplacian Kernels • General Form Definition: • Resistance: • Diffusion: • P-step Random Walk: • … How to choose hyper parameters? Recall: How to choose g(λ)?

17. Non-parametric kernel • Learn the transformation g(λ) directly from training data. • If we know some terms are similar, we want to maximize their similarities. • At the same time, we want to have a smoother metric.

18. An Optimization Problem ‘ ‘ ‘ ‘ ‘ : the set of eigenvalues of original Laplacian graph tin‘ and tjn’ are pair of similar terms in the training document n’

19. An Optimization Problem Maximize for known similar terms tin and tjn ‘ ‘ ‘ ‘ ‘ : the set of eigenvalues of original Laplacian graph tin‘ and tjn’ are pair of similar terms in the training document n’

20. An Optimization Problem Penalize more for large eigenvalues ‘ ‘ ‘ ‘ ‘ : the set of eigenvalues of original Laplacian graph tin‘ and tjn’ are pair of similar terms in the training document n’

21. Kernel to Distances • Given the kernel K, we can define distances between any pair of nodes, d(i,j), in the graph. µ2 µ1 Recall: µn We define: ti tj

22. Kernel to Distances • Given the kernel K, we can define distances between any pair of nodes, d(i,j), in the graph. µ2 µ1 Recall: µn We define: Euclidean Distance ti tj d(i,j) = Kii+Kjj-2Kij

23. Kernel to Distances • Given the kernel K, we can define distances between any pair of nodes, d(i,j), in the graph. µ2 µ1 Recall: µn The distance metric derived from graph Laplacian Kernel is the Euclidean distances in the kernel space We define: Euclidean Distance ti tj d(i,j) = Kii+Kjj-2Kij

24. Using term-term similarity in IR • Deal with similarity between sets and terms. • In query expansion tasks, set of query terms is ‘S’ and a candidate expansion term is ‘t’. • Transform the pair-wise distances, ‘d’, to set-to-term similarity. • Naïve methods: • dmax=max(d(S,t)) • davg=avg(d(S,t)) • dmin=min(d(S,t))

25. Set-to-term Similarity • Query collapsing

26. Query Collapsing • We have to compute eigen-decompostion again for each query. • It is too expensive for the online task. • Approximation is possible. • We want to approximate the projection of `new’ point ‘S’ in the kernel space. • We need to add one element in the original eigenvector. µ2 µ2 µ1 µ1 µn µn A A E E S

27. Nystrőm Approximation • For all nodes in the graph Laplacian, we have • If the new point s’ was in the graph, it would satisfy the above as well.

28. Nystrőm Approximation • For all nodes in the graph Laplacian, we have • If the new point s’ was in the graph, it would satisfy the above as well.

29. Nystrőm Approximation • For all nodes in the graph Laplacian, we have • If the new point s’ was in the graph, it would satisfy the above as well.

30. Evaluation • Two tasks: • Term prediction (scientific publication) • Give the terms in the abstracts, predict the possible terms in the full body • Compare with TFIDF, PRF, PLSI • Query expansion • Compare with Lemur/Indri + PRF and Terrier + PRF • Kernels: • Diffusion (optimized by line search) • Resistance • Non-parametric (optimized by line search) • Set-to-term: • Average • Query collapse

31. Term prediction • 6000 articles about 10 cancers downloaded from PubMed. • 80% as training and 20% as testing • Given the terms in abstracts, rank all the candidate terms using the distance metrics. • The smaller the distances between candidate terms and query terms, the higher rank these terms are. • Use AUC to evaluate (Joachims, ICML 2005)

32. Results

33. Query Expansion • Four TREC datasets: Genomic 03 & 04, Adhoc TREC 7 & 8. • We built graph using different set of terms in these datasets: • genes/proteins on Genomic 03 data • 5000 terms with highest TFIDF scores on Genomic 04 data. • 25% subsamples from all (~100k) unique terms from TREC7 & 8. • Use Mean Average Precision (MAP) to evaluate the performance. • Only Resistance Kernel

34. Results