Learning to Rank

1. Learning to Rank From Pairwise Approach to Listwise Approach

2. Agenda • Introduction • Ranking Problem • Ranking • Pairwise Ranking (Brief) • Listwise Ranking • Probability Models • Permutation Probability • Top one Probability • Results

3. Introduction • Construct model or method that learns to rank • Area of use: • Anti Spam • Product Rating • Expert Finding • ...

4. d_1^i d_2^i d_3^i . . d_n^i Introduction • Ranking Problem – Document retrieval Documents: {d_1, d_2, ... d_n} Ranking of documents Ranking System Query: q

5. Query Q D = {d , d , ...., d } 1 2 n Instance : (d , d ) 1 2 Pairwise Ranking • Classification of objects Relevance label Classification Model

6. Pairwise Ranking • Support Vector Machine • Ranking SVM • Boosting • RankBoost • Neural Network • RankNet

7. m (i) (i) L (y , z ) i=1 Listwise Ranking • Training (1) (m) Q = {q , ...., q } Queries d = {d , ...., d } Documents y = {y , ...., y } Judgements x = {x , ...., x } Features f (x ) Score Func. z = {f(x ) , ...., f(x )} Scores (i) (i) (i) (i) n 1 (i) (i) (i) (i) m (i) (i) 1 n T = {(x , y )} (i) i=1 (i) (i) (i) 1 n (i) j Listwise loss function (i) (i) (i) (i) n 1

8. Listwise Ranking • Ranking • New Query : q • Associated Docs. : d • Feature vectors : x • Trained rank. Func. : f (x ) • Rank documents in descending order ( i’ ) ( i’ ) ( i’ ) ( i’ ) j’

9. Probability of pi given s o(S ) o(S ) n Pi(j) Pi(k) P (pi) = s n j=1 k=j Permutation Probability • f : s probability distribution pi = <pi(1), pi(2), ...., pi(n)> s = (s , s , .... s ) 1 2 n

10. Top one prob. of j P (j) = P (p). s s P(1)=j,p n Top one Probability • Probability of being ranked on top of list

11. f = f x = f (x ) z (f ) = {f (x ), ..., f (x )} w (i) (i) w j j (i) (i) (i) w w 1 1 ListNet • Optimizing loss function • Neural Network as model • Gradient Descent as optimization alg. w = neural network

12. Results • TREC • Web pages from .gov domain • OSHUMED • Documents and queries on medicine • CSearch • Data from commerciel search engine

13. Results • NDCG – Normalized Discounted Cumulative Gain • Relevance judgements > 2 • Korrekt – Delvist korrekt - Ukorrekt • MAP – Mean Average Precision • Relevance judgements = 2 • Korrekt - Ukorrekt

14. Results • NDCG on TREC

15. Results • NDCG on OSHUMED