1 / 27

Evaluating the Robustness of Learning from Implicit Feedback Filip Radlinski Thorsten Joachims

Evaluating the Robustness of Learning from Implicit Feedback Filip Radlinski Thorsten Joachims. Presentation by Dinesh Bhirud bhiru002@d.umn.edu. Introduction. The paper evaluates the robustness of learning to rank documents based on Implicit feedback. What is implicit feedback?

verna
Download Presentation

Evaluating the Robustness of Learning from Implicit Feedback Filip Radlinski Thorsten Joachims

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Evaluating the Robustness of Learning from Implicit FeedbackFilipRadlinskiThorsten Joachims Presentation by Dinesh Bhirud bhiru002@d.umn.edu

  2. Introduction • The paper evaluates the robustness of learning to rank documents based on Implicit feedback. • What is implicit feedback? • Relevance feedback obtained from search engine log files • Easier to collect large amount of such training data as against explicitly collecting relevance feedback.

  3. Osmot • Osmot – Search engine developed at Cornell University based on Implicit Feedback • Name Osmot comes from the word “osmosis” – learning from the users by osmosis • Query Chains – Sequence of reformulated queries. • Osmot learns ranked retrieval function by observing query chains and monitoring user clicks.

  4. High Level Block Diagram

  5. Data Generation • Set of W words are chosen, word frequencies obeying a Ziph’s law • T topics are picked by picking N words/topic uniformly from W. • Each document d is generated as • Pick kd binomially from [0,T] • Repeat kd times • Pick topic t • Pick L/kd words from topic t.

  6. Relevance • 3 kinds of relevance • Relevance with respect to topic • Can be measured/known because document collection and topics are synthetic • Used for evaluating the ranking function. • Relevance with respect to query • Actual relevance score of a document with respect to a query • Used to rank documents • Observed relevance • Relevance of a document as judged by the user seeing only the abstract. • Used to simulate user behavior.

  7. User behavior parameters • Noise – Accuracy of user’s relevance estimate • Affects observed relevance. (obsRel) • obsRel is drawn from an incomplete Beta distribution where α gives noise level and β is selected so that mode is at rel(d,q) • Threshold – User selectivity over results (rT) • Patience – Number of results user looks at before giving up (rP) • Reformulation – How likely is the user to reformulate query(Preform)

  8. User Behavior Model While question T is unanswered 1.1 Generate query q (Let d1,d2..,dn be results for q) 1.2 Start with document 1 iei = 1 1.3 while patience (Rp) > 0 1.3.1 if obsRel(di,q) > rT 1.3.1.1 if obsRel(di+1, q) > obsRel(di,q) + c then continue looking further in the list 1.3.1.2 else di is a good document, click on it. If rel(di,T) is 1, user is DONE Decrease patience Rp. 1.3.2 else Decrease patience Rp Rp = Rp - (rT – obsRel(di,q)) 1.3. 3 Set i = i + 1 1.4 With probability (1 – Preform), user gives up.

  9. User Preference Model • Based on the clickthrough log files, users’ preferences for documents given query q can be found. • Clickthrough logs generated by simulating users. • From preference, features values are calculated.

  10. Feedback Strategies Single Query Strategy • Click >q Skip Above • For query q, if document di is clicked, di is preferred over all dj, j < i. • Click 1st >q No-Click 2nd • For query q, if document 1 is clicked, it is preferred over the 2nd document in the list.

  11. Feedback Strategies 2-Query Strategy 1 • This strategy uses 2 queries in a query chain, but document rankings only for the later query. • Given queries q' and q in a query chain • Click >q' Skip Above • For query q', if document di is clicked in query q, di is preferred over all dj, j < i • Click 1st >q' No-Click 2nd • For query q', if document 1 is clicked, it is preferred over the 2nd document in the list for q

  12. Feedback Strategies 2-Query Strategy 2 • This strategy uses 2 queries in a query chain, and document rankings for both used. • Given queries q' and q in a query chain • Click >q' Skip Earlier Query • For query q', if document di is clicked in query q, di is preferred over seen documents in query previous query. • Click >q' Top two earlier Query • If no document clicked for query q', then di preferred over top two in previous query.

  13. Example

  14. Features • Document di would be mapped to feature vector with respect to query q. • 2 types of features defined • Rank Features • Term/Document Features

  15. Rank Features • Rank features allow representation of ranking given by the existing static retrieval function. • Used a simple TFIDF weighted cosine similarity metric (rel0) • 28 rank features used for ranks 1,2,..,10,15,20,…100. • Set to 1 if clicked document is at or above specified rank.

  16. Term Features • Allows representation of fine grained relationship between query terms and documents. • If for query q, document d is clicked, then for each word , • Forms a sparse feature vector, as only very few words are included in query.

  17. Learning • Retrieval Function rel(di, q) defined as where is the weight vector. • Intuitively, weight vector assigns weight to each feature identified. • Task of learning a ranking function is reduced to the task of learning an optimal weight vector.

  18. How does affect ranking? • Points are ordered by their projections onto • For the ordering will be 1,2,3,4. • For the ordering will be 2,3,1,4. • Weight vector needs to be learnt that will minimize number of discordant rankings.

  19. Learning Problem Learning problem can be formalized as follows • Find weight vector such that maximum of following inequalities fulfilled. such that then • Without using slack variables, this is NP-hard problem.

  20. SVM Learning • Equivalent optimization problem would be Minimize Subject to rearranging which we get constraint and and

  21. Re-ranking using the learnt model • SVM-Light package is used. • Model provides values for all support vectors. • User behavior is again simulated, this time using the learnt ranking function. • How does reranking work? • First, a ranked list of documents is obtained using the original ranking function. • This list is re-ordered, using the weights of each feature obtained from the learnt model.

  22. Experiments • Experiments done to study the behavior of the search engine by varying parameters like • Noise in users’ relevance judgement • Ambiguity of words in topics and queries • Threshold value which user considers good document • Users’ trust in ranking • Users’ probability of reformulation of query.

  23. Results - Noise

  24. Noise – My experiment • Did implementation for extracting preferences and encoding them in features.

  25. Topic and Word Ambiguity

  26. Probability of user reformulating query

  27. Thank You 

More Related