Usefulness of Quality Click-through Data for Training

1. Usefulness of Quality Click-through Data for Training Craig Macdonald, ladhOunis Department of Computing Science University of Glasgow, Scotland, UK {craigm,ounis}@dcs.gla.ac.uk WSCD 2009

2. Outline • Abstract • Introduction • Select training query • Rank strategy • Experiments • Conclusions & Future work

3. Abstract • Modern IR system often employ document weighting models with many parameters • To obtain these parameter • This work , use click through for training • Compare

4. Introduction • Parameters which affect the selection and ordering of results • There has been much research done over recent years to develop new methods for training models with many parameters • For instance by attempting to directly optimise rank-based evaluation measures • The wider Learning to Rank combined field of machine learning and information retrieval

5. Introduction • Traditionally, training to find a setting of the parameters using the corresponding relevance judgements. • Test set : a set of unseen queries • Deriving relevance judgements is expensive,

6. Introduction • Examine how quality click-through data • In aggregate form means that no individual user is treated as absolutely correct • Perform an analysis of the usefulness of sampling training data from a large query log • Three different sampling strategies are investigated, and results drawn across three user search tasks

7. Select training query • Data Set : MSN Search Asset Data collection • 15 million query log with click-through documents • 7 million unique queries • Users clicked on documents which are in the GOV Web test collection • 25,375 in the the GOV Web test collection

8. Select training query • Classified Web search queries into three categories: • Navigational query 15-25% • informational query 60% • Transactional query 25-35% • The most frequent queries are often navigational

9. Select training query • Head-First • Ranking the most commonly clicked query-document pairs, we select the top 1000 pairs • Unbiased Randomly • Select 1000 random queries from the query list providing a random sample of both frequent and infrequent queries. • Biased Randomly • Select 1000 random queries from the query list (with repetitions). The queries in this sample are more likely to be frequent.

10. Select training query • TREC Web tracks investigated user retrieval tasks in the Web setting • home page finding task • named page finding task • topic distillation task • For the TREC Web track tasks, each task forms a test collection comprising of a shared corpus of Web documents (the GOV corpus in this case), a set of queries, and corresponding binary relevance judgements made by human assessors. • Relevance assessments is expensive • Automatically derive data

11. Rank strategy • Use textual features from the documents. • PL2F field-based weighting model cfis a hyper-parameter for each field controlling the term frequency normalisation wfcontroll the contribution of the field

12. Training PL2F • 6 parameters: wbody,wanchor,wtitle,cbody,canchor,ctitle • Train the parameters using simulated annealing to directly optimise a given evaluation measure on a training set of queries • simulated annealing would be very time consuming • the independence cfto perform concurrent optimisations

13. Experiments • Compare using the parameter settings obtained from the click-through training and real human relevance judgements • Baseline : trained using a mixed set of TREC Web task queries, with human relevance judgements.

14. Experiments • 8 drop in retrieval performance using the click-through training compared to training on the TREC mixed query tasks. • 5 is significantly better. • 23 is no significant differences

15. Experiments • The random samples are, in general, more effective than the head-first sample • High performance on the home page finding and named page finding tasks. • MAP appears to be marginally better for training using click-through. • Due to the high number of queries which have only one clicked document in the training set

16. Conclusions • Our results show that the click-through data is usually as good as training on bona fide relevance-assessed TREC dataset, and occasionally significantly better

17. Future work • Could be expanded to train many more features: document features • link analysis • URL length • Directly learning features into the ranking strategy.