Recommender Systems and Collaborative Filtering

1. Recommender Systems and Collaborative Filtering Jon Herlocker Assistant Professor School of Electrical Engineering and Computer Science Oregon State University Corvallis, OR (also President, MusicStrands, Inc.)

2. Personalized Recommender Systems and Collaborative Filtering (CF)

3. Outline • The recommender system space • Pure collaborative filtering (CF) • CF algorithms for prediction • Evaluation of CF algorithms • CF in web search (if time)

4. Recommender Systems • Help people make decisions • Examples: • Where to spend attention • Where to spend money • Help maintain awareness • Examples: • New products • New information • In both cases • Many options, limited resources

5. Stereotypical Integrator of RS Has: • Large product (item) catalog • With product attributes • Large user base • With user attributes (age, gender, city, country, …) • Evidence of customer preferences • Explicit ratings (powerful, but harder to elicit) • Observations of user activity (purchases, page views, emails, prints, …)

6. Links derived from similar attributes, explicit connections Links derived from similar attributes, similar content, explicit cross references The RS Space Users Items User-User Links Item-ItemLinks Observed preferences (Ratings, purchases, page views, laundry lists, play lists)

7. Individual Personalization Users Items User-User Links Item-ItemLinks Observed preferences

8. Classic CF Users Items User-User Links Item-ItemLinks Observed preferences In the end, most models will be hybrid

9. Collaborative Filtering Unseen items Community Opinions Collaborative Filtering Process Predictions Items you’ve experienced Your Opinions You

10. Find a Restaurant!

11. Find a Restaurant!

12. Find a Restaurant!

13. Find a Restaurant!

14. Find a Restaurant!

15. Find a Restaurant!

16. Find a Restaurant!

17. Advantages of Pure CF • No expensive and error-prone user attributes or item attributes • Incorporates quality and taste • Works on any rate-able item • One data model => many content domains • Serendipity • Users understand it!

18. Predictive Algorithms for Collaborative Filtering

19. Predictive Algorithms for Collaborative Filtering • Frequently proposed taxonomy for collaborative filtering systems • Model-based methods • Build a model offline • Use model to generate recommendations • Original data not needed at predict-time • Instance-based methods • Use the ratings directly to generate recommendations

20. Model-Based Algorithms • Probabilistic Bayesian approaches, clustering, PCA, SVD, etc. • Key ideas • Reduced dimension representations (aggregations) of original data • Ability to reconstruct an approximation of the original data

21. Stereotypical model-based approaches • Lower dimensionality => faster performance • Can explain recommendations • Can over-generalize • Not using the latest data • Force a choice of aggregation dimensions ahead of time

22. Instance-Based Methods • Primarily nearest neighbor approaches • Key ideas • Predict over raw ratings data (sometimes called memory-based methods) • Highly personalized recommendations

23. Stereotypical model-based approaches • Use most up-to-date ratings • Are simple and easy to explain to users • Are unstable when there are few ratings • Have linear (w.r.t. users and items) run-times • Allow a different aggregation method for each user, possibly chosen at runtime

24. Evaluating CF Recommender Systems

25. Evaluation – User Tasks • Evaluation depends on the user task • Most common tasks • Annotation in context • Predict ratings for individual items • Find good items • Produce top-N recommendations • Other possible tasks • Find all good items • Recommend sequence • Many others…

26. Novelty and Trust - Confidence • Tradeoff • High confidence recommendations • Recommendations are obvious • Low utility for user • However, they build trust • Recommendations with high prediction yet lower confidence • Higher variability of error • Higher novelty => higher utility for user

27. Training Set Test Set Test Users Ratings Data Community

28. Predictive Accuracy Metrics • Mean absolute error (MAE) • Most common metric • Characteristics • Assumes errors at all levels in the ranking have equal weight • Sensitive to small changes • Good for “Annotate in Context” task • May not be appropriate for “Find Good Items” task

29. Classification Accuracy Metrics • Precision/Recall • Precision: Ratio of “good” items recommended to number of items recommended • Recall: Ratio of “good” items recommended to the total number of “good” items • Characteristics • Simple, easy to understand • Binary classification of “goodness” • Appropriate for “Find Good Items” • Can be dangerous due to lack of ratings for recommended items

30. ROC Curves • “Relative Operating Characteristic” or “Receiver Operating Characteristic” • Characteristics • Binary classification • Not a single number metric • Covers performance of system at all points in the recommendation list • More complex

33. Prediction to Rating Correlation Metrics • Pearson, Spearman, Kendall • Characteristics • Compare non-binary ranking to non-binary ranking • Rank correlation metrics suffer from “weak orderings” • Can only be computed on rated items • Provide a single score

34. Half-life Utility Metric • Characteristics • Explicitly incorporates idea of decreasing user utility • Tuning parameters reduce comparability • Weak orderings can result in different utilities for the same system ranking • All items rated less than the max contribute equally • Only metric to really consider non-uniform utility

35. Does it Matter What Metric You Use? • An empirical study to gain some insight…

36. Analysis of 432 variations of an algorithm on a 100,000 rating movie dataset

37. Comparison among results provided by all the per-user correlation metrics and the mean average precision per user metric. These metrics have strong linear relationships with each other.

39. Comparison between metrics that are averaged overall rather than per-user. Note the linear relationship between the different metrics.

40. A comparison of representative metrics from the three subsets that were depicted in the previous slides. Within each of the subsets, the metrics strongly agree, but this figure shows that metrics from different subsets do not correlate well

41. Does it Matter What Metric You Use? • Yes.

42. Want to try CF? • CoFE “Collaborative Filtering Engine” • Open source Java • Easy to add new algorithms • Includes testing infrastructure (this month) • Reference implementations of many popular CF algorithms • One high performance algorithm • Production ready (see Furl.net) • http://eecs.oregonstate.edu/iis/CoFE

43. Improving Web Search Using CF With Janet Webster, OSU Libraries

44. Controversial Claim • Improvements in text analysis will substantially improve the search experience • Focus on improving results of Mean Average Precision (MAP) metric

45. Evidence of the Claim • Human Subjects Study by Turpin and Hersh (SIGIR 2001) • Compared human performance of • 1970s search model (basic TF/IDF) • Recent OKAPI search model with greatly improved MAP • Task: locating medical information • No statistical difference

46. Bypass the Hard Problem! • The hard problem – Automatic analysis of text • Software “understanding” language • We propose: Let humans assist with the analysis of text! • Enter Collaborative Filtering

50. The Human Element • Capture and leverage the experience of every user • Recommendations are based on human evaluation • Explicit votes • Inferred votes (implicit) • Recommend (question, document) pairs • Not just documents • Human can determine if questions have similarity • System gets smarter with each use • Not just each new document