Prediction Strategies in a TV Recommender System

1. Prediction Strategies in a TV Recommender System Method and Experiments IADIS WWW/Internet 2003, 6 November 2003Mark van Setten, Mettina Veenstra Telematica Instituut, The Netherlands Anton Nijholt, Betsy van DijkUniversity of Twente

2. Outline Introduction Prediction techniques and prediction strategies Generic model Techniques versus strategies Experiment Datasets Method for gathering data Measuring accuracy Validation process Used prediction techniques and strategy Results Conclusions and future work

3. Introduction For information delivery, personalisation is an important issue Address issue of information overload (necessity) Support people to easily find interesting information (user support) Prediction users� interest in information

4. Prediction Techniques Content-Based Techniques: Structured Querying Information Filtering Case-Based Reasoning (CBR) Content Categories Social-Based Techniques: Social Filtering Item-Item Filtering Social CBR Top-N Demographics

5. Generic Model

6. Prediction Strategies Choose one or a combination of prediction techniques at the moment a prediction is required taking into account the most actual knowledge about the current user other users the information for which a prediction is required other information and the system itself

7. Prediction Strategy Approaches Making decisions about which prediction techniques to use and how to combine them Possible approaches: Hard decision rules (if � then � else �) Fuzzy rules Artificial neural networks Bayesian networks Case-based reasoning � In this experiment, we used hard decision rules created by experts

8. Techniques vs Strategies: Blackbox

9. Open Blackbox: Prediction Strategy

10. Open Blackbox: Prediction Technique

11. Main Advantage Independent development of prediction techniques Easy reuse of prediction techniques in different domains Creating a library of prediction techniques (toolkit) Easy creation and tuning of predition strategies Example A prediction technique based on CBR can be created without having any domain knowledge In each domain only one function has to be implemented for CBR to have enough domain knowledge: similarity between two items Prediction strategies provide more accurate predictions

12. Experiments Two datasets MovieLens: Movie Recommendation System TiV: Personalised Electronic TV Guide Results of MovieLens experiment can be found in van Setten, M., Veenstra, M. & Nijholt, A. (2002). Prediction Strategies: Combining Prediction Techniques to Optimize Personalization. Proceedings of workshop �Personalization in Future TV�02� at Hypermedia 2002. Malaga, Spain, 28 May 2002 Results showed that prediction strategies improved the accuracy of the predictions However, question remained if this would also work in different domains

13. TiV

14. Used Prediction Techniques AlreadyRated The rating of an item if the user already rated that item UserAverage The average of all ratings provided by the user TopNDeviation Prediction based on all predictions from other users that already rated the item Social Filtering Prediction based on the idea that people who have rated the same items the same way will probably have similar interests patterns Case-Based Reasoning Prediction based on the idea that if two items are similar and if a rating is known for one of them, the rating for the other will probably be the same

15. Used Prediction Techniques GenreLMS Prediction is based on the learned interests about the main genres of TV programs SubGenreLMS Prediction is based on the learned interests about the sub genres of TV programs InformationFiltering Similar to GenreLMS and SubGenreLMS, except that it uses all (stemmed) words from the description of TV programs, their frequency and the learned interests in these words to determine a prediction Default The neutral prediction value of zero

16. Prediction Strategy

17. TiV Dataset 24 users rated 4 weeks of TV programs containing 40,539 broadcasts distributed over 47 different channels resulting in 31,368 ratings Half-way during the 4 weeks there was the transition of the summer TV season to the winter TV season

18. Measuring Accuracy Accuracy measure, combination of: Mean Absolute Error (mae) = Coverage = Global Mean Absolute Error (gmae) = mae, except when no prediction can be made the neutral prediction value (zero) is assumed Calculate gmae for each prediction technique and each prediction strategy, including the main strategy Perform paired samples T-tests with 95% confidence interval to determine if differences in gmae are statistically valid (p < 0.05)

19. Measuring Accuracy Validating throughout system lifecyle: Validation throughout usage lifecycle First 100 ratings of each user Remaining ratings of each user

20. Validation Process

21. Validation Process

22. Validation Process

23. Validation Process

24. Validation Process

25. Validation Process

26. Validation Process

27. Validation Process

28. Validation Process

29. Validation Process

30. Validation Process

31. Validation Process

32. Validation Process

33. Validation Process

34. Validation Process

35. Validation Process

36. Validation Process

37. Validation Process

38. Validation Process

39. Validation Process

40. Validation Process

41. Validation Process

42. Validation Process

43. Validation Process

44. Accuracy

45. Used Techniques

46. First Time versus Established Users

47. Removing Prediction Techniques

48. Conclusions Described a new way of looking at prediction engines using a generic method based on prediction strategies Prediction strategies make it possible to quickly create, use and test different strategies using several prediction techniques Showed that prediction strategies provide more accurate predictions because they only decide at the moment a prediction is required which prediction technique(s) to use Be aware of drawback: performance penalty

49. Future Work Automated prediction strategies Using algorithms that can teach themselves when which prediction techniques can best be used Combining predictions of multiple prediction techniques First results show that combining predictions result in worse performance Looking at motiviations of people to be interested in information Let those (domain dependent) motivations guide the design of prediction strategies

50. For More Information