180 likes | 319 Views
Anish Das Sarma , Atish Das Sarma , Sreenivas Gollapudi , Rina Panigrahy WSDM’10, February 4–6, 2010, pp.21-30. Ranking Mechanisms in Twitter-like Forums. Presented by Wei-Ding Liao. Outline. Introduction A taxonomy of approaches Desirable properties Preliminaries
E N D
Anish Das Sarma, Atish Das Sarma, SreenivasGollapudi, RinaPanigrahy WSDM’10, February 4–6, 2010, pp.21-30 Ranking Mechanisms in Twitter-like Forums Presented by Wei-Ding Liao Natural Language Processing Lab
Outline • Introduction • A taxonomy of approaches • Desirable properties • Preliminaries • Scheduling items for review • Shoutvelocity system • Experimental results • Conclusions Natural Language Processing Lab
Introduction(1/2) • Drawbacks of popular method such as “star-ratings”, “thumb up-down ratings”, “reputation points” • Aid in the rich gets richer phenomena. • Giving an item a score, independent of other items results in unnormalized scores. • In the absence of any incentives, it is impractical to expect all users to participate in the feedback process. • Effectiveness of a rating mechanism. • Incentive/reward systems coupled with user feedback. • Comparison-based ranking. Natural Language Processing Lab
Introduction(2/2) • Advocate a comparison-based ranking scheme and implements a system called Shoutvelocity. • Feedback from users are sought in the form of comparison. • Users are shown a pair of items and express which they prefer. • Both theoretical & empirical results shows that it achieves good rankings with very little feedback from users. • Mitigates drawbacks mentioned above. • The techniques are ideally suited for • Ranking posts in public forums. digg, twitter, etc. • Ranking message on social networks such as FB. • Generic or personalized movie recommendations, IMDb. • Ranking multimedia photo or video. Flicker, Youtube. Natural Language Processing Lab
A taxonomy of approaches(1/2) Natural Language Processing Lab
A taxonomy of approaches(2/2) • Review module • Explicit • If there is a separate rate link that when a reader clicks is shown one or more specific item to rare. • Implicit • The reader rates as s/he is browsing through the entries. • 2 approaches to reviewing items • Independent scoring • Each item is independently shown to a use, and s/he scores the item based on how much they like the item. • Comparison-based scoring • A pair of items is shown to a user, and s/he responds by only telling the system which item they find better. Natural Language Processing Lab
Desirable properties • Ranking accuracy • The system should be able to rank the items as accurately as possible within the review budget. • Review feedback bandwidth • Ranking should converge to the correct one within the desired level of accuracy quickly with a small amount of feedback per item. • Low latency • Users should not have to wait long before receiving an estimate on their score/rank. • Fairness • Items should be treated equally with respect to ranking and allocation of review bandwidth. Natural Language Processing Lab
Preliminaries(1/2) • Probability model. • Feedback BW and ranking accuracy of an Alg. • The distribution of scores of the item. • g(x): probability density function of the scores. • gc(x): cumulative distribution function. • How items are rated by the Alg. • : normal random variable with mean 0 & variance exceeds x. • Logistic function: Natural Language Processing Lab
Preliminaries(2/2) • Estimating scores from reviews. • Thumbs • Fraction of reviews in which the item received a thumbs-up rating. • Comparisons • Elo rating system.[Arpad Elo, 1978.] • x’ = x + K(SA-EA) • EA: the point that suppose A was expected to receive in a comparison. • SA: the actually received point. • K: a parameter that decays with the number of times the item has been reviewed. • x’ = x +K(SA-) in probability model • SA=1 if A wins, otherwise SA=0 Natural Language Processing Lab
Scheduling items for review(1/2) • Thumb-based Alg. • Can not approximate the ranks of items within some multiplicative error with bounded feedback. • Comparison-base ranking Alg. • Static. • Dynamic. Natural Language Processing Lab
Scheduling items for review(2/2) • Rank all items by the time discounted score & pick an item with rank r with probability proportional to • γ is a parameter & compare it with next ranking. • γ = 0 • Sampling uniformly. • 0 < γ < 1 • Sampling biased towards the items with higher score • γ > 1 • The bias is large & most of the probability is concentrated in the top few ranks. Natural Language Processing Lab
Shoutvelocitysystem(1/2) • System architecture Natural Language Processing Lab
Shoutvelocitysystem(2/2) • Top shouts • Review screen Natural Language Processing Lab
Experimental results(1/4) • Simulation over synthetic data • Thumb-based approach • Pick a random item biased towards those with high score estimates. • Pick the rth item with probability proportional to • In each thumb review, the item with score x wins with probability • Comparison-based approach • Pick the rth item with probability proportional to • Supposed the rthis picked, the 2nd item is the r+1th ranked one. • The item with score x wins with probability f(x-y) Natural Language Processing Lab
Experimental results(2/4) • Comparison-based approach(cont’d) • Discounting factor • Supposed items i1, i2 have been evaluated k1,k2 times. • ci=, where i=1,2 • If i1 gets voted in this comparison • S1 is incremented by • S2 in decremented by • Evaluation metric • MRR(Mean Reciprocal Rank) • MRR is at most 1 and an MRR of 1 means the top item was always correctly identified as the best item. Natural Language Processing Lab
Experimental results(3/4) Natural Language Processing Lab
Experimental results(4/4) • Shoutvelocitysystem • 4853 pairwise comparisons performed by users, over a set of 1245 items. Natural Language Processing Lab
Conclusions • Addressed the problem of designing ranking mechanisms for forums. • Studied independent thumb-based & comparison-based reviewing of items in forums. • Shoutvelocity, an online forum that fully implements the comparison-based ranking mechanism. • Experimental results showed that shoutvelocity comparison-based ranking significantly outperforms thumb-based ranking on the desired properties. Natural Language Processing Lab