The Social Aspect of Voting for Useful Reviews

1. The Social Aspect of Voting for Useful Reviews Asher Levi Osnat (Ossi) Mokryn International Social Computing, Behavioral Modeling and Prediction Conference (SBP 2014) April 2014, Washington DC USA.

2. Motivation • Reviews: opinions on products • Useful / helpful count: Crowdvoting • Early detection of useful reviews: • Users / sites • Improved shopping experience The Social Aspect of Voting for Useful Reviews. Levi, Mokryn. SBP 2014.

3. Agenda • Classify whether a review is useful or not • Predict count of useful votes per review The Social Aspect of Voting for Useful Reviews. Levi, Mokryn. SBP 2014.

4. Main Previous Findings • Text-and-user-based prediction • Lu at al. [WWW 2010] –review quality prediction • Text-based predictor +social context features • Number of reviews, average rating and graph of trusted users. • Ghose at al. [Trans.on.Knowledge 2011] - classification • Text features (number of sentences, words, etc.), readability of the text • Reviewer: subjectivity, characteristics (top 50, 100 etc.) • Average accuracy is 85%, and the AUC is 0.87 • Text-sentiment-based prediction • Siersdorfer at el. [WWW 2010] YouTube comments usefulness • Words td-idf + sentiment features for SVM. • Precision-recall break-even points (BEPs) ~ 0.73 on average • In Min Kim at al. [2006] , Xiong at al. [ACL 2011] – useful prediction • Text and semantic: text, lexical (unigram/bigram), semantic words, rating. • Pearson correlation (r) is 0.65 and 0.62, respectively. The Social Aspect of Voting for Useful Reviews. Levi, Mokryn. SBP 2014.

5. Spotlight: Exploiting Social Context for Review Quality Prediction [www’10] • Directed social graph of reviewers • Link from user A to B if A voted on a reviews written by B AND A trusts B • Users’ ranks and relations hypotheses: • An author reviews rank similar (consistency) • User A trusts B if B is ranked at least as high • A “PageRank” algorithm to calculate user’s importance • Users’ consistency improved prediction by up to 13% (over text-based prediction only), avg is 6%. The Social Aspect of Voting for Useful Reviews. Levi, Mokryn. SBP 2014.

6. Observations • Reviewer’s impact • Can we define influential writers? • Do people prefer (and indicate as useful) reviews written by influential reviewers? • Emotions • From Psychology: “what you feel is how you compare” • Is it possible that reviews that induce feelings are seen as more useful? • Text Features • Are useful reviews longer, do they obtain more information? The Social Aspect of Voting for Useful Reviews. Levi, Mokryn. SBP 2014.

7. Common Measures of Reviewer’s Impact • Yelp • Yelpers form a social network • Followers, friends, … • Elite members elected yearly • Based on ‘Good Yelp Citizenship’ • Online badges, offline parties • Amazon • High relative useful score • Percent of useful votes out of total over all reviews • Online badges The Social Aspect of Voting for Useful Reviews. Levi, Mokryn. SBP 2014.

8. Detour: Impact in Science • In science:“A scientist has index h if h of his/herNppapers have at leasth citations each, and the other (Np − h) papers have no more thanh citations each.” [Wiki] The Social Aspect of Voting for Useful Reviews. Levi, Mokryn. SBP 2014.

9. Reviewer Impact • Reviewer h-index • A reviewer has index h if h of herNpreviews have at leasth useful votes each, and the other (Np − h) reviews have no more thanh useful votes each. • Hence, an h-index of 10 means the reviewer has at least 10 reviews which were voted useful by at least 10 other people, and the rest of the reviewer’s reviews have less than 10 useful votes each. • Reviewer -index • A reviewer has index if s/he has reviews with at least 5 useful votes each. The Social Aspect of Voting for Useful Reviews. Levi, Mokryn. SBP 2014.

10. Calculated for Yelp Bay Area Dataset The Social Aspect of Voting for Useful Reviews. Levi, Mokryn. SBP 2014.

11. Calculated for Yelp Bay Area Dataset The Social Aspect of Voting for Useful Reviews. Levi, Mokryn. SBP 2014.

12. Emotions • We are trained from infancy to identify emotions in faces • Hence, icons are international cross-cultural symbols • Emotions are contagious • Psychology: Social induction by effect The Social Aspect of Voting for Useful Reviews. Levi, Mokryn. SBP 2014.

13. Basic Emotions Advanced Emotions Plutchik wheel of emotions The Social Aspect of Voting for Useful Reviews. Levi, Mokryn. SBP 2014.

14. NLP: Emotions in Text • Recently, exciting research mainly on these Datasets: • SemEval-2007: Affective Text corpus • newspaper headlines labeled with the six Ekman emotions by six annotators [Strapparava and Mihalcea, 2007] • ISEAR (International Survey on Emotion Antecedents and Reactions): • 7,666 sentences annotated by 1,096 participants with different cultural backgrounds [Scherer and Wallbott,1994] • Fairy Tales: • Grimms’, H.C. Andersen’s, and B. Potter’s stories. • labeled with five emotion categories [alm 2009] • GEMEP (The Geneva Multimodal Emotion Portrayal Corpus): • Over 7,000 audio-video emotional portrayals, representing 18 emotions [Benziger, Scherer 2007] The Social Aspect of Voting for Useful Reviews. Levi, Mokryn. SBP 2014.

15. Existing Emotion Detection Approaches • Corpus-driven approach • Large, structured annotated set of tagged texts (supervised). • Thesaurus/Lexicon based approach • Synonyms/glosses of lexical resources • Determines emotion of words or documents. • ANEW: Basic Valance, Arousal, Dominance [Bradley at al. ‘99] • Affective WordNet (Senti-WordNet) [Strapparava at al. ‘04] • NRC: emotions [Mohammad, Turney‘11] • Crowdsourcing-based (mechanical-turk) • Each word is tagged as {0,1} per emotion The Social Aspect of Voting for Useful Reviews. Levi, Mokryn. SBP 2014.

16. Calculated for Yelp Bay Area Dataset The Social Aspect of Voting for Useful Reviews. Levi, Mokryn. SBP 2014.

17. Calculated for Yelp Bay Area Dataset The Social Aspect of Voting for Useful Reviews. Levi, Mokryn. SBP 2014.

18. Calculated for Yelp Bay Area Dataset The Social Aspect of Voting for Useful Reviews. Levi, Mokryn. SBP 2014.

19. Text • Text contains additional information • The following might seem more useful: • Informative reviews • Expressive reviews • Punctuations (!!!) • Questions marks • Longer reviews The Social Aspect of Voting for Useful Reviews. Levi, Mokryn. SBP 2014.

20. Calculated for Yelp Bay Area Dataset The Social Aspect of Voting for Useful Reviews. Levi, Mokryn. SBP 2014.

21. Reviews score of positive/negative : Calculated for Yelp Bay Area Dataset The Social Aspect of Voting for Useful Reviews. Levi, Mokryn. SBP 2014.

22. # of words in review: Calculated for Yelp Bay Area Dataset The Social Aspect of Voting for Useful Reviews. Levi, Mokryn. SBP 2014.

23. Hypotheses • Users with high i-index and h-index write useful reviews • Reviews expressing emotions are more likely to induce feelings and be perceived as useful • Reviews that contain more information (have a higher nouns/adjs relation), and other expressive text expressions are more likely to be perceived as useful The Social Aspect of Voting for Useful Reviews. Levi, Mokryn. SBP 2014.

24. Data Sets The Social Aspect of Voting for Useful Reviews. Levi, Mokryn. SBP 2014.

25. Text Features • Word count • Sentence count • Average words in sentence • Number of punctuations • Number of adjectives, nouns, adverbs, verbs • Number of question sentences • Number of exclamation marks (!!!) The Social Aspect of Voting for Useful Reviews. Levi, Mokryn. SBP 2014.

26. Features Users Emotions and sentiment Basic emotions Anger, anticipation, disgust, fear, joy, sadness, surprise, trust Advanced emotions Optimism, love, submission, awe, disappointment, remorse, contempt, aggression Sentiment: Positive, negative • Number of reviews • User useful average, sum • Products average rating • Product rating • h-index • i5 index The Social Aspect of Voting for Useful Reviews. Levi, Mokryn. SBP 2014.

27. Amazon Useful metrics • A review is considered useful if: • At least 5 votes AND At least 50% found it helpful • Otherwise – not useful • Rate prediction: Predict actual helpful votes for useful reviews • In this example: 198 The Social Aspect of Voting for Useful Reviews. Levi, Mokryn. SBP 2014.

28. YELPUsefulmetrics • A review is considered useful if: • At least 5 useful • Otherwise – not useful • Rate prediction: Predict actual useful votes for useful reviews • In this example: 13 • IMDB: The Social Aspect of Voting for Useful Reviews. Levi, Mokryn. SBP 2014.

29. Yelp Useful Distribution The Social Aspect of Voting for Useful Reviews. Levi, Mokryn. SBP 2014.

30. Evaluation • SVM classifier • Stratified6-fold cross validation • Results: accuracy and area under the ROC curve. The Social Aspect of Voting for Useful Reviews. Levi, Mokryn. SBP 2014.

31. Results - Classification General hypothesis: User Metrics + Emotions + Text Features The Social Aspect of Voting for Useful Reviews. Levi, Mokryn. SBP 2014.

32. Results - Rate prediction (linear regression) General hypothesis: User Metrics + Emotions + Text Features The Social Aspect of Voting for Useful Reviews. Levi, Mokryn. SBP 2014.

33. Evaluating Emotions Effect General hypothesis: Emotions The Social Aspect of Voting for Useful Reviews. Levi, Mokryn. SBP 2014.

34. Evaluating Text Features Effect General hypothesis: Text The Social Aspect of Voting for Useful Reviews. Levi, Mokryn. SBP 2014.

35. Evaluating Text + Emotions General hypothesis: Emotions + Text Features The Social Aspect of Voting for Useful Reviews. Levi, Mokryn. SBP 2014.

36. Evaluating Users’ Effect General hypothesis: User Info The Social Aspect of Voting for Useful Reviews. Levi, Mokryn. SBP 2014.

37. Conclusion: It is about the People Possible explanations • Experts shine • Their review is really something “outa” this world • Preferential attachment • If we see a review by someone considered influential, we tend to give it a higher value • Alternate explanation? The Social Aspect of Voting for Useful Reviews. Levi, Mokryn. SBP 2014.

38. Reviewers with high h-index write a lot of reviews • …… Or is it the other way around? The Social Aspect of Voting for Useful Reviews. Levi, Mokryn. SBP 2014.

39. Trust and Familiarity • Trust: prerequisite of social behavior, especially regarding important decisions • Familiarity: precondition for trust • “I saw one of her reviews before and liked it”Familiarity + positive experience = trust • More likely to pay attention to her reviews • More likely to rate it positively (‘useful’) • Local Celebrity The Social Aspect of Voting for Useful Reviews. Levi, Mokryn. SBP 2014.

40. The Social Aspect of Voting for Useful Reviews. Levi, Mokryn. SBP 2014.

41. The Social Aspect of Voting for Useful Reviews. Levi, Mokryn. SBP 2014.

42. The Social Aspect of Voting for Useful Reviews. Levi, Mokryn. SBP 2014.

43. Conclusions Reviewers investing time and energy in writing reviews become known in on- line communities, in what we term local celebs, and their reviews are therefore perceived as useful. The Social Aspect of Voting for Useful Reviews. Levi, Mokryn. SBP 2014.

44. Future Work: Is it about the PeopleAnd their decision processwhich in this case is irrational? Every person would claim their vote was on the usefulness of the review. Can we show that their decision process was influenced by their experience And that their decision was not necessarily a rational one? The Social Aspect of Voting for Useful Reviews. Levi, Mokryn. SBP 2014.