Opinion Mining using Econometrics A Case Study on Reputation Systems

1. Opinion Mining using Econometrics A Case Study on Reputation Systems Panos Ipeirotis New York University Joint work with Anindya Ghose and Arun Sundararajan

2. Comparative Shopping in e-Marketplaces

3. Are Customers Irrational?

4. Are Customers Irrational? $18.28 $11.04 -$0.61 -$1.04 -$9.00 -$11.40 Price Premiums (customers pay more than the minimum price)

5. Price premiums @ Amazon Irrational (?)

6. Average price premiums @ Amazon Irrational (?)

7. Why not buying the cheapest? You buy more than a product • Customers do not pay only for the product • Customers also pay for a set of fulfillment characteristics • Delivery • Packaging • Responsiveness • … Reputation Matters!

8. Reputation Systems Facilitate electronic commerce Reputation in ecommerce is complex • Integral part of online marketplaces • Provide information about unobserved fulfillment characteristics (most of which we take for granted in traditional commerce) • Different buyers value different fulfillment characteristics • Sellers have varying abilities on these characteristics

9. Example of a reputation profile

11. Reputation profiles: Observations Reputation profile capture more than “averages” Reputation in ecommerce is complex • Well beyond “average score” and “lifetime” • Rich textual content: information about a seller on a variety of dimensions (fulfillment characteristics). • How the seller’s performance (potentially on each of these characteristics) has evolved over time • Buyer-seller networks • Different buyers value different fulfillment characteristics • Sellers have varying abilities on these characteristics • Previous work studied only effect of “average score” and “lifetime”

12. Our research questions What are the dimensions of online reputation? How to evaluate the reputation across these dimensions? Can prior reputation predict marketplace outcomes? • What characteristics comprise the important parts of a seller’s overall reputation? (politeness? packaging? delivery?) • How can we measure the reputation across each dimension? • How can we measure polarity and strength of each individual evaluation? • Is good service better than ok service? • Is superfast delivery faster than supersuperfast delivery? • Is good packaging a positive evaluation? • Given a set of sellers, their reputations, and their prices, can one predict which seller will successfully make the sale?

13. Data Overview Summary • Panel of 280 software products sold by Amazon.com • Data on all “secondary” market transactions • Amazon Web services facilitate capturing transactions • Complete reputation profile for all sellers who completed one or more transactions during this period • 280 products X 180 days • 1,078 sellers, of which 122 transacted • 12,232 transactions • 107,922 “observations” (seller-competitor pairs)

14. Data: Transactions

15. Data: Transactions Sales of (mostly new) software

16. Data: Transactions Capturing transactions and “price premiums” Item Listing Price Seller When item is sold, listing disappears

17. Data: Transactions Capturing transactions and “price premiums” Jan 1 Jan 2 Jan 3 Jan 4 Jan 5 Jan 6 Jan 7 Jan 8 Jan 9 Jan 10 time While listing appears, item is still available

18. Data: Transactions Capturing transactions and “price premiums” Jan 1 Jan 2 Jan 3 Jan 4 Jan 5 Jan 6 Jan 7 Jan 8 Jan 9 Jan 10 time Item still not sold on 1/7 While listing appears, item is still available

19. Data: Transactions Capturing transactions and “price premiums” Jan 1 Jan 2 Jan 3 Jan 4 Jan 5 Jan 6 Jan 7 Jan 8 Jan 9 Jan 10 time Item sold on 1/9 When item is sold, listing disappears

20. Data: Variables of Interest Price Premium Average Price Premium • Difference in the price charged by a seller and the listed price of a competing seller at the time the transaction occurred (Seller Price – Competitor Price) • Calculated for each seller-competitor pair, for each transaction • Each transaction generates M observations, (M: number of competing sellers) • Difference in the price charged by a seller and the average price of all competing sellers at the time the transaction occurred (Seller Price – Avg. (Competitor Price) ) • Calculated for each transaction • Each transaction generates 1 observation

21. Price premiums @ Amazon

22. Average price premiums @ Amazon

23. The dimensions of reputation Our conjecture: Price premiums measure reputation

24. Decomposing reputation Is reputation just a scalar metric? • Previous studies assumed a “monolithic” reputation. • We break down reputation in individual components • Sellers characterized by a set of n fulfillment characteristics

25. Decomposing reputation What are these characteristics (valued by consumers?) • We think of each characteristic as a dimension, represented by a noun or verb phrase (“shipping”, “packaging”, “delivery”, “arrived”) • We scan the textual feedback to discover these dimensions

28. Decomposing and scoring reputation Decomposing and scoring reputation • We think of each characteristic as a dimension, represented by a noun or verb phrase (“shipping”, “packaging”, “delivery”, “arrived”) • The sellers are rated on these dimensions by buyers using modifiers (adjectives or adverbs), not numerical scores • “Fast shipping!” • “Great packaging” • “Awesome unresponsiveness” • “Unbelievable delays” • “Unbelievable price”

29. Dimensions from text: Example Parsing the feedback Identified modifier-dimension pairs Reducing textual feedback to a n X p matrix • P1: I was impressed by the speedydelivery! Great Service! • P2: The item arrived in awful packaging, and the delivery was slow • … • P1: “speedy – delivery”, “great – service” • P2: “awful – packaging”, “slow – delivery” • … • Dimensions: 1-delivery, 2-packaging, 3-service Postings

30. Decomposing and scoring reputation Scoring reputation • “Fast shipping!” • “Great packaging” • “Awesome unresponsiveness” • “Unbelievable delays” • “Unbelievable price” How can we find out the meaning of these adjectives?

31. The dimensions of reputation • We assume that each modifier assigns a “score” to each dimension • :score associated with m appearing as the modifier for the k-th dimension • ri: weight of posting that appears on the i-th position (weight down old posts) • wi: weight assigned to the i-th dimension • Thus, the overall (text) reputation score Π(i) is: Sum of ri weights in which mj modifies dimension i scores for n-th dimension scores for first dimension estimated coefficients scores for first posting

32. The dimensions of reputation Scoring the dimensions Regressions • Use price premiums as “true” reputation score • Use regression to assess scores (coefficients) for each dimension-modifier pair • Control for all variables that affect price premiums • Control for all numeric scores of reputation • Examine effect of text: E.g., seller with “fast delivery” has premium $10 over seller with “slow delivery” estimated coefficients

33. Some indicative dollar values Natural method for extraction of sentiment strength and polarity Positive Negative

34. Results Some dimensions that matter • Delivery and contract fulfillment (extent and speed) • Product quality and appropriate description • Packaging • Customer service • Price (!) • Responsiveness/Communication (speed and quality) • Overall feeling (transaction)

35. Results Further evidence • Classifier (aka choice model) that predicts sale given set of sellers • Binary decision between seller and competitor • Naïve Bayes and Decision Trees (SVM’s forthcoming) • Only prices and characteristics: 53% • + numerical reputation, lifetime: 74% • + encoded textual information: 89%

36. Other applications Summarize and query reputation data Pricing reputation • Give me all merchants that deliver fast SELECT merchant FROM reputation WHERE delivery > ‘fast’ • Summarize reputation of seller XYZ Inc. • Delivery: 3.8/5 • Responsiveness: 4.8/5 • Packaging: 4.9/5 • Given the competition, merchant XYZ can charge $20 more and still make the sale (confidence: 83%)

37. Summary Key contributions Broader contribution • New technique that automatically scores “sentiment” based on economic data • Validation by multiple methods (estimating an econometric model, building classifiers) • New evidence of the extent to which interdisciplinary research can be fun and distracting • Economic data is abundant and there is rich literature on how to handle such data • Economic data can be used for training for MANY applications

38. Thank you! http://economining.stern.nyu.edu

39. Prior studies of reputation Positive feedback significant, negative not Negative feedback significant, positive not Both positive and negative feedback significant • Ba and Pavlou (2002) for CD’s, software, electronics; Bajari and Hortacsu (2003) for collectible coins? • Lee et al. (2000) for computer equipment, Reiley et al. (2000) for collectible coins • Dewan and Hsu (2004) for rare stamps, Melnik and Alm (2002) for gold coins, Houser and Wooders (2005) for Pentium chips Nature of price: winning online auction bid (usually eBay) Measure of reputation: average numerical score, # of transactions