Reputation Systems For Open Collaboration, CACM 2010 Bo Adler, Luca de Alfaro et al.

1. Reputation Systems For Open Collaboration, CACM 2010 Bo Adler, Luca de Alfaro et al. Nishith Agarwal nishitha@usc.edu

2. What are Reputation Systems? • Wikipedia Definition: A reputation system computes reputation scores for a set of objects within a community or domain, based on a collection of opinions other users hold about the objects. • Why do we need them? • Reputation Systems can help stem abuse of content, and can offer indications of content quality. • In many ways, reputation systems are the on-line equivalent of the body of laws that regulates the real-world interaction of people. • Who uses reputation systems?

3. Types of Reputation Systems?

4. WikiTrust • A content-driven reputation system for Wiki authors and content on Wikipedia • Goals: • Incentivize users to give lasting contributions • Help users and editors increase the quality of content • Offer content consumers a guide to content quality • Components: • User reputation system: Users gain reputation when they add content that is preserved by subsequent users. • Content reputation system: Content gains reputation when it is revised by highly reputed authors. • Firefox Extension: Lets users view content reputation by changing text background color.

5. WikiTrust – User Reputation System Contribution quality: Relies on edit distance between revisions. • -1 if changes made by b are completely reverted • +1 if changes made by b are completely preserved • Contributions are considered good quality if the change is preserved in subsequent revisions • User reputation is computed according to quality and quantity of contributions they make • User Reputation: • Is proportional to the edit distance and contribution quality of b. • r(B) ≈ d(a,b) + q(b | a,c) + r(C) • r(B) being the reputation of author B of revision b • r(C) being the reputation of author C of revision c

6. WikiTrust – Content Reputation System (TextTrust) • Based on extent to which content was revised, and reputation of users who revised it. • High content reputation requires consensus from reputed authors. Basic Algorithm: • Content that is edited is assigned a small fraction of the revision user’s reputation. • Unedited content gains more reputation. Some Tweaks: • Ensures that re-arranging or deleting text leaves a low reputation mark • Content reputation cannot exceed the revision user’s reputation • Users cannot raise arbitrary reputation by multiple edits

7. Crowdsensus • A content-driven reputation system built to analyze user edits to Google Maps • Goals: • To measure accuracy of a user who contributes information • To display accurate details for a business (title, address, phone etc.) • Differences from WikiTrust: • There exists a “ground truth” • User reputation is not visible. Hence, no need to keep algorithm simple. • User identity is stronger

8. Crowdsensus - Algorithm Structured as fixed point graph algorithm • Vertices are users u, and business attributes a • Edges are attribute values v • Each user has truthfulness value qu Algorithm Details: • User vertices send (qu , vu) pairs to value vertices • An attribute inference algorithm is used to derive probability distribution over values (v1 , v2..vn) • Crowdsensus sends back to user u, the estimate probability that vu iscorrect • Different attribute inference algorithms tailored to every attribute type Comparison to Bayesian Inference Model: • For 1000 attributes, 100 users, 10 attribute values: • CrowdSensus error rate: 2.8% • Bayesian error rate: 7.9% • For 1000 attributes, 100 users, 5 attribute values • CrowdSensus error rate: 12.6% • Bayesian error rate: 22%

9. Design Considerations

10. Conclusion Research Directions: • How can reputation systems lead to happy, active, healthy communities? • How can we build reputation systems that meet multiple goals?

11. Pros

12. Cons