If You Build It? Benefits and Costs of Creating Your Own Online Community

1. If You Build It? Benefits and Costs of Creating Your Own Online Community Loren Terveen Computer Science & Engineering The University of Minnesota August 2011

2. Background: ways of knowing • Theory • Simulation • Lab studies • Surveys • Qualitative studies • Build and learn • (e.g., Google, Facebook, Wikipedia) • Build To Learn

3. Build to learn • GroupLens Research • Create new interaction / social computing techniques • Do empirical, quantitative research • Learn from what we and others build

4. To answer the kinds of research questions we like to ask, we need: • Data • Experimental Control

5. The rest of the talk • Learning from others’ data • Learning from our own data • Exercising experimental control

6. 1. Learning from others’ data • Q&A systems • Wikipedia

7. GroupLens Wikipedia Research • WP:Clubhouse? An Exploration of Wikipedia’s Gender Imbalance. Lam, S.K., Uduwage, A., Dong, Z., Sen, S., Musicant, D.R., Terveen, L., Riedl, J. WikiSym 2011. • NICE: Social translucence through UI intervention. A. Halfaker, B. Song, D. A. Stuart, A. Kittur and J. Riedl. Wikisym 2011. • Don't bite the Newbies: How Reverts Affect the Quantity and Quality of Wikipedia Work. A. Halfaker, A. Kittur and J. Riedl. Wikisym2011. • Mentoring in Wikipedia: A Clash of Cultures. D. Musicant, Y. Ren, J. Johnson and J. Riedl. Wikisym2011. • The Effects of Group Composition on Decision Quality in a Social Production Community, Lam, S.K., Karim, J., Riedl, J. Group 2010. • The Effects of Diversity on Group Productivity and Member Withdrawal in Online Volunteer Groups, Chen, J., Ren, Y., Riedl, J. CHI 2010. • rv you're dumb: Identifying Discarded Work in Wiki Article History, Ekstrand, M.D., Riedl, J.T. Wikisym 2009. • A Jury of Your Peers: Quality, Experience and Ownership in Wikipedia, Halfaker, A., Kittur, N., Kraut, R., Riedl, J. Wikisym 2009. • Is Wikipedia Growing a Longer Tail?, Lam, S.K., Riedl, J. Group 2009. • Wikipedians are born, not made: a study of power editors on Wikipedia, Panciera, K., Halfaker, A., Terveen, L. Group 2009. • SuggestBot: Using Intelligent Task Routing to Help People Find Work in Wikipedia, Cosley, D., Frankowski, D., Terveen, L., Riedl, J. IUI 2007. • Creating, Destroying, and Restoring Value in Wikipedia, Priedhorsky, R., Chen, J., Lam, S.K., Panciera, K., Terveen, L., Riedl, J. Group 2007.

8. WP:Clubhouse? An Exploration of Wikipedia’s Gender Imbalance. Lam, S.K., Uduwage, A., Dong, Z., Sen, S., Musicant, D.R., Terveen, L., Riedl, J. • www.grouplens.org/node/466

9. Trigger… • http://www.nytimes.com/2011/01/31/business/media/31link.html?_r=1&src=busln • A topic generally restricted to teenage girls, like friendship bracelets, can seem short at four paragraphs when compared with lengthy articles on something boys might favor, like, toy soldiers or baseball cards, whose voluminous entry includes a detailed chronological history of the subject. • (BTW, it’s not about the friendship bracelets)

10. Findings • Only 16% of new editors joining Wikipedia during 2009 identified themselves as women • Women made only 9% of the edits by this cohort • New women editors are more likely to stop editing and leave Wikipedia when their edits are reverted • Topics of particular interest to women appear to get less (and poorer) coverage in Wikipedia • (Hmm… maybe Wikipedia has a low collective IQ!) • Come to Wikisym to get the details!

11. 2. Learning from our own data

12. GroupLens online communities • MovieLens • Cyclopath

14. 200 Union St SE Lagoon Theatre

16. Research Question • How do contributors to open content systems become contributors? • Inspired by…

17. Becoming WikipedianBryant, Forte, & Bruckman 2005 • Wikipedians fill different niches than non-Wikipedians • Wikipedians branch out to new areas and topics as they mature • Wikipedians take on more “community work” as they mature Qualitative study with nine participants self-reporting

18. Our goal: test these findings quantitatively Evidence for “becoming”? Quantity of work Quality of work Nature of work Are Wikipedians Born or Made?

19. Wikipedian A registered editor with 250+ edits over his/her lifetime If editors reach 250 edits within our data set, they are labeled Wikipedian from the beginning

20. Data English Wikipedia dump (January 13, 2008) Edits from bots and other non-human means removed We counted: Only registered editors Wikipedians (users with 250+ edits) - 38K Non-wikipedians - random sample of 38K Edits per day per editor (“User days”) (“Day 1”)

21. Quantity

25. Is a user’s fate sealed? Wikipedians are Born Made Quantity

26. Quality • Measure: Persistent Word Revisions (PWRs) • Proportion of words added that persist five revisions

30. Other quality metrics? Wikipedians are Born Made Quality

31. Nature of Work • Conjecture: Wikipedians take on community maintenance work over time • Several ways to formalize • Editing in “talk” (and other) namespaces • (Nope: still “born”) • Referring to “community norms” (Wikipedia policies) to explain edits

35. Community Learning norms vs. learning to appeal to the norms? Training: effective editing Wikipedians are Born Made

36. Summary of findings • Common pattern: Initial burst of activity, decline, steady state • Wikipedians look different from day one • Little evidence for “Becoming Wikipedian”: Wikipedians are born, not made • Can we reconcile? • This is depressing! • Possible responses: • Early interventions • Change the culture • Systemic initiatives, e.g., APS Wikipedia Initiative: http://www.psychologicalscience.org/index.php/members/aps-wikipedia-initiative • Accept the reality of the long tail

37. But: methodological worries • We can’t ask Wikipedia users about our interpretations • What if the learning happened before users registered?

38. Cyclopath: viewing and pre-registration activities are visible • As of September 2009, we identified: • 1172 “unambiguous” users • 268 of these users made some edits • 440 “ambiguous” users • For unambiguous users • Day 1 = First time a user came to the site (not the day they registered)

39. Same pattern as for Wikipedia

40. And few users edited before registration

41. Some viewing before registration A minute or two <= 5 min. <= 15 <= 30 <= 60

42. But amount of viewing before registration (or before editing) does not predict subsequent behavior “Born, Not Made” still seems true

43. Followups • Cyclopath user surveys – Wikisym 2011 paper • Why these patterns? • What ‘triggers’ initial contribution? • And how might we nurture ongoing participation? • Cyclopath contextual interviews • planned

44. 3. Exercising Experimental Control

45. Research Question • Motivating participation: How can we get more work done in open content systems? • Idea: match users with tasks they’re likely to be interested in and capable of doing • Requirements: • Introduce tasks matching algorithms/interfaces • Assign users to different conditions • Gather data necessary for evaluation • Survey users

46. Tools Recommender algorithms Interaction design Intelligent Task Routing Goals Get work done Nurture new users Serve community Theory Collective Effort Model Social Influence

47. MovieLens Task: Edit movie content

48. Four strategies to suggest movies to a user theory-based

49. The experiment • Assign ML users to four groups, one per algorithm • About 2,000 subjects, 200 contributors • Count # editors, contributions, fields

50. Rare rated: dominant Needs work: bang for buck Random: not bad here High prediction: lousy