Expertise Networks in Online Communities: Structure and Algorithms

1. Expertise Networks in Online Communities:Structure and Algorithms Lada Adamic joint work with Jun Zhang and Mark Ackerman School of Information, University of Michigan NetSci May 24th, 2007

2. Knows Knowledge iN Have you sought knowledge here?

3. Knowledge In Oozing out knowledge Largest search engine in Korea - 70% of search (Google: 2%) Comprehensive portal – integrated news, blogs, ‘knowledge search’ ``Knowledge search is like oozing out knowledge in human brains to the Internet. People who know something better than others can present their know-how, skills or knowledge'' NHN CEO Chae Hwi-young Knowledge-In had 60 million questions and answers as of Feb 2007 popular: why fingernails grow faster than toenails how fast a fly can fly why seagulls sit in the same direction

4. Knowledge In Ranking the contributors

5. Culture of generosity “(It is) the next generation of search… (it) is a kind of collective brain -- a searchable database of everything everyone knows. It's a culture of generosity. The fundamental belief is that everyone knows something.” -- Eckart Walther (Yahoo Research) 90 million users worldwide

6. Limitations of Current Systems • The Response Time Gap • The Expertise Gap • Difficult to infer reliability of answers Automatically ranking expertise may be helpful.

7. Related work • Analysis of online communities • NetScan (Smith, Fisher, et al. at Microsoft) • Social network analysis (LiveJournal, blog communities) • Motivations of online participation (Lakhani & Hippel) • Graph-based ranking algorithms • PageRank, HITS, etc. • Expertise sharing studies • Expertise recommenders • ContactFinder (Krulwich et al.), Answer Garden (Ackerman) • Small Blue (Lin) • Automatic evaluating expertise levels • Using different text resources (Kautz, et al, and a lot of others) • Using email networks (Campbell et al.)

8. Overview • Social network analysis • Constructing Expertise Networks • Finding meaningful metrics • Empirical evaluation of ranking algorithms • Human Rating vs. Algorithmic Ranking • Simulation • Understanding underlying dynamics • Predicting performance of ranking algorithms in yet-unobserved community dynamics

9. Java Forum • 87 sub-forums • 1,438,053 messages • community expertise network constructed: • 196,191 users • 796,270 edges

10. 0.9 1 1 A A A A A B B B B B C C C C C Constructing a community expertise network unweighted 1 weighted by # threads 2 Thread 1 Thread 2 1/2 weighted by shared credit 1+1//2 Thread 1: Large Data, binary search or hashtable? user ARe: Large...user BRe: Large...user C Thread 2: Binary file with ASCII data user ARe: File with...user C weighted with backflow 0.1

11. Not Everyone Asks/Replies The Web is a bow tie The JavaForum network is an uneven bow tie • Core: A strongly connected component, in which everyone asks and answers • IN: Mostly askers. • OUT: Mostly Helpers

12. 0 10 2 a = 1.87 fit, R = 0.9730 -1 10 -2 10 cumulative probability number of people one received replies from -3 10 -4 10 0 1 2 3 10 10 10 10 degree (k) Uneven participation • ‘answer people’ may reply to thousands of others • ‘question people’ are also uneven in the number of repliers to their posts, but to a lesser extent number of people one replied to

13. Who Answers Whom Degree-degree correlations between asker and helper

14. Summary of JavaForum Network • Different types of participation • Askers, ask-help-er, helpers • Different levels of participation • top helpers, others • Who replied to whom • Top repliers answer questions for everyone • Other helpers help those with somewhat lower expertise

15. Relating network structure to Java expertise • Human-rated expertise levels • 2 raters • 135 JavaForum users with >= 10 posts • inter-rater agreement (t = 0.74, r = 0.83) • for evaluation of algorithms, omit users where raters disagreed by more than 1 level (t = 0.80, r = 0.83)

16. Structural Info Based Expertise Ranking Metrics • # replies posted (# answers) • experts can answer many questions • # people replied to (# indegree) • experts can answer questions from many different people • z-score for the 2 above (observed – m)/s • experts are above the mean in the above two metrics • PageRank replying to people who reply to people • higher level experts can answer mid-level experts • HITS experts answer questions by people whose questions other experts have answered hubs point to good authorities

17. automated vs. human ratings # answers indegree automated ranking z # answers z indegree PageRank HITS authority human rating

18. 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 # answers z-score # answers indegree z-score indegree PageRank HITS authority Kendall’s t Spearman’s r Top K Algorithm Rankings vs. Human Ratings simple local measures do as well (and better) than measures incorporating the wider network topology

19. Modeling community structure to explain algorithm performance Control Parameters: • Distribution of expertise • Who asks questions most often? • Who answers questions most often? • best expert most likely • someone a bit more expert ExpertiseNet Simulator

20. Simulating probability of expertise pairing suppose: expertise is uniformly distributed probability of posing a question is inversely proportional to expertise pij= probability a user with expertise j replies to a user with expertise i 2 models: ‘best’ preferred ‘just better’ preferred j>i

21. Visualization Best “preferred” just better

22. Degree correlation profiles asker indegree Java Forum Network asker indegree asker indegree best preferred (simulation) just better (simulation)

23. The Simulation of JavaForum • Settings: • Distribution of expertise (skewed) • Who asks questions most often? (novices) • Who answers questions? (best expert most likely) • Results • Similar bow tie structure • Similar degree distribution • Slightly different correlation profiles • Similar algorithm performance • PageRank does not outperform simpler degree-based metrics

24. Different ranking algorithms perform differently In the ‘just better’ model, a node is correctly ranked by PageRank but not by HITS

25. It can tell us when to use which algorithms Preferred Helper: ‘best available’ Preferred Helper: ‘just better’

26. Summary • Expertise Networks have interesting characteristics • A set of useful metrics • Ranking algorithms are affected by network structures • Simulation as an analysis tool • There are rich design opportunities • Find experts with the help of structural information (and content analysis) • Predict good answers • Re-order questions/answers to match expertise working paper: “Expertise-Level based Interface Personalization for Online Help-seeking Communities”

27. Future Work • Looking at diverse sets of question-answer forums (Yahoo Answers) • Expertise across different topics • Using explicit ratings for evaluation of automated expertise identification & incorporation into algorithms (battling spam) • Users’ expertise change over time • Continually developing and evaluating our systems built upon these findings beauty & style cars & transportation hair maintenance & repairs

28. for more info • ExpertiseRank algorithms and evaluations Zhang, J., Ackerman, M.S., Adamic, L., Expertise Networks in Online Communities: Structure and Algorithms, WWW’07 • Simulations of expertise networks Zhang, J., Ackerman, M.S., Adamic, L., CommunityNetSimulator: Using Simulations to Study Online Community Network Formation and Implications, C&T2007 Jun Zhang junzh@umich.edu http://www-personal.si.umich.edu/~junzh Mark Ackerman ackerm@eecs.umich.edu http://www.eecs.umich.edu/~ackerm/ Lada Adamic ladamic@umich.edu http://www-personal.umich.edu/~ladamic NSF (IRI-9702904)

29. ads • Jun Zhang is graduating and on the job market (junzh@umich.edu) • Lada is looking for a postdoc (ladamic@umich.edu)

30. Simplest models do not capture all ‘local’ interactions