Statistical properties of network community structure

1. Jure Leskovec, CMU Kevin Lang, Anirban Dasgupta and Michael Mahoney Yahoo! Research Statistical properties of network community structure

2. Network communities • Communities: • Sets of nodes with lots of connections inside and few to outside (the rest of the network) • Assumption: • Networks are (hierarchically) composed of communities (modules) Our question: Are large networks really like this? Communities, clusters, groups, modules

3. Community score (quality) S • How community like is a set of nodes? • Need a natural intuitive measure • Conductance (normalized cut) S’ • Φ(S) = # edges cut / # edges inside • SmallΦ(S) corresponds to more community-like sets of nodes

4. Community score (quality) What is “best” community of 5 nodes? • Score: Φ(S) = # edges cut / # edges inside

5. Community score (quality) What is “best” community of 5 nodes? Bad community Φ=5/6 = 0.83 • Score: Φ(S) = # edges cut / # edges inside

6. Community score (quality) What is “best” community of 5 nodes? Bad community Φ=5/7 = 0.7 Better community Φ=2/5 = 0.4 • Score: Φ(S) = # edges cut / # edges inside

7. Community score (quality) What is “best” community of 5 nodes? Bad community Φ=5/7 = 0.7 Best community Φ=2/8 = 0.25 Better community Φ=2/5 = 0.4 • Score: Φ(S) = # edges cut / # edges inside

8. Network Community Profile Plot • We define: Network community profile (NCP) plot Plot the score of best community of size k • Search over all subsets of size k and find best: Φ(k=5) = 0.25 • NCP plot is intractable to compute • Use approximation algorithm

9. NCP plot: Small Social Network • Dolphin social network • Two communities of dolphins NCP plot Network

10. NCP plot: Zachary’s karate club • Zachary’s university karate club social network • During the study club split into 2 • The split (squares vs. circles) corresponds to cut B Network NCP plot

11. NCP plot: Network Science • Collaborations between scientists in Networks Network NCP plot

12. Geometric and Hierarchical graphs Geometric (grid-like) network – Small social networks – Geometric and – Hierarchical network have downward NCP plot Hierarchical network

13. Our work: Large networks • Previously researchers examined community structure of smallnetworks (~100 nodes) • We examined more than 70 different large social and information networks Large real-world networks look very different!

14. Example of our findings • Typical example: General relativity collaboration network (4,158 nodes, 13,422 edges)

15. NCP: LiveJournal (N=5M, E=42M) Better and better communities Best communities get worse and worse Community score Best community has 100 nodes Community size

16. Explanation: Downward part • Whiskersare responsible for downward slope of NCP plot NCP plot Largest whisker Whiskeris a set of nodes connected to the network by a single edge

17. Explanation: Upward part • Each new edge inside the community costs more Φ=1/3 = 0.33 Φ=2/4 = 0.5 NCP plot Φ=8/6 = 1.3 Φ=64/14 = 4.5 Each node has twice as many children

18. Suggested network structure Denser and denser core of the network Core contains 60% node and 80% edges Whiskers are responsible for good communities Network structure: Core-periphery (jellyfish, octopus)

19. Caveat: Bag of whiskers What if we allow cuts that give disconnected communities? • Cut all whiskers • Compose communities out of whiskers • How good “community” do we get?

20. Communities made of whiskers We get better community scores when composing disconnected sets of whiskers Community score Connected communities Bag of whiskers Community size

21. Comparison to rewire network • Take a real network G • Rewire edges for a long time • We obtain a random graph with same degree distribution as the real network G

22. Comparison to a rewired network Rewired network: random network with same degree distribution

23. What is a good model? • What is a good model that explains such network structure? • None of the existing models work Flat and Down Flat Down and Flat Geometric Pref. Attachment Small World Pref. attachment

24. Forest Fire model works • Forest Fire: connections spread like a fire • New node joins the network • Selects a seed node • Connects to some of its neighbors • Continue recursively As community grows it blends into the core of the network

25. Forest Fire NCP plot rewired network Bag of whiskers

26. Conclusion and connections • Whiskers: • Largest whisker has ~100 nodes • Independent of network size • Dunbar number: a person can maintain social relationship to at most 150 people • Bond vs. identity communities • Core: • Core has little structure (hard to cut) • Still more structure than the random network

27. Conclusion and connections • NCP plot is a way to analyze network community structure • Our results agree with previous work on small networks (that are commonly used for testing community finding algorithms) • Butlarge networks are different • Large networks • Whiskers + Core structure • Small well isolated communities blend into the core of the networks as they grow