Nonparametric Link Prediction in Dynamic Graphs

1. Nonparametric Link Prediction in Dynamic Graphs PurnamritaSarkar (UC Berkeley) DeepayanChakrabarti (Facebook) Michael Jordan (UC Berkeley)

2. Link Prediction • Who is most likely to be interact with a given node? Should Facebook suggest Alice as a friend for Bob? Alice Bob Friend suggestion in Facebook

3. Link Prediction Alice Should Netflix suggest this movie to Alice? Bob Charlie Movie recommendation in Netflix

4. Link Prediction • Prediction using simple features • degree of a node • number of common neighbors • last time a link appeared • What if the graph is dynamic?

5. Related Work • Generative models • Exp. family random graph models [Hanneke+/’06] • Dynamics in latent space [Sarkar+/’05] • Extension of mixed membership block models [Fu+/10] • Other approaches • Autoregressive models for links [Huang+/09] • Extensions of static features [Tylenda+/09]

6. Goal • Link Prediction • incorporating graph dynamics, • requiring weak modeling assumptions, • allowing fast predictions, • and offering consistency guarantees.

7. Outline • Model • Estimator • Consistency • Scalability • Experiments

8. The Link Prediction Problem in Dynamic Graphs YT+1 (i,j)=? Y1 (i,j)=1 Y2 (i,j)=0 …… G2 G1 YT+1(i,j) | G1,G2, …,GT ~ Bernoulli(gG1,G2,…GT(i,j)) GT+1 Features of previous graphsand this pair of nodes Edge in T+1

9. Including graph-based features • Example set of features for pair (i,j): • cn(i,j) (common neighbors) • ℓℓ(i,j) (last time a link was formed) • deg(j) • Represent dynamics using “datacubes” of these features. • ≈ multi-dimensional histogram on binned feature values ηt = #pairs in Gt with these features 1 ≤ cn ≤ 33 ≤ deg ≤ 61 ≤ ℓℓ ≤ 2 high ηt+/ηt this feature combination is more likely to create a new edge at time t+1 cn deg ηt+ = #pairs in Gt with these features, which had an edge in Gt+1 ℓℓ

10. Including graph-based features 1 ≤ cn(i,j) ≤ 3 3 ≤ deg(i,j) ≤ 6 1 ≤ ℓℓ (i,j) ≤ 2 • How do we form these datacubes? • Vanilla idea: One datacube for Gt→Gt+1aggregated over all pairs (i,j) • Does not allow for differently evolving communities YT+1 (i,j)=? Y2 (i,j)=0 Y1 (i,j)=1 …… G2 G1 GT

11. Our Model 1 ≤ cn(i,j) ≤ 3 3 ≤ deg(i,j) ≤ 6 1 ≤ ℓℓ (i,j) ≤ 2 • How do we form these datacubes? • Our Model: One datacube for each neighborhood • Captures local evolution Y2 (i,j)=0 YT+1 (i,j)=? Y1 (i,j)=1 …… G2 G1 GT

12. Our Model 1 ≤ cn(i,j) ≤ 33 ≤ deg(i,j) ≤ 61 ≤ ℓℓ (i,j) ≤ 2 Neighborhood Nt(i)= nodes within 2 hops Features extracted from (Nt-p,…Nt) Datacube Number of node pairs- with feature s- in the neighborhood of i- at time t Number of node pairs- with feature s- in the neighborhood of i- at time t- which got connected at time t+1

13. Our Model • Datacubedt(i) captures graph evolution • in the local neighborhood of a node • in the recent past • Model: • What is g(.)? g(dt(i), st(i,j)) YT+1(i,j) | G1,G2, …,GT ~ Bernoulli( gG1,G2,…GT(i,j)) Features of the pair Local evolution patterns

14. Outline • Model • Estimator • Consistency • Scalability • Experiments

15. Kernel Estimator for g G1 G2 …… { { { { { { { { { { { { { { { { { { { { { { { { { { query data-cube at T-1 and feature vector at time T compute similarities GT-1 GT-2 … … … GT datacube, feature pair t=3 datacube, feature pair t=2 datacube, feature pair t=1

16. K( , )I{ == } Kernel Estimator for g • Factorize the similarity function • Allows computation of g(.) via simple lookups } } }

17. Kernel Estimator for g G1 G2 …… compute similarities only between data cubes w1 η1 , η1+ w2 η2 , η2+ GT-1 η3 , η3+ w3 datacubes t=3 datacubes t=2 datacubes t=1 GT-2 η4 , η4+ w4 GT

18. K( , )I{ == } Kernel Estimator for g • Factorize the similarity function • Allows computation of g(.) via simple lookups • What is K( , )? } } }

19. Similarity between two datacubes Idea 1 • For each cell s, take(η1+/η1 – η2+/η2)2 and sum • Problem: • Magnitude of η is ignored • 5/10 and 50/100 are treated equally • Consider the distribution η1 , η1+ η2 , η2+

20. Similarity between two datacubes Idea 2 • For each cell s, compute posterior distribution of edge creation prob. • dist = total variation distance between distributions • summed over all cells η1 , η1+ η2 , η2+ 0<b<1 As b0, K( , ) 0 unless dist( , ) =0

21. Kernel Estimator for g Want to show:

22. Outline • Model • Estimator • Consistency • Scalability • Experiments

23. Consistency of Estimator • Lemma 1: As T→∞, for some R>0, • Proof using: As T→∞,

24. Consistency of Estimator • Lemma 2: As T→∞,

25. Consistency of Estimator • Assumption: finite graph • Proof sketch: • Dynamics are Markovian with finite state space • the chain must eventually enter a closed, irreducible communication class • geometric ergodicity if class is aperiodic(if not, more complicated…) • strong mixing with exponential decay • variances decay as o(1/T)

26. Consistency of Estimator • Theorem: • Proof Sketch: • for some R>0 • So

27. Outline • Model • Estimator • Consistency • Scalability • Experiments

28. Scalability • Full solution: • Summing over all n datacubesfor all T timesteps • Infeasible • Approximate solution: • Sum over nearest neighbors of query datacube • How do we find nearest neighbors? • Locality Sensitive Hashing (LSH)[Indyk+/98, Broder+/98]

29. Using LSH • Devise a hashing function for datacubes such that • “Similar” datacubestend to be hashed to the same bucket • “Similar” = small total variation distance between cells of datacubes

30. Using LSH • Step 1: Map datacubes to bit vectors Use B1 buckets to discretize [0,1] Use B2 bits for each bucket Total M*B1*B2 bits, where M = max number of occupied cells << total number of cells For probability mass p the first bits are set to 1

31. Using LSH • Step 1: Map datacubes to bit vectors • Total variation distance L1 distance between distributions Hamming distance between vectors • Step 2: Hash function = k out of MB1B2 bits

32. Fast Search Using LSH 0000 0001 1111111111000000000111111111000 0011 10000101000011100001101010000 . . . . 1011 10101010000011100001101010000 101010101110111111011010111110 1111111111000000000111111111001 1111

33. Outline • Model • Estimator • Consistency • Scalability • Experiments

34. Experiments • Baselines • LL: last link (time of last occurrence of a pair) • CN: rank by number of common neighbors in • AA: more weight to low-degree common neighbors • Katz: accounts forlonger paths • CN-all: apply CN to • AA-all, Katz-all: similar s s

35. Setup • Pick random subset S from nodes with degree>0 in GT+1 • , predict a ranked list of nodes likely to link to s • Report mean AUC (higher is better) G2 G1 GT Test data Training data GT+1

36. Simulations • Social network model of Hoff et al. • Each node has an independently drawn feature vector • Edge(i,j) depends on features of i and j • Seasonality effect • Feature importance varies with season • different communities in each season • Feature vectors evolve smoothly over time • evolving community structures

37. Simulations • NonParamis much better than others in the presence of seasonality • CN, AA, and Katz implicitly assume smooth evolution

38. Sensor Network* * www.select.cs.cmu.edu/data

39. Summary • Link formation is assumed to depend on • the neighborhood’s evolution • over a time window • Admits a kernel-based estimator • Consistency • Scalability via LSH • Works particularly well for • Seasonal effects • differently evolving communities

40. Thanks!

41. Problem statement • We are given {G1, G2,…, Gt}. Want to predict Gt+1 • Model 1: Yt+1(i,j) = f(Yt-p+1(i,j), …, Yt(i,j)) • Takes all edges as independent • Only looks at one feature. • Model2: Gt+1 = f(Gt-p+1, Gt-p+2,…, Gt) • Huge dimensionality • Probably intractable • Middle ground • Learn local prediction model for Yt+1(i,j) using a few features and patch these together to predict the entire graph.

42. Our Model • Idea: Yt+1(i,j) depends on features of (i,j) and the neighborhood of i in the ‘’p’’ previous graphs. Features specific to (i,j) in t {deg(i), deg(j), cn(i,j), ℓℓ(i,j)} Features of the neighborhood of i Should be amenable to fast algorithms. Should reflect the evolution of the graph. But should also be similar to the features of (i,j).

43. Estimation • Kernel Estimator of g } Once you have computed the kernel similarities between two datacubes, everything boils down to table lookups.

44. Distance between two datacubes • Can just compare rates of link formation, i.e. η+/η, but this does not take into account the variance. • Instead, make a normal approximation to η+/η and look at the total variation distance. As b0, K(dt(i), dt’(i’)) 0 unless D(K(dt(i), dt’(i’)) =0

45. Distance between two datacubes • Can just compare rates of link formation, i.e. η+/η, but this does not take into account the variance. • Instead, make a normal approximation to η+/η and look at the total variation distance. As b0, K(dt(i), dt’(i’)) 0 unless D(K(dt(i), dt’(i’)) =0

46. Consistency of Estimator • Define • Kind of behaves like a bias term.

47. Consistency of Estimator • Show • Assumption 1. b0 as nT∞ [similar to kernel density estimation] • Show that for bounded q, • Assumption 2. Introduce strong mixing coefficient α(k), roughly this bounds the degree of dependence between two neighborhoods at distance k. • The total covariance between all neighborhoods is bounded. • Assume

48. Including graph-based features 1 ≤ cn(i,j) ≤ 3 3 ≤ deg(i,j) ≤ 6 1 ≤ ℓℓ (i,j) ≤ 2 YT+1 (i,j)=? Y2 (i,j)=0 Y1 (i,j)=1 …… G2 G1 Too local, not to mention expensive Too global. Idea2: Make one datacube for each pair of nodes. Idea1: Make one datacube per (Gt ,Gt+1 ) transition. Learn how successful this feature combination has been in generating links over the past. GT

49. Our Model 1 ≤ cn(i,j) ≤ 3 3 ≤ deg(i,j) ≤ 6 1 ≤ ℓℓ (i,j) ≤ 2 sT (i,j) • Datacubedt(i)captures the evolution of a small (2-hop) neighborhood around node i • Close nodes will have overlapping neighborhoods  similar datacubes. Y2 (i,j)=0 Y1 (i,j)=1 YT+1 (i,j)=? …… G2 G1 GT {dT-1(i) ,sT (i,j)}

50. Building neighborhood features • Let S=range of s(i,j). Assume S is finite. Datacube Number of pairs with feature s in the neighborhood of i at time t Number of pairs which got connected at time t+1 out of ηit (s) • Captures the evolution of the neighborhood from tt+1 • We use the past evolution pattern of a neighborhood in predicting future evolution. • But how do we estimate g efficiently? We will show that the inference of g will boil down to table lookups in the datacubesdt(i)