1 / 64

Algorithm Design Meets Big Data

Bahman Bahmani bahman@stanford.edu. Algorithm Design Meets Big Data. Outline. Fundamental Tradeoffs Drug Interaction Example [Adapted from Ullman’s slides, 2012] Technique I: Grouping Similarity Search [Bahmani et al., 2012] Technique II: Partitioning

ghalib
Download Presentation

Algorithm Design Meets Big Data

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Bahman Bahmani bahman@stanford.edu Algorithm Design Meets Big Data

  2. Outline • Fundamental Tradeoffs • Drug Interaction Example [Adapted from Ullman’s slides, 2012] • Technique I: Grouping • Similarity Search [Bahmani et al., 2012] • Technique II: Partitioning • Triangle Counting [Afrati et al., 2013; Suri et al., 2011] • Technique III: Filtering • Community Detection [Bahmani et al., 2012] • Conclusion

  3. Drug interaction problem • 3000 drugs, 1M patients, 20 years • 1MB of data per drug • Goal: Find drug interactions

  4. MapReduce algorithm • Map • Input: <i; Ri> • Output: <{i,j}; Ri> for any other drug j • Reduce • Input: <{i,j}; [Ri,Rj]> • Output: Interaction between drugs i,j

  5. Example Mapper for drug 1 Reducer for {1,2} Drug 3 data Drug 2 data Drug 3 data Drug 1 data Drug 1 data {2, 3} {1, 2} {1, 3} {1, 2} {1, 3} {2, 3} Drug 2 data Mapper for drug 2 Reducer for {1,3} Mapper for drug 3 Reducer for {2,3}

  6. Example Reducer for {1,2} {1, 2} Drug 1 data Drug 2 data Reducer for {1,3} Drug 1 data Drug 3 data {1, 3} Reducer for {2,3} Drug 2 data Drug 3 data {2, 3}

  7. Total work 4.5M pairs × 100 msec/pair = 120 hours Less than 1 hour using ten 16-core nodes

  8. All good, right? Network communication = 3000 drugs × 2999 pairs/drug × 1MB/pair = 9TB Over 20 hours worth of network traffic on a 1Gbps Ethernet

  9. Improved algorithm • Group drugs • Example: 30 groups of size 100 each • G(i) = group of drug i

  10. Improved algorithm • Map • Input: <i; Ri> • Output: <{G(i), G’}; (i, Ri)> for any other group G’ • Reduce • Input: <{G,G’}; 200 drug records in groups G,G’> • Output: All pairwise interactions between G,G’

  11. Total work • Same as before • Each pair compared once

  12. All good now? 3000 drugs × 29 replications × 1MB = 87GB Less than 15 minutes on 1Gbps Ethernet

  13. Algorithm’s tradeoff • Assume ngroups • #key-value pairs emitted by map= 3000×(n-1) • #input records per reducer = 2×3000/n The more parallelism, the more communication

  14. Reducer size • Maximum number of inputs a reducer can have, denoted as λ

  15. Analyzing drug interaction problem • Assume ddrugs • Each drug needs to go to (d-1)/(λ-1) reducers • needs to meet d-1 other drugs • meets λ-1 other drugs at each reducer • Minimum communication = d(d-1)/(λ-1) ≈ d2/λ

  16. How well our algorithm trades off? • With ngroups • Reducer size = λ= 2d/n • Communication = d×(n-1) ≈ d × n = d × 2d/λ = 2d2/λ • Tradeoff within a factor 2 of ANY algorithm

  17. Fundamental MapReduce tradeoff • Increase in reducer size (λ) • Increases reducer time, space complexities • Drug interaction: time ~ λ2, space ~ λ • Decreases total communication • Drug interaction: communication ~ 1/λ

  18. Technique I: Grouping • Decrease key resolution • Lower communication at the cost of parallelism • How to group may not always be trivial

  19. Similarity Search • Near-duplicate detection • Document topic classification • Collaborative filtering • Similar images • Scene completion

  20. Many-to-Many Similarity Search • N data objects and M query objects • Goal: For each query object, find the most similar data object

  21. Candidate generation • M=107, N=106 • 1013pairs to check • 1μsec per pair • 116 days worth of computation

  22. Locality Sensitive Hashing: Big idea • Hash functions likely to map • Similar objects to same bucket • Dissimilar objects to different buckets

  23. … 9 8 7 … 3 2 1 6 0 …

  24. MapReduce implementation • Map • For data point p emit <Bucket(p) ; p> • For query point q • Generate offsets qi • Emit <Bucket(qi); q> for each offset qi

  25. MapReduce implementation • Reduce • Input: <v; p1,…,pt,q1,…,qs> • Output: {(pi,qj)| 1≤i≤t, 1≤j≤s}

  26. All good, right? • Too many offsets required for good accuracy • Too much network communication

  27. … 9 8 7 … 3 2 1 6 0 …

  28. MapReduce implementation • G another LSH • Map • For data point p emit <G(Bucket(p)); (Bucket(p),p)> • For query point q • Generate offsets qi • Emit <G(Bucket(qi)); q> for all distinct keys

  29. MapReduce implementation • Reduce • Input: <v; [(Bucket(p1),p1), …, (Bucket(pt),pt),q1,…,qs]> • Index pi at Bucket(pi) (1≤i≤t) • Re-compute all offsets of qj and their buckets (1≤j≤s) • Output: candidate pairs

  30. Experiments Shuffle Size Runtime

  31. Technique II: Partitioning • Divide and conquer • Avoid “Curse of the Last Reducer”

  32. Graph Clustering Coefficient • G = (V,E) undirected graph • CC(v) = Fraction of v’s neighbor pairs which are neighbors themselves • Δ(v) = Number of triangles incident on v

  33. Graph Clustering Coefficient • Spamming activity in large-scale web graphs • Content quality in social networks • …

  34. MapReduce algorithm • Map • Input: <v; [u1,…,ud]> • Output: • <{ui,uj}; v> (1≤i<j≤d) • <{v,ui}; $> (1≤i≤d) v

  35. MapReduce algorithm • Reduce • Input: <{u,w}; [v1,…,vT, $?]> • Output: If $ part of input, emit <vi; 1/3> (1≤i≤T) u … v1 v2 vT w

  36. All good, right? • For each node v, Map emits all neighbor pairs • ~ dv2/2 key-value pairs • If dv=50M, even outputting 100M pairs/second, takes ~20 weeks!

  37. Experiment: Reducer completion times

  38. Partitioning algorithm • Partition nodes: p partitions V1,…,Vp • ρ(v) = partition of node v • Solve the problem independently on each subgraphViUVjUVk (1≤i≤j≤k≤p) • Add up results from different subgraphs

  39. Partitioning algorithm 3

  40. Partitioning algorithm • Map • Input: <v; [u1,…,ud]> • Output: for each ut (1≤t≤d) emit <{ρ(v), ρ(ut),k}; {v,ut}> for any 1≤k≤p

  41. Partitioning algorithm • Reduce • Input: <{i,j,k}; [(u1,v1), …, (ur,vr)]> (1≤i≤j≤k≤p) • Output: For each node x in the input emit (x; Δijk(x)) Final results computed as: Δ(x) = Sum of all Δijk(x) (1≤i≤j≤k≤p)

  42. Example • 1B edges, p=100 partition • Communication • Each edge sent to 100 reducers • Total communication = 100B edges • Parallelism • #subgraphs ~ 170,000 • Size of each subgraph < 600K

  43. Experiment: Reducer completion times

  44. Experiment: Reducer size vs shuffle size

  45. Experiment: Total runtime

  46. MapReduce tradeoffs Approximation Communication Parallelism

  47. Technique III: Filtering • Quickly decrease the size of the data in a distributed fashion… • … while maintaining the important features of the data • Solve the small instance on a single machine

  48. Densest subgraph • G=(V,E) undirected graph • Find a subset S of V with highest density: ρ(S) = |E(S)|/|S| |S|=6, |E(S)|=11

  49. Densest subgraph • Functional modules in protein-protein interaction networks • Communities in social networks • Web spam detection • …

  50. Exact solution • Linear Programming • Hard to scale

More Related