1 / 118

Data Mining using Fractals and Power laws

Data Mining using Fractals and Power laws. Christos Faloutsos Carnegie Mellon University. THANK YOU!. Tao Li Martha Soledad. Thanks to. Deepayan Chakrabarti (CMU/Yahoo) Michalis Faloutsos (UCR) George Siganos (UCR). Overview. Goals/ motivation: find patterns in large datasets:

aric
Download Presentation

Data Mining using Fractals and Power laws

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Data Mining using Fractals and Power laws Christos Faloutsos Carnegie Mellon University

  2. THANK YOU! • Tao Li • Martha Soledad C. Faloutsos

  3. Thanks to • Deepayan Chakrabarti (CMU/Yahoo) • Michalis Faloutsos (UCR) • George Siganos (UCR) C. Faloutsos

  4. Overview • Goals/ motivation: find patterns in large datasets: • (A) Sensor data • (B) network/graph data • Solutions: self-similarity and power laws • Discussion C. Faloutsos

  5. Applications of sensors/streams • ‘Smart house’: monitoring temperature, humidity etc • Financial, sales, economic series C. Faloutsos

  6. Applications of sensors/streams • ‘Smart house’: monitoring temperature, humidity etc • Financial, sales, economic series C. Faloutsos

  7. Motivation - Applications • Medical: ECGs +; blood pressure etc monitoring • Scientific data: seismological; astronomical; environment / anti-pollution; meteorological C. Faloutsos

  8. # cars Automobile traffic 2000 1800 1600 1400 1200 1000 800 600 400 200 0 time Motivation - Applications (cont’d) • civil/automobile infrastructure • bridge vibrations [Oppenheim+02] • road conditions / traffic monitoring C. Faloutsos

  9. Motivation - Applications (cont’d) • Computer systems • web servers (buffering, prefetching) • network traffic monitoring • ... http://repository.cs.vt.edu/lbl-conn-7.tar.Z C. Faloutsos

  10. Web traffic • [Crovella Bestavros, SIGMETRICS’96] C. Faloutsos

  11. survivable,self-managing storageinfrastructure a storage brick(0.5–5 TB) ~1 PB . . . . . . Self-* Storage (Ganger+) • “self-*” = self-managing, self-tuning, self-healing, … • Goal: 1 petabyte (PB) for CMU researchers • www.pdl.cmu.edu/SelfStar C. Faloutsos

  12. survivable,self-managing storageinfrastructure a storage brick(0.5–5 TB) ~1 PB . . . . . . Self-* Storage (Ganger+) • “self-*” = self-managing, self-tuning, self-healing, … C. Faloutsos

  13. Problem definition • Given: one or more sequences x1 , x2 , … , xt , …; (y1, y2, … , yt, …) • Find • patterns; clusters; outliers; forecasts; C. Faloutsos

  14. Problem #1 # bytes • Find patterns, in large datasets time C. Faloutsos

  15. Problem #1 # bytes • Find patterns, in large datasets time Poisson indep., ident. distr C. Faloutsos

  16. Problem #1 # bytes • Find patterns, in large datasets time Poisson indep., ident. distr C. Faloutsos

  17. Problem #1 # bytes • Find patterns, in large datasets time Poisson indep., ident. distr Q: Then, how to generate such bursty traffic? C. Faloutsos

  18. Overview • Goals/ motivation: find patterns in large datasets: • (A) Sensor data • (B) network/graph data • Solutions: self-similarity and power laws • Discussion C. Faloutsos

  19. Problem #2 - network and graph mining • How does the Internet look like? • How does the web look like? • What constitutes a ‘normal’ social network? • What is the ‘network value’ of a customer? • which gene/species affects the others the most? C. Faloutsos

  20. Network and graph mining Food Web [Martinez ’91] Protein Interactions [genomebiology.com] Friendship Network [Moody ’01] Graphs are everywhere! C. Faloutsos

  21. Problem#2 Given a graph: • which node to market-to / defend / immunize first? • Are there un-natural sub-graphs? (eg., criminals’ rings)? [from Lumeta: ISPs 6/1999] C. Faloutsos

  22. Solutions • New tools: power laws, self-similarity and ‘fractals’ work, where traditional assumptions fail • Let’s see the details: C. Faloutsos

  23. Overview • Goals/ motivation: find patterns in large datasets: • (A) Sensor data • (B) network/graph data • Solutions: self-similarity and power laws • Discussion C. Faloutsos

  24. What is a fractal? = self-similar point set, e.g., Sierpinski triangle: zero area: (3/4)^inf infinite length! (4/3)^inf ... Q: What is its dimensionality?? C. Faloutsos

  25. What is a fractal? = self-similar point set, e.g., Sierpinski triangle: zero area: (3/4)^inf infinite length! (4/3)^inf ... Q: What is its dimensionality?? A: log3 / log2 = 1.58 (!?!) C. Faloutsos

  26. Q: fractal dimension of a line? Q: fd of a plane? Intrinsic (‘fractal’) dimension C. Faloutsos

  27. Q: fractal dimension of a line? A: nn ( <= r ) ~ r^1 (‘power law’: y=x^a) Q: fd of a plane? A: nn ( <= r ) ~ r^2 fd== slope of (log(nn) vs.. log(r) ) Intrinsic (‘fractal’) dimension C. Faloutsos

  28. log(#pairs within <=r ) 1.58 log( r ) Sierpinsky triangle == ‘correlation integral’ = CDF of pairwise distances C. Faloutsos

  29. log(#pairs within <=r ) 1.58 log( r ) Observations: Fractals <-> power laws Closely related: • fractals <=> • self-similarity <=> • scale-free <=> • power laws ( y= xa ; F=K r-2) • (vs y=e-ax or y=xa+b) C. Faloutsos

  30. Outline • Problems • Self-similarity and power laws • Solutions to posed problems • Discussion C. Faloutsos

  31. #bytes time Solution #1: traffic • disk traces: self-similar: (also: [Leland+94]) • How to generate such traffic? C. Faloutsos

  32. 20% 80% Solution #1: traffic • disk traces (80-20 ‘law’) – ‘multifractals’ #bytes time C. Faloutsos

  33. 80-20 / multifractals 20 80 C. Faloutsos

  34. 80-20 / multifractals 20 80 • p ; (1-p) in general • yes, there are dependencies C. Faloutsos

  35. More on 80/20: PQRS • Part of ‘self-* storage’ project time cylinder# C. Faloutsos

  36. p q r s More on 80/20: PQRS • Part of ‘self-* storage’ project q r s C. Faloutsos

  37. Overview • Goals/ motivation: find patterns in large datasets: • (A) Sensor data • (B) network/graph data • Solutions: self-similarity and power laws • sensor/traffic data • network/graph data • Discussion C. Faloutsos

  38. Problem #2 - topology How does the Internet look like? Any rules? C. Faloutsos

  39. Patterns? • avg degree is, say 3.3 • pick a node at random – guess its degree, exactly (-> “mode”) count ? avg: 3.3 degree C. Faloutsos

  40. Patterns? • avg degree is, say 3.3 • pick a node at random – guess its degree, exactly (-> “mode”) • A: 1!! count avg: 3.3 degree C. Faloutsos

  41. Patterns? • avg degree is, say 3.3 • pick a node at random - what is the degree you expect it to have? • A: 1!! • A’: very skewed distr. • Corollary: the mean is meaningless! • (and std -> infinity (!)) count avg: 3.3 degree C. Faloutsos

  42. -0.82 att.com log(degree) ibm.com log(rank) Solution#2: Rank exponent R • A1: Power law in the degree distribution [SIGCOMM99] internet domains C. Faloutsos

  43. Solution#2’: Eigen Exponent E Eigenvalue • A2: power law in the eigenvalues of the adjacency matrix Exponent = slope E = -0.48 May 2001 Rank of decreasing eigenvalue C. Faloutsos

  44. Power laws - discussion • do they hold, over time? • do they hold on other graphs/domains? C. Faloutsos

  45. Power laws - discussion • do they hold, over time? • Yes! for multiple years [Siganos+] • do they hold on other graphs/domains? • Yes! • web sites and links [Tomkins+], [Barabasi+] • peer-to-peer graphs (gnutella-style) • who-trusts-whom (epinions.com) C. Faloutsos

  46. att.com -0.82 log(degree) ibm.com log(rank) Time Evolution: rank R Domain level • The rank exponent has not changed! [Siganos+] C. Faloutsos

  47. The Peer-to-Peer Topology count [Jovanovic+] • Number of immediate peers (= degree), follows a power-law degree C. Faloutsos

  48. epinions.com count • who-trusts-whom [Richardson + Domingos, KDD 2001] (out) degree C. Faloutsos

  49. Why care about these patterns? • better graph generators [BRITE, INET] • for simulations • extrapolations • ‘abnormal’ graph and subgraph detection C. Faloutsos

  50. Recent discoveries [KDD’05] • How do graphs evolve? • degree-exponent seems constant - anything else? C. Faloutsos

More Related