1 / 45

Applications of Sketch Based Techniques to Data Mining Problems

Applications of Sketch Based Techniques to Data Mining Problems. Nikos Koudas AT&T Labs Research joint work with: G. Cormonde, P. Indyk, S. Muthukrishnan. Taming Massive Data Sets. Requirements of data mining algorithms operate on very large data sets scalability incremental

sheri
Download Presentation

Applications of Sketch Based Techniques to Data Mining Problems

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Applications of Sketch Based Techniques to Data Mining Problems Nikos Koudas AT&T Labs Research joint work with: G. Cormonde, P. Indyk, S. Muthukrishnan DIMACS Summer School

  2. Taming Massive Data Sets • Requirements of data mining algorithms • operate on very large data sets • scalability • incremental • Most data mining algorithms have super-linear complexity • Deploying mining algorithms on very large data sets, most likely will result in terrible performance DIMACS Summer School

  3. Sketching • Reduce the dimensionality of the data in a systematic way, constructing short “data summaries” • Effectively reduce data volume. • Deploy mining algorithms on the reduced data volume. • Main issue: Preserve data properties, the mining algorithm is concerned with, (e.g., distances) in the reduced data volume. DIMACS Summer School

  4. Applications • Clustering time series data • Clustering tabular data sets DIMACS Summer School

  5. Introduction • Time series data abound in many applications. • Financial, performance data, geographical and meteorological information, solar and space data etc. • Various works deal with management and analysis aspects of time series data: • Indexing, storage and retrieval • Analysis and mining (forecasting, outlier and deviation detection, etc) • Active research area in many research communities. DIMACS Summer School

  6. Representative Trends Relaxed period Average trend DIMACS Summer School

  7. Usage Examples • Mining/Prediction • Identifying ‘periodic trends’ • Uncovering unexpected ‘periodic trends’ • Performance management • Networking (routing, traffic engineering, bandwidth allocation) • System Tuning • Financial databases • Cyclic behavior DIMACS Summer School

  8. Definitions • Given a time series V and an integer T define V(T) = {(V[iT+1], V[iT+2], …, V[iT+T])}, 0 <= i<= n/T-1 • Define: • Ci(V(T)) = S1<j<n/T-1 D(vi,uj) • Thus: • Relaxed Period: min C0(V(T)), for T in [l,u] • Average Trend: min Ci(V(T)), for T in [l,u] DIMACS Summer School

  9. Definitions n points in each time series n/T vectors for each T, T in [l,u] Relaxed period T n/T vectors for each T, each of them a candidate avg. trend, T in [l,u] Average Trend T DIMACS Summer School

  10. Algorithms • There exists a quadratic algorithm for identifying relaxed periods. • There exists a cubic algorithm for identifying average trends. • Simply evaluate the clustering for each T in [l,u], it takes linear time to evaluate relaxed periods for each T and quadratic time to evaluate for average trends. DIMACS Summer School

  11. Algorithms • But can we really run these? • Consider length of sessions in an AT&T service for each second for a year, it is more than 31M values and approximately 256MB. • Consider running the previous algorithms for say 10 years or on a finer time scale. • Both brute force algorithms are impractical. • Can we run faster on large datasets, too large to be in memory? DIMACS Summer School

  12. Our Approach • Identify representative trends faster but provide approximate answers: • General approach (expresses various notions of representative trends) • Provides guaranteed approximation performance, with high probability • We present our approach in the following steps: • Define the sketch of a vector • Algorithms for finding the sketch of all sub-vectors of width T • Determine the sketch of all sub-vectors of width in a given range. DIMACS Summer School

  13. Sketch of a vector • Given a vector t of length l, we generate its sketch S(t) as follows: • Pick a random vector u of length l, by picking each component u[i] from a normal distribution N(0,1) (normalized to 1). • Define: • S(t)[i] = t.u = Sjt[j].u[j] DIMACS Summer School

  14. Sketch Properties • Theorem • For any given set L of vectors of length l, for a fixed e < 1/2, if k = 9 log|L|/e2, then for any pair of vectors u,w we have • (1-e)||u-w||2 < ||S(u)-S(w)||2<(1+e)||u-w||2 with probability 1/2. • By increasing k we can increase the probability of success • This is the Johnson-Lindenstrauss (JL) lemma. DIMACS Summer School

  15. Fixed window sketches • Compute all sketches of sub-vectors of length l in a sequence of length n. • There are n-l+1 such sub-vectors. • Straightforward application of JL would require O(nlk) time since there are O(n) sub-vectors, each of length l and the sketch is of size k, not practical. l DIMACS Summer School k

  16. Key Observation • We can compute ALL such sketches fast by using the fast fourier transform. • The problem of computing sketches of all sub-vectors of length l simultaneously is exactly the problem of computing the convolution of two vectors t and u • Given two vectors A[1…a] and B[1…b] their convolution is C[1…a+b] where • C[k] = S1<i<bA[k-i]B[i] for 2 <= k <= a+b DIMACS Summer School

  17. Example (-0.97,-0.2) = (-0.4,-2.14,-1.57,-3.11,-0.97) S1[0] S2[0] S3[0] 2 1 3 1 Convolution with (0.11,0.99) = (1.98,1.21,3.08,1.32,0.11) S1[1] S2[1] S3[1] DIMACS Summer School

  18. Computing all sketches of width in a given range • Compute all sketches of all sub-vectors of length between l and u. • Brute force is cubic and prohibitive • Applying our observation would be quadratic and still prohibitive • Can we compute all sketches of width in a given range faster? DIMACS Summer School

  19. Approach • We will construct a pool of sketches that we will pre-compute and store. Following this preprocessing we will be able to determine the sketch of any sub-vector in O(1) fairly accurately. • Pick an l <= L <= u and construct all sketches of length L as before using convolutions, this is O(nlog2n) in the worst case. Assume for now that L a power of 2 actually construct two such pools S1 and S2 DIMACS Summer School

  20. Approach • Consider any vector t[i,….i+j-1] we have two cases: • j = some power of 2 (L), in this case we have it in the pool and we can look it up in O(1) • 2r < j < 2r+1 in this case we can compute the sketch as follows: • S(t[i,….,i+j-1])[j] = S1(t[i,…,i+2r-1])[j] + S2[t[i+j-2r,….,i+j-1])[j] both terms belong to the pool DIMACS Summer School

  21. Example S(U) = S10 + S21 U 2 1 3 1 2 3 2 1 S1 S2 DIMACS Summer School

  22. Why is this enough? • Theorem • For any given set L of vector of length l, for fixed e < 1/2 if k = 9 log L/e2, then for any pair of vectors u,w in L • (1-e)||u-w||2 <= ||S(u)-S(w)||2 <= 2 (1+e) ||u-w||2 with probability 1/2 DIMACS Summer School

  23. Putting it all together • Given V and [l,u] range • relaxed period: • Compute sketches in time O(nlog(u-l)klogu) • Consider every T in [l,u] and compute C0(V(T)) for every T. • Choosing k as described will guarantee that we are at most 2+e away from the true relaxed period • Average Trends • Proceed similarly by evaluating Ci(V(T)) for every i DIMACS Summer School

  24. Implementation Issues • Computing Sketches • The pool of sketches can be computed with a single pass over the data set. We only need to keep a window worth of data across successive sketch computations. • Retrieving Sketches • Required sketches are retrieved by performing random IO. However across successive evaluations for various values if T, required sketches are related. Random IO can be limited due to prefetching. DIMACS Summer School

  25. Experimental Evaluation • Real data from a service AT&T provides (utilization information). • Size varying from 16MB (approx. 1 month) to 256MB (approx 1 year) worth of data. • Evaluated: • Time to construct sketches • Scalability of sketch construction • Efficiency of the proposed convolution based technique • Time to compute relaxed period and average trends • computing sketches from scratch and with pre-computed sketches • Using brute force approaches • Accuracy of sketching • Comparison with other time series reduction techniques DIMACS Summer School

  26. Time to construct sketches DIMACS Summer School

  27. Time to construct sketches DIMACS Summer School

  28. Time to construct without convolution DIMACS Summer School

  29. Time to construct sketches without convolution DIMACS Summer School

  30. Computing relaxed periods DIMACS Summer School

  31. Computing Relaxed Periods DIMACS Summer School

  32. Computing relaxed period with precomputed sketches DIMACS Summer School

  33. Computing Relaxed Periods Without Precomputed Sketches DIMACS Summer School

  34. Brute Force Algorithms DIMACS Summer School

  35. Brute Force Algorithms DIMACS Summer School

  36. Computing Average Trend DIMACS Summer School

  37. Computing Average Trend DIMACS Summer School

  38. Accuracy of Sketches DIMACS Summer School

  39. Clustering Tabular Data • Many applications produce data in two dimensional array form. • Consider traditional telecommunication applications: • Data are collected from a variety of collection stations across the country, recording call volume at some temporal granularity. • 2d call volume data set (spatial ordering of collection stations versus time) recording temporal call activity, approx. 18MB/day. DIMACS Summer School

  40. DIMACS Summer School

  41. Clustering tabular data • Data elements to be clustered are rectangular data regions. • Clustering might reveal interesting similarities (in call volume and time) between geographical regions. • One month ~ 600MB of data. • Sketch rectangular regions: • extend sketches in 2d • sketching with respect to any Lp norm p in (0.2] DIMACS Summer School

  42. DIMACS Summer School

  43. Summary of results • Sketch construction scales nicely with respect to data volume and sketch size. • Convolution based sketch computation is very effective. • Sketch based approach is orders of magnitude better than brute force for computing relaxed periods and average trends. • Performance benefits increase for larger data sets. • If sketches are pre-computed, clustering can be performed in seconds even for very large data sets. • In practice sketches of low dimensionality provide great accuracy. • Compared with other dimensionality reduction techniques, the sketch based approach is more accurate and effective. DIMACS Summer School

  44. Conclusions • Scalability to large data volume requirement of the data mining process. • Effectively reduce data volume using sketches. • Preserve data properties required by mining algorithms (e.g., various distances). • Core techniques, various algorithms could benefit from them. • Very large performance benefits, small loss in accuracy. DIMACS Summer School

  45. Contact • koudas@research.att.com • www.research.att.com/~koudas DIMACS Summer School

More Related