1 / 17

Data mining

Data mining. A Linear Method for Deviation Detection in Large databases Presented by: Ali Triki Date: 09/30/1999. Content. What are Deviations Approach Exact exception problem Sequential exception problem Algorithm Dissimilarity function Experimental results conclusion.

quinn-heath
Download Presentation

Data mining

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Data mining A Linear Method for Deviation Detection in Large databases Presented by: Ali Triki Date: 09/30/1999

  2. Content • What are Deviations • Approach • Exact exception problem • Sequential exception problem • Algorithm • Dissimilarity function • Experimental results • conclusion

  3. What are Deviations? • Deviations are errors or noise in data • Several approaches for detecting deviations (or exceptions) in the areas of Databases and Machine Learning • Statistical approach (Hoaglin 1983) • Extending learning algorithms to cope with small amount of noise (Aha 1991) • Impact of erroneous examples on the learning results (Quinlan 1986)

  4. Approach • Use the implicit redundancy in the data to detect deviations. • Clustering data into 2 clusters: deviation and non deviations. • Do not discard deviation as noise, but try to isolate small minorities.

  5. Exact Exception Problem • Problem description • Set of Items I= {1,4,4,4} • Cardinality function: C(I) • Dissimilarity Function: the variance of the numbers in the set = 1/n (xi- x)2 • Smoothing factor: C(I-Ij) * (D(I)-D(I-Ij)) • By computing each candidate exception set Ij we get the following results:

  6. Example • The candidate set = {1} is an exception because it has a large smoothing factor SF

  7. Sequential Exception Problem • After seeing a series of similar data, an element disturbing the series is considered an exception • Given: • A set of items I • A sequence S of subsets:: Ij I and Ij-1Ij • Cardinality function • Smoothing factor: SF(Ij)=C(Ij-Ij-1) * (D(Ij)-D(Ij-1)) The Smoothing factor consider the difference with the preceding set instead of the complimentary set

  8. Algorithm • 1- Get the first element i1 of the item set I making up the element subset I1I and compute Ds(I1) • 2- For each following element ij in S, create the subset Ij taking Ij= Ij-1U {ij} and compute the difference in dissimilarity values dj=Ds(Ij) – Ds(Ij-1) • 3- Consider that element ij with the maximal value of dj>0 to be the answer for this iteration. • If dj 0 for all Ij in S, there is no exception

  9. Algorithm • If an exception ij is found: • For each element ik where k>j compute • dk0=Ds(Ij-1U {ik}) –Ds (Ij-1) • dk1=Ds(IjU {ik}) –Ds (Ij) • Add to Ix those ik for which dk0 –dk1  dj • For m iterations, we get m competing exception sets Ix, select the one with the largest value of difference in dissimilarity dj scaled with the dissimilarity function C

  10. Dissimilarity function • Handles the comparison of the character strings, it maintains a pattern of a regular expression that matches all the character strings seen so far. • Starting with the pattern of the 1st string, we introduce wildcard characters as more strings need to be covered. • Ds(Ij)= Ds(Ij-1) + J*(Ms(Ij)-Ms(Ij-1))/Ms(Ij) • Auxiliary function Ms(Ij )= 1/ (3*c-w+2) • With c being the total number of characters • And w being the number of needed wildcards

  11. Experimental Results 1

  12. Experimental Results 2

  13. Experimental Results 3

  14. A Failure example

  15. Why did it fail? • The dissimilarity function used couldn’t catch the exception. • Once 2 values ‘..,n,..’ and ‘..,y,..’ are seen , the pattern takes the form ‘...,*,…’ from then on, there is no change in pattern when ‘?’ appears in the same column as the pattern covers it. • Need a more powerful dissimilarity function.

  16. Conclusion • We presented a linear algorithm for sequential exception problem. • Experimental evaluation shows that the effectiveness of the algorithm depends on the dissimilarity function used. • It seems helpful to have some predefined D.F that works well for particular datasets.

  17. References: • A. Arning, R. Agrawal, P. Raghavan: "A Linear Method for Deviation Detection in Large Databases", Proc. of the 2nd Int'l Conference on Knowledge Discovery in Databases and Data Mining, Portland, Oregon, August, 1996 • S. Sarawagi, R. Agrawal, N. Megiddo: "Discovery-driven exploration of OLAP data cubes", Proc. of the Sixth Int'l Conference on Extending Database Technology (EDBT), Valencia, Spain, March 1998 • R. Agrawal and R Srikant “Fast Algorithms for mining association rules” In Proceedings of the VLDB Conference 1994

More Related