Introduction to Data Mining Mining Association Rules

1. Introduction to Data MiningMining Association Rules Reference: Tan et al: Introduction to data mining. Some slides are adopted from Tan et al

2. What is Data Mining? • Data mining is the exploration and analysis of large quantities of data in order to discover valid, novel, potentially useful, and ultimately understandable patterns in data.

3. What is data mining (cont.)? Valid: The patterns hold in general. Novel: We did not know the pattern beforehand. Useful: We can devise actions from the patterns. Understandable: We can interpret and comprehend the patterns. Data mining falls into the broad field of knowledge discovery

4. The data mining process

5. Origins of Data Mining • Draws ideas from machine learning/Artificial Intelligence, pattern recognition, statistics, and database systems • Human analysis and traditional Techniquesmay be unsuitable due to • Enormity of data • High dimensionality of data • Heterogeneous data • Distributed nature of data Statistics/AI

6. Data Mining Tasks • Prediction Methods • Use some variables to predict unknown or future values of other variables. • Description Methods • Find human-interpretable patterns that describe the data.

7. Data Mining Tasks... • Association Rule Discovery [Descriptive] • Clustering [Descriptive] • Classification [Predictive] (for discrete variables) • Sequential Pattern Discovery [Descriptive] • Regression [Predictive] (for continuous variable) • Deviation Detection [Predictive]

8. Association Rule Discovery: Definition • Given a set of records each of which contain some number of items from a given collection; • Produce dependency rules which will predict occurrence of an item based on occurrences of other items. Rules Discovered: {Milk} --> {Coke} {Diaper, Milk} --> {Beer}

9. Association Rule Discovery: Sample Application • Supermarket shelf management. • Goal: To identify items that are bought together by sufficiently many customers. • Approach: Process the point-of-sale data collected with barcode scanners to find dependencies among items.

10. Clustering: Definition • Given a set of data points, each having a set of attributes, and a similarity measure among them, find clusters such that • Data points in one cluster are more similar to one another. • Data points in separate clusters are less similar to one another. • Similarity Measures: • Euclidean Distance if attributes are continuous. • Other Problem-specific Measures.

11. Illustrating Clustering • Euclidean Distance Based Clustering in 3-D space.

12. Clustering: Sample Application 1 • Market Segmentation: • Goal: subdivide a market into distinct subsets of customers where any subset may conceivably be selected as a market target to be reached with a distinct marketing mix. • Approach: • Collect different attributes of customers based on their geographical and lifestyle related information. • Find clusters of similar customers. • Measure the clustering quality by observing buying patterns of customers in same cluster vs. those from different clusters.

13. Clustering: sample application 2 • Document Clustering: • Goal: To find groups of documents that are similar to each other based on the important terms appearing in them. • Approach: To identify frequently occurring terms in each document. Form a similarity measure based on the frequencies of different terms. Use it to cluster. • Gain: Information Retrieval can utilize the clusters to relate a new document or search term to clustered documents.

14. Classification: Definition • Given a collection of records (training set ) • Each record contains a set of attributes, one of the attributes is the class. • Find a model for class attribute as a function of the values of other attributes. • Goal: previously unseen records should be assigned a class as accurately as possible. • A test set is used to determine the accuracy of the model. Usually, the given data set is divided into training and test sets, with training set used to build the model and test set used to validate it.

15. Classification and Clustering • Classification: • Classes pre-defined • Uses training set (thus also known as supervised learning) • Clustering: • Classes not defined in advance • No training set (thus also known as unsupervised learning)

16. Classification: Example

17. Classification: sample application 1 • Direct Marketing • Goal: Reduce cost of mailing by targeting a set of consumers likely to buy a new cell-phone product. • Approach: • Use the data for a similar product introduced before. • We know which customers decided to buy and which decided otherwise. This {buy, don’t buy} decision forms the class attribute. • Collect various demographic, lifestyle, and company-interaction related information about all such customers. • Type of business, where they stay, how much they earn, etc. • Use this information as input attributes to learn a classifier model.

18. Classification: sample application 2 • Fraud Detection • Goal: Predict fraudulent cases in credit card transactions. • Approach: • Use credit card transactions and the information on its account-holder as attributes. • When does a customer buy, what does he buy, how often he pays on time, etc • Label past transactions as fraud or fair transactions. This forms the class attribute. • Learn a model for the class of the transactions. • Use this model to detect fraud by observing credit card transactions on an account.

19. Sequential Pattern Discovery: Definition • Given a set of objects, with each object associated with its own timeline of events, find rules that predict strong sequential dependencies among different events. • Rules are formed by first discovering patterns. Event occurrences in the patterns are governed by timing constraints.

20. Regression • Predict a value of a given continuous valued variable based on the values of other variables, assuming a linear or nonlinear model of dependency. • Greatly studied in statistics, neural network fields. • Examples: • Predicting sales amounts of new product based on advetising expenditure. • Predicting wind velocities as a function of temperature, humidity, air pressure, etc. • Time series prediction of stock market indices.

21. Deviation/Anomaly Detection • Detect significant deviations from normal behavior • Applications: • Credit Card Fraud Detection • Network Intrusion Detection

22. Mining Association Rules

23. Association Rule Mining • Given a set of transactions, find rules that will predict the occurrence of an item based on the occurrences of other items in the transaction Market-Basket transactions Example of Association Rules {Diaper}  {Beer},{Milk, Bread}  {Eggs,Coke},{Beer, Bread}  {Milk}, Implication means co-occurrence!

24. Definition: Frequent Itemset • Itemset • A collection of one or more items • Example: {Milk, Bread, Diaper} • k-itemset • An itemset that contains k items • Support count () • Frequency of occurrence of an itemset • E.g. ({Milk, Bread,Diaper}) = 2 • Support • Fraction of transactions that contain an itemset • E.g. s({Milk, Bread, Diaper}) = 2/5 • Frequent Itemset • An itemset whose support is greater than or equal to a minsup threshold

25. Example: Definition: Association Rule • Association Rule • An implication expression of the form X  Y, where X and Y are itemsets • Example: {Milk, Diaper}  {Beer} • Rule Evaluation Metrics • Support (s) • Fraction of transactions that contain both X and Y • Confidence (c) • Measures how often items in Y appear in transactions thatcontain X

26. Association Rule Mining Task • Given a set of transactions T, the goal of association rule mining is to find all rules having • support ≥ minsup threshold • confidence ≥ minconf threshold • Brute-force approach: • List all possible association rules • Compute the support and confidence for each rule • Prune rules that fail the minsup and minconf thresholds  Computationally prohibitive!

27. Mining Association Rules Example of Rules: {Milk,Diaper}  {Beer} (s=0.4, c=0.67){Milk,Beer}  {Diaper} (s=0.4, c=1.0) {Diaper,Beer}  {Milk} (s=0.4, c=0.67) {Beer}  {Milk,Diaper} (s=0.4, c=0.67) {Diaper}  {Milk,Beer} (s=0.4, c=0.5) {Milk}  {Diaper,Beer} (s=0.4, c=0.5) • Observations: • All the above rules are binary partitions of the same itemset: {Milk, Diaper, Beer} • Rules originating from the same itemset have identical support but can have different confidence • Thus, we may decouple the support and confidence requirements

28. Mining Association Rules • Two-step approach: • Frequent Itemset Generation • Generate all itemsets whose support  minsup • Rule Generation • Generate high confidence rules from each frequent itemset, where each rule is a binary partitioning of a frequent itemset • Frequent itemset generation is still computationally expensive

29. Frequent Itemset Generation Given d items, there are 2d possible candidate itemsets

30. Frequent Itemset Generation • Brute-force approach: • Each itemset in the lattice is a candidate frequent itemset • Count the support of each candidate by scanning the database • Match each transaction against every candidate • Complexity ~ O(NMw) => Expensive since M = 2d!!!

31. Computational Complexity • Given d unique items: • Total number of itemsets = 2d • Total number of possible association rules: If d=6, R = 602 rules

32. Frequent Itemset Generation Strategies • Reduce the number of candidates (M) • Complete search: M=2d • Use pruning techniques to reduce M • Reduce the number of transactions (N) • Reduce size of N as the size of itemset increases • Used by DHP and vertical-based mining algorithms • Reduce the number of comparisons (NM) • Use efficient data structures to store the candidates or transactions • No need to match every candidate against every transaction

33. Reducing Number of Candidates • Apriori principle: • If an itemset is frequent, then all of its subsets must also be frequent • Apriori principle holds due to the following property of the support measure: • Support of an itemset never exceeds the support of its subsets • This is known as the anti-monotone property of support

34. Illustrating Apriori Principle Found to be Infrequent Pruned supersets

35. Illustrating Apriori Principle Items (1-itemsets) Pairs (2-itemsets) (No need to generatecandidates involving Cokeor Eggs) Minimum Support count = 3 Triplets (3-itemsets) If every subset is considered, 6C1 + 6C2 + 6C3 = 41 With support-based pruning, 6 + 6 + 1 = 13

36. Apriori Algorithm • Method: • Let k=1 • Generate frequent k-itemsets • Repeat until no new frequent itemsets are identified • Generate candidate (k+1)-itemsets from frequent k-itemsets • Prune candidate itemsets containing subsets that are infrequent • Count the support of each candidate by scanning the DB • Eliminate candidates that are infrequent, leaving only those that are frequent

37. Example:apriori method for finding frequent itemset Find itemsets with support >=50% • Frequent 1-itemsets: • {bread}, {cookies}, {juice}, {milk} • 2. Generate candidate 2-itemsets: {bread, cookies}, {bread, Juice}, {bread, milk}, {cookies, juice}, {cookies, milk}, {juice, milk} • 3. Frequent 2-itemsets • {bread, cookies} {bread, Juice} {bread, milk} { juice, milk} • 4. candidate 3-itemsets • {bread, cookies, juice}, {bread, cookies,milk}, {bread, juice, milk}

38. Counting the support • support counting: • Scan the database of transactions to determine the support of each candidate itemset • To reduce the number of comparisons, store the candidates in a hash structure • Instead of matching each transaction against every candidate, match it against candidates contained in the hashed buckets (details omitted)

39. Pruned Rules Rule Generation for Apriori Algorithm Lattice of rules Low Confidence Rule

40. Rule Generation for Apriori Algorithm • Candidate rule is generated by merging two rules that share the same prefixin the rule consequent • join(CD=>AB,BD=>AC)would produce the candidaterule D => ABC • Prune rule D=>ABC if itssubset AD=>BC does not havehigh confidence

41. Example: generating rules • {bread, milk} {diaper} • {bread, diaper}  {milk} • {diaper, milk} {bread} • 2. {bread} {diaper, milk} 2/3 2/3 2/4 2/3 Find rules with confidence >=2/3 from the itemset {bread, milk, diaper}

42. Next • Clustering • classification