1 / 52

Chapter 10 ASSOCIATION RULES

Chapter 10 ASSOCIATION RULES. Cios / Pedrycz / Swiniarski / Kurgan. Outline. Introduction Association Rules and Transactional Data Basic Concepts Mining Single Dimensional, Single-Level Boolean Association Rules Naïve Algorithm Apriori Algorithm

jelder
Download Presentation

Chapter 10 ASSOCIATION RULES

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Chapter 10ASSOCIATION RULES Cios / Pedrycz / Swiniarski / Kurgan

  2. Outline • Introduction • Association Rules and Transactional Data • Basic Concepts • Mining Single Dimensional, Single-Level Boolean Association Rules • Naïve Algorithm • Apriori Algorithm • Generating Association Rules from Frequent Itemsets • Improving Efficiency of the Apriori Algorithm • Finding Interesting Association Rules

  3. Introduction Association rules mining is another, after clustering, key unsupervised data mining method that is able to find interesting associations (relationships, dependencies) in large not labeled sets of data.

  4. Introduction AR are used to describe associations or correlations among a set of items in transaction databases, relational databases, and data warehouses • Applications in basket data analysis, cross-marketing, catalog design, clustering, data preprocessing, genomics, etc. - rule format: LHS  RHS [support, confidence] Examples: buys(x, diapers)  buys(x, beer) [5%, 60%] major(x, CS) AND takes(x, Data Mining)  level(x, PhD) [1%, 75%]

  5. Introduction in this shopping basket customer bought tomatoes, carrots, bananas, bread, eggs, sup, milk, etc. how the demographical information affects what the customer buys? is bread usually bought with milk? does a specific milk brand make any difference? what can be learned using association rules? where we place the tomatoes in the store to maximize their sales? is the bread bought when both milk and eggs are bought together?

  6. Introduction AR can be derived from data that describe events occurring at the same time or in close proximity. They are: • useful, when containing high quality, actionable information • diapers  beer • trivial, when they are supported by data, but useless if they describe well known facts • milk  eggs • inexplicable, when they are valid and new, but cannot be utilized • grocery store  milk is sold as often as bread

  7. Introduction Some of the common uses: • planning store layouts • we can place products together when they have a strong relationship r OR • we can place such products far apart to increase traffic to motivate people to purchase other items • planning product bundles and offering coupons • knowing that buys(x, diapers)  buys(x, beer), discounts are not offered on both beer and diapers at the same time • in general we discount one to increase sales and make money on the other • designing direct marketing campaign • mailing a camcorder promotion to people who bought VCR is best when it comes approximately two to three months after the VCR purchase

  8. Introduction How do we derive ARs: • techniques are based on probability and statistics • the process consists of 4 steps • prepare input data in the required format • choose items of interest… (itemsets) • compute probabilities and conditional probabilities • generate (the most probable) association rules

  9. Transactional Data Data are provided in a transactional form • each record consists of transaction ID and information about all items that constitute the transaction Subset of all available items Transaction ID

  10. Transactional Data Transactional data example • rules can be derived just by looking at the data Beer  Eggs Apples  Celery

  11. Transactional Data Nominal data can also be transformed into transactions • transformed into transactional format

  12. Association Rules What do we want to do? • given database of transactions, where each transaction is a list of items (e.g., purchased) • find all rules that correlate presence of one set of items with that of another set of items • how many items do we consider in each set? • how do we define correlation? • how strong the correlation should be? • how do we extract useful rules? Let us answer these questions

  13. Basic Definitions itemset transaction association rule dataset (set of transactions)

  14. Basic Definitions How to measure interestingness of the rules? • each rule has two measures • support (P(A^B)) which indicates the frequencies of the occurring patterns ((A+B)/Total) • defined as ratio of # of transactions containing A and B to the total # of transactions • confidence (P(B|A)) which denotes the strength of implication in the rule ((A+B)/A) • defined as ratio of # of transactions containing A and B to the #of transactions containing A

  15. customer buys diapers customer buys both customer buys beer Basic Definitions Interestingness of the rules example: diapers  beer • support • confidence

  16. Basic Definitions How to measure interestingness of the rules? • let’s explain the concepts in terms of probabilities • find all the rules A and B  C with minimum support and confidence • support is a probability that a transaction contains {A, B, C} • confidence is a conditional probability that a transaction having {A, B}, also contains C A and B  C (support is 25%, confidence is 100%) if we want minimum support of 50% and minimum confidence of 50%, we get two rules: A  C (support 50%, confidence 66.6%) C  A (support 50%, confidence 100%)

  17. Basic Definitions What is I? What is T for TID=2000? What is support(Beer  Eggs)? What is confidence(Beer  Eggs)?

  18. Basic Definitions What is I? Apples, Beer, Celery, Diapers, Eggs m (# of items) = 5 What is T for TID=2000? Beer, Celery, Eggs What is support(Beer  Eggs)? 3/4 = 75% What is confidence(Beer  Eggs)? 3/3 = 100%

  19. Basic Definitions Frequent itemsets • itemset is any set of items • k-itemset is an itemset containing k items • frequent itemset is an itemset that satisfies a minimum support level • the problem arises when we try to analyze dataset that contains m items • how many itemsets are there? • many if m is large

  20. Strong Association Rules Given an itemset we can write association rules in many different formats. for itemset {Beer, Diapers}:  Beer, Diapers Beer  Diapers Diapers  Beer Diapers, Beer  • we are interested only in strong rules • those which satisfy minimum support and minimum confidence • both are user-specified

  21. Strong Association Rules Given itemsets, it is relatively easy to generate association rules • for these two itemsets: {Beer, Diapers} with support 40% {Beer, Diapers, Eggs} with support 20% • rule Beer, Diapers => Eggs IF customer buys Beer and Diapers THEN we can infer the probability that she buys Eggs is 50%

  22. Association Rules To generate AR we follow two basic steps • Find all frequent itemsets • those that satisfy minimum support • Find all strong association rules • generate association rules from frequent itemsets • select and keep rules that satisfy minimum confidence

  23. Naïve Algorithm How do we generate frequent itemsets (step 1)? • Naïve algorithm n = |D| for each subset s of I { counter = 0 for each transaction T in D { if s is a subset of T counter = counter + 1 } if minimum support ≤ counter / n add s to frequent itemsets }

  24. Naïve Algorithm Does the Naïve algorithm work well? • we have 2m subsets of I • we have to scan n transactions for each subset • thus we perform O(2mn) tests • O(2mn) complexity • grows exponentially with the number of items m • thus we must use some other approach!

  25. Frequent Itemsets Frequent itemsets satisfy the apriori property • if A is not a frequent itemset, then any superset of A is not a frequent itemset either • this property is used to speed up computations Proof n is # of transactions suppose A is a subset of i transactions if A’  A, then A’ is a subset of i’  i transactions thus if i/n < minimum support, so is i’/n

  26. Frequent Itemsets Using the apriori property • candidate k-itemsets are build from frequent (k-1)-itemsets: • find all frequent 1-itemsets • extend (k-1)-itemsets to candidate k-itemsets • prune candidate itemsets that do not meet the minimum support

  27. Apriori Algorithm Improved algorithm to generate frequent itemsets L1 = {frequent 1-itemsets} for (k=2; L(k-1) is not empty; k++) { Ck is generated as k-itemset candidate from L(k-1) for each transaction T in D { Ct=subset(Ck,T) // k-itemsets that are subsets of T for each k-itemset c in Ct c.count++; } Lk = {c in Ck such that c.count ≥ minimum support} } the frequent itemsets are the union of the Lk

  28. L1 = {frequent 1-itemsets} for (k=2; L(k-1) is not empty; k++) { Ck is generated as k-itemset candidates from L(k-1) for each transaction T in D { Ct=subset(Ck,T) // candidates that are subsets of T for each candidate c in Ct c.count++; } Lk = {c in Ck such that c.count ≥ minimum suport} } the frequent itemsets are the union of the Lk Frequent Itemsets Apriori approach reduces number of considered itemsets (# of scans of D) • how do we generate k-itemset candidates? • for each item i that is not in a given frequent (k-1)-itemset, but in some other frequent (k-1) itemset in Lk-1 • add i to the (k-1)-itemset to create a k-itemset candidate • remove duplicates • example frequent 1-itemsets {A}, {B}, {C} candidate 2-itemsets {A, B}, {A, C}, {B, A}, {B, C}, {C, A}, {C, B} eliminate duplicates {A, B}, {A, C}, {B, C} • join together frequent (k-1)-itemsets • if frequent (k-1)-itemsets have (k-2)-items in common, create a k-itemset candidate by adding two different items to (k-2) common items • example{A, B, C} joined with {A, B, D} gives {A, B, C, D}

  29. Find all frequent itemsets • those that satisfy minimum support • Find all strong association rules • generate association rules from frequent itemsets • select and keep rules that satisfy minimum confidence Association Rules • The frequent itemset is computed iteratively • 1st iteration • large 1-candidate itemsets are found by scanning • kth iteration • large k-candidate itemsets are generated by applying apriori-based generation to large (k-1)-itemsets • apriori rule generates only those k-itemsets whose every (k-1)-itemset subset is frequent (above the threshold) • Generating rules • for each frequent itemset X output all rulesY  (X – Y) if s(X) / s(Y) > minimum confidence • Y is a subset of X

  30. Example We will generate AR from the below transactional data with • minimum support = 50% • minimum confidence = 60% 

  31. generate candidate 1-itemsets generate frequent 2-itemsets generate candidate 2-itemsets generate frequent 1-itemsets generate frequent 3-itemsets generate candidate 3-itemsets L1 Example C1 delete candidates below minimum support C2 L2 we do not use 3-itemsets with (A, B) and (A, E)they are below minimum support C3 L3

  32. Example Finally, we generate strong association rules • from the frequent 3-itemset {B, C, E} that satisfies s = 50% • we need to satisfy minimum confidence of 60% B and C  E with support = 50% and confidence = 2/2=100% B and E  C with support = 50% and confidence = 2/3=66.7% C and E  B with support = 50% and confidence = 2/2=100% B  C and E with support = 50% and confidence = 2/3=66.7% C  B and E with support = 50% and confidence = 2/3=66.7% E  B and C with support = 50% and confidence = 2/3=66.7%

  33. Improving Efficiency of Apriori The Apriori algorithm has been modified to improve its efficiency (computational complexity) by: • hashing • removal of transactions that do not contain frequent itemsets • sampling of the data • partitioning of the data • mining frequent itemsets without generation of candidate itemsets

  34. Improving efficiency: Hashing • Hashing • is used to reduce the size of the candidate k-itemsets, i.e., itemsets that are generated from frequent itemsets from iteration k-1, Ck, for k>1 • for instance, when scanning D to generate L1 from the candidate 1-temsets in C1, we can at the same time generate all 2-itemsets for each transaction, hash (map) them into different buckets of the hash table structure and increase the corresponding bucket counts • a 2-itemset, which corresponding bucket count is below the support threshold, cannot be frequent and thus we can remove it from the candidate set C2. In this way we reduce the number of candidate 2-itemsets that must be examined to obtain L2

  35. Improving efficiency: Hashing • To add itemset, start at root and go down until a leaf is reached • At interior node at depth d, decide which branch to follow by applying hash function to the dth item of the itemset • When # of items in a leaf node exceeds some threshold convert leaf node to an internal node

  36. Improving efficiency: Hashing • To find candidates contained in given transaction, t • Hash on every item in t at the root node • to ensure that itemsets that start with an item not in t are ignored • At interior node reached by hashing the item i in the transaction hash on each item that comes after i in t

  37. Improving efficiency: Hashing Let C3 = {{1 2 3}, {1 2 4}, {1 3 4}, {1 3 5}, {2 3 4}} Ct = subset(C3, {1 0 1 1 0}) First, build hash-tree with candidate itemsets Then, check which of those itemsets are actually in the given transaction

  38. At root (d=1), hash on items in t: 1, 3, 4. 3, 4 return nothing since no itemsets start with 3 or 4. 2 Is ignored since not in t. Hash-Tree t = {1 0 1 1 0} 2 Check which are in t. Those that are get added to Ct as output from subset function. Manually verify that of the 5 itemsets in C3 only {1 3 4} was actually present in the transaction

  39. Improving Efficiency of Apriori • Removal of transactions that do not contain frequent itemsets In general, if a transaction does not contain any frequent k-itemsets, it cannot contain any frequent (k+1) itemsets, and thus can be removed from computation of any frequent t-itemsets, where t > k

  40. Improving Efficiency of Apriori • Sampling of the data • we generate association rules based on a sampled subset of transactions in D • a randomly selected subset S of D is used to search for the frequent itemsets • generation of frequent itemsets from S is more efficient (faster) but some of the rules that would have been generated from D may be missing, and some rules generated from S may not be present in D

  41. Improving Efficiency of Apriori • Partitioning of the data • partitioning generates frequent itemsets by finding frequent itemsets in subsets (partition) of D

  42. Improving Efficiency of Apriori • Mining frequent itemsets without generation of candidate itemsets • one of the main limiting aspects of the Apriori is that it can generate very large number of candidate itemsets • for instance, for 10,000 1-itemsets, the Apriori algorithm generates approximately 10,000,000 candidate 2-itemsets • other limiting aspect is that the Apriori may need to repeatedly scan the data set D • to address these issues, a divide-and-conquer method, which decomposes the overall problem into a set of smaller tasks, is used • the method, referred to as frequent-pattern growth (FP-growth) compresses the set of frequent (individual) items from D into a frequent pattern tree (FP-tree)

  43. Finding Interesting Association Rules Depending on the minimum support and confidence values the user may generate a large number of rules to analyze and assess How to filter out the rules that are potentially the most interesting? • whenever a rule is interesting (or not) can be evaluated either objectively or subjectively • the ultimate but subjective user’s evaluation cannot be quantified or anticipated; they are different for different users • that is why objective interestingness measures, based on the statistical information present in D are used

  44. Finding Interesting Association Rules The subjective evaluation of association rules often boils down to checking if a given rule is unexpected (i.e., surprises the user) and actionable (i.e., the user can use it for something useful) • useful, when they provide high quality actionable informatione.g. diapers  beers • trivial, when they are valid and supported by data but useless since they confirm well known factse.g. milk  bread • inexplicable, when they contain valid new facts but cannot be utilizede.g. grocery_store  milk_is_sold_as_often_as_bread

  45. Finding Interesting Association Rules In most cases, confidence and support values associated with each rule are used as the objective measure to select the most interesting rules • rules that have these values higher with respect to other rules are preferred • although this simple approach works in many cases, we will show that sometimes rules with high confidence and support may be uninteresting or even misleading

  46. Finding Interesting Association Rules Objective interestingness measures • example • let us assume that transactional data contains milk and bread as the frequent items • 2,000 transactions were recorded • in 1,200 customers bought milk • in 1,650 customers bought bread • in 900 customers bought both milk and bread

  47. Finding Interesting Association Rules Objective interestingness measures • Example • given the minimum support threshold of 40% and minimum confidence threshold of 70% rule “milk  bread [45%, 75%]” would be generated • on the other hand, due to low support and confidence values the rule “milk  not bread [15%, 25%]” would not be generated • the latter rule is by far more “accurate” while the first may be misleading

  48. Finding Interesting Association Rules Objective interestingness measures • example • “milk  bread [45%, 75%]” rule • probability of buying bread is 82.5%, while confidence of milk  bread is lower and equals 75% • bread and milk are negatively associated, i.e., buying one decreases buying the other • obviously using this rule would not be a wise decision

  49. Finding Interesting Association Rules Objective interestingness measures • alternative approach to evaluate interestingness of association rules is to use measures based on correlation • for A  B rule, the itemset A is independent of the occurrence of the itemset B if P(A  B) = P(A)P(B). Otherwise, itemsets A and B are dependent and correlated as events. • correlation measure (also referred to as lift and interest), is defined between itemsets A and B as

  50. Finding Interesting Association Rules Objective interestingness measures • correlation measure • if the correlation value is less than 1, then the occurrence of A is negatively correlated (inhibits) the occurrence of B • if the value is greater than 1, then A and B are positively correlated, which means that occurrence of one promotes occurrence of the other • if correlation equals 1, then A and B are independent, i.e., there is no correlation between these itemsets • correlation value for the rule milk  bread is 0.45 / (0.6*0.825) = 0.45 / 0.495 = 0.91

More Related