DATA MINING LECTURE 4

1. DATA MININGLECTURE 4 Frequent Itemsets, Association Rules Evaluation Alternative Algorithms

2. RECAP

3. Mining Frequent Itemsets • Itemset • A collection of one or more items • Example: {Milk, Bread, Diaper} • k-itemset • An itemset that contains k items • Support () • Count: Frequency of occurrence of an itemset • E.g. ({Milk, Bread,Diaper}) = 2 • Fraction: Fraction of transactions that contain an itemset • E.g. s({Milk, Bread, Diaper}) = 40% • Frequent Itemset • An itemset whose support is greater than or equal to a minsupthreshold, minsup • Problem Definition • Input: A set of transactions T, over a set of items I, minsupvalue • Output: All itemsets with items in I having minsup

4. The itemsetlattice Given d items, there are 2dpossible itemsets Too expensive to test all!

5. The Apriori Principle • Aprioriprinciple (Main observation): • If an itemset is frequent, then all of its subsets must also be frequent • If an itemset is not frequent, then all of its supersets cannot be frequent • The support of an itemsetnever exceeds the support of its subsets • This is known as the anti-monotone property of support

6. Illustration of the Apriori principle Frequent subsets Found to be frequent

7. Found to be Infrequent Pruned Illustration of the Apriori principle Infrequent supersets

8. The Apriori algorithm Ck = candidate itemsets of size k Lk = frequentitemsets of size k Level-wise approach • k = 1,C1 = all items • While Ck not empty • Scan the database to find which itemsets in Ckare frequent and put them into Lk • Use Lkto generate a collection of candidateitemsetsCk+1of size k+1‏ • k = k+1 Frequent itemset generation Candidate generation R. Agrawal, R. Srikant: "Fast Algorithms for Mining Association Rules", Proc. of the 20th Int'l Conference on Very Large Databases, 1994.

9. Candidate Generation • Basic principle (Apriori): • An itemset of size k+1 is candidate to be frequent only if all of its subsets of size k are known to be frequent • Main idea: • Construct a candidate of size k+1 by combining two frequentitemsets of size k • Prunethe generated k+1-itemsets that do not have allk-subsets to be frequent

10. Computing Frequent Itemsets • Given the set of candidateitemsetsCk, we need to compute the support and find the frequentitemsetsLk. • Scan the data, and use a hash structure to keep a counter for each candidate itemset that appears in the data Ck

11. A simple hash structure • Create a dictionary (hash table) that stores the candidate itemsets as keys, and the number of appearances as the value. • Initialize with zero • Increment the counter for each itemset that you see in the data

12. Example Suppose you have 15 candidate itemsets of length 3: {1 4 5}, {1 2 4}, {4 5 7}, {1 2 5}, {4 5 8}, {1 5 9}, {1 3 6}, {2 3 4}, {5 6 7}, {3 4 5}, {3 5 6}, {3 5 7}, {6 8 9}, {3 6 7}, {3 6 8} Hash table stores the counts of the candidate itemsets as they have been computed so far

13. Example Tuple {1,2,3,5,6} generates the following itemsetsof length 3: {1 2 3}, {1 2 5}, {1 2 6}, {1 3 5}, {1 3 6}, {1 5 6}, {2 3 5}, {2 3 6}, {3 5 6}, Increment the counters for the itemsets in the dictionary

14. Example Tuple {1,2,3,5,6} generates the following itemsetsof length 3: {1 2 3}, {1 2 5}, {1 2 6}, {1 3 5}, {1 3 6}, {1 5 6}, {2 3 5}, {2 3 6}, {3 5 6}, Increment the counters for the itemsets in the dictionary

15. Example: Mining Association Rules • Association Rule • An implication expression of the form X  Y, where X and Y are itemsets • {Milk, Diaper}  {Beer} • Rule Evaluation Metrics • Support (s) • Fraction of transactions that contain both X and Y= the probabilityP(X,Y) that X and Y occur together • Confidence(c) • How often Y appears in transactions that contain X = the conditional probability P(Y|X)that Yoccurs given that Xhas occurred. • Problem Definition • Input A set of transactions T, over a set of items I, minsup, minconfvalues • Output: All rules with items in I having s ≥ minsupand c≥ minconf

16. Mining Association Rules • Two-step approach: • Frequent Itemset Generation • Generate all itemsets whose support  minsup • Rule Generation • Generate high confidence rules from each frequent itemset, where each rule is a partitioning of a frequent itemset into Left-Hand-Side (LHS) and Right-Hand-Side (RHS) Frequent itemset: {A,B,C,D} Rule:ABCD

17. Association Rule anti-monotonicity • Confidence is anti-monotone w.r.t. number of items on the RHS of the rule (or monotone with respect to the LHS of the rule) • e.g., L = {A,B,C,D}:c(ABC  D)  c(AB  CD)  c(A  BCD)

18. Rule Generation for APrioriAlgorithm • Candidate rule is generated by merging two rules that share the same prefixin the RHS • join(CDAB,BDAC)would produce the candidaterule D ABC • Prune rule D  ABCif itssubset ADBCdoes not havehigh confidence • Essentially we are doing APriori on the RHS

19. RESULT POST-PROCESSING

20. Compact Representation of Frequent Itemsets • Some itemsets are redundant because they have identical support as their supersets • Number of frequent itemsets • Need a compact representation

21. Maximal Frequent Itemset An itemset is maximal frequent if none of its immediate supersets is frequent Maximal Itemsets Maximalitemsets = positive border Infrequent Itemsets Border Maximal: no superset has this property

22. Negative Border Itemsets that are not frequent, but all their immediate subsets are frequent. Infrequent Itemsets Border Minimal: no subset has this property

23. Border • Border = Positive Border + Negative Border • Itemsets such that all their immediate subsets are frequent and all their immediate supersets are infrequent. • Either the positive, or the negative border is sufficient to summarize all frequent itemsets.

24. Closed Itemset • An itemset is closed if none of its immediate supersets has the same supportas the itemset

25. Maximal vs Closed Itemsets Transaction Ids Not supported by any transactions

26. Maximal vs Closed Frequent Itemsets Closed but not maximal Minimum support = 2 Closed and maximal # Closed = 9 # Maximal = 4

27. Maximal vs Closed Itemsets

28. Pattern Evaluation • Association rule algorithms tend to produce too many rules but many of them are uninteresting or redundant • Redundant if {A,B,C}  {D} and {A,B}  {D} have same support & confidence • Summarization techniques • Uninteresting, if the pattern that is revealed does not offer useful information. • Interestingness measures: a hard problem to define • Interestingness measures can be used to prune/rank the derived patterns • Subjective measures: require human analyst • Objective measures: rely on the data. • In the original formulation of association rules, support & confidence are the only measures used

29. f11: support of X and Yf10: support of X and Yf01: support of X and Yf00: support of X and Y Computing Interestingness Measure • Given a rule X  Y, information needed to compute rule interestingness can be obtained from a contingency table Contingency table for X  Y Used to define various measures • support, confidence, lift, Gini, J-measure, etc. : itemset X appears in tuple : itemset Y appears in tuple : itemset X does not appear in tuple : itemset Y does not appear in tuple

30. Drawback of Confidence Number of people that drink tea Number of people that drink coffee and tea Number of people that drink coffee but not tea Number of people that drink coffee • Association Rule: Tea  Coffee • Confidence= P(Coffee|Tea) = 0.75 • but P(Coffee) = 0.9 • Although confidence is high, rule is misleading • P(Coffee|Tea) = 0.9375

31. Statistical Independence • Population of 1000 students • 600 students know how to swim (S) • 700 students know how to bike(B) • 420 students know how to swim and bike (S,B) • P(S,B)= 420/1000 = 0.42 • P(S) P(B)= 0.6  0.7 = 0.42 • P(S,B)=P(S)  P(B)=> Statistical independence

32. Statistical Independence • Population of 1000 students • 600 students know how to swim (S) • 700 students know how to bike (B) • 500students know how to swim and bike (S,B) • P(S,B)= 500/1000 = 0.5 • P(S)  P(B)= 0.6  0.7 = 0.42 • P(S,B)>P(S)  P(B)=> Positively correlated

33. Statistical Independence • Population of 1000 students • 600 students know how to swim (S) • 700 students know how to bike (B) • 300students know how to swim and bike (S,B) • P(S,B)= 300/1000 = 0.3 • P(S)  P(B)= 0.6  0.7 = 0.42 • P(S,B)<P(S)  P(B)=> Negatively correlated

34. Statistical-based Measures • Measures that take into account statistical dependence • Lift/Interest/PMI In text mining it is called: Pointwise Mutual Information • Piatesky-Shapiro • All these measures measure deviation from independence • The higher, the better (why?)

35. Example: Lift/Interest • Association Rule: Tea  Coffee • Confidence= P(Coffee|Tea) = 0.75 • but P(Coffee) = 0.9 • Lift = 0.75/0.9= 0.8333 (< 1, therefore is negatively associated) = 0.15/(0.9*0.2)

36. Another Example If I was creating a document by picking words randomly, (of, the) have more or less the same probability of appearing together by chance No correlation (hong, kong) have much lower probability to appear together by chance. The two words appear almost always only together Positive correlation (obama, karagounis) have much higher probability to appear together by chance. The two words appear almost never together Negative correlation

37. Drawbacks of Lift/Interest/Mutual Information Rare co-occurrences are deemed more interesting. But this is not always what we want

38. ALTERNATIVE FREQUENT ITEMSET COMPUTATION Slides taken from Mining Massive Datasets course by AnandRajaraman and Jeff Ullman.

39. All pairs of items from L1 Count the items Count the pairs All items Frequent pairs Frequent items Finding the frequent pairs is usually the most expensive operation Filter Filter Construct Construct C1 L1 C2 L2 C3 First pass Second pass

40. Picture of A-Priori Item counts Frequent items Counts of pairs of frequent items Pass 1 Pass 2

41. PCY Algorithm • During Pass 1 (computing frequent items) of Apriori, most memory is idle. • Use that memory to keep a hash table where pairs of items are hashed. • The hash table keeps just countsof the number of pairs hashed in each bucket, not the pairs themselves. Item counts Pass 1

42. Needed Extensions • Pairs of items need to be generated from the input file; they are not present in the file. • We are not just interested in the presence of a pair, but we need to see whether it is present at least s (support) times.

43. PCY Algorithm – (2) • A bucket is frequent if its count is at least the support threshold. • If a bucket is not frequent, no pair that hashes to that bucket could possibly be a frequent pair. • The opposite is not true, a bucket may be frequent but hold infrequent pairs • On Pass 2 (frequent pairs), we only count pairs that hash to frequent buckets.

44. PCY Algorithm – Before Pass 1 Organize Main Memory • Space to count each item. • One (typically) 4-byte integer per item. • Use the rest of the space for as many integers, representing buckets, as we can.

45. Picture of PCY Hash table Item counts Pass 1

46. PCY Algorithm – Pass 1 FOR (each basket) { FOR (each item in the basket) add 1 to item’s count; FOR (each pair of items in the basket) { hash the pair to a bucket; add 1 to the count for that bucket } }

47. Observations About Buckets • A bucket that a frequent pair hashes to is surely frequent. • We cannot use the hash table to eliminate any member of this bucket. • Even without any frequent pair, a bucket can be frequent. • Again, nothing in the bucket can be eliminated. 3. But in the best case, the count for a bucket is less than the support s. • Now, all pairs that hash to this bucket can be eliminated as candidates, even if the pair consists of two frequent items.

48. PCY Algorithm – Between Passes • Replace the buckets by a bit-vector: • 1 means the bucket is frequent; 0 means it is not. • 4-byteintegers are replaced by bits, so the bit-vector requires 1/32 of memory. • Also, find which items are frequent and list them for the second pass. • Same as with Apriori

49. Picture of PCY Hash table Item counts Frequent items Bitmap Counts of candidate pairs Pass 1 Pass 2

50. PCY Algorithm – Pass 2 • Count all pairs {i, j } that meet the conditions for being a candidate pair: • Both i and j are frequent items. • The pair {i, j }, hashes to a bucket number whose bit in the bit vector is 1. • Notice both these conditions are necessary for the pair to have a chance of being frequent.