Review

1. Review

2. Topics to review for the final exam • Evaluation of classification • predicting performance, confidence intervals • ROC analysis • Precision, recall, F-Measure • Association Analysis • APRIORI • FP-Tree/FPGrowth • Maximal, Closed frequent itemsets • Cross support, h-measure • Confidence vs. interestingness • Mining sequences • Mining graphs • Cluster Analysis • K-Means, Bisecting K-means • SOM • DBSCAN • Hierarchical clustering • Web search • IR • Reputation ranking Single-side help sheet allowed.

3. Mining Association Rules Two-step approach: • Frequent Itemset Generation • Generate all itemsets whose support  minsup (these itemsets are called frequent itemsets) • Rule Generation • Generate high confidence rules from each frequent itemset • The computational requirements for frequent itemset generation are more expensive than those of rule generation. Candidate itemsets are generated and then tested against the database to see whether they are frequent. • Apriori principle: • If an itemset is frequent, then all of its subsets must also be frequent • Apriori principle conversely said: • If an itemset is infrequent, then all of its supersets must be infrequent too.

4. Illustrating Apriori Principle Found to be Infrequent Pruned supersets

5. Apriori Algorithm • Method: • Let k=1 • Generate frequent itemsets of length 1 • Repeat until no new frequent itemsets are identified k=k+1 • Generate length k candidate itemsets from length k-1 frequent itemsets • Prune candidate itemsets containing subsets of length k-1 that are infrequent • Count the support of each candidate by scanning the DB and eliminate candidates that are infrequent, leaving only those that are frequent

6. Fk-1Fk-1 Method • Merge a pair of frequent (k-1)­itemsets only if their first k-2 items are identical. • E.g. frequent itemsets {Bread, Diapers} and {Bread, Milk} are merged to form a candidate 3­itemset {Bread, Diapers, Milk}.

7. Fk-1Fk-1 • Completeness • We don’t merge {Beer, Diapers} with {Diapers, Milk} because the first item in both itemsets is different. • Do we loose {Beer, Diapers, Milk}? • Prunning • Before checking a candidate against the DB, a candidate pruning step is needed to ensure that the remaining subsets of k-1 elements are frequent. • Counting • Finally, the surviving candidates are tested (counted) on the DB.

8. Rule Generation • Computing the confidence of an association rule does not require additional scans of the transactions. • Consider {1, 2}{3}. • The rule confidence is  ({1, 2, 3}) /  ({1, 2}) • Because {1, 2, 3} is frequent, the anti­monotone property of support ensures that {1, 2} must be frequent, too, and we do know the supports of frequent itemsets. • Initially, all the high­confidence rules that have only one item in the rule consequent are extracted. • These rules are then used to generate new candidate rules. • For example, if • {acd}  {b} and {abd}  {c} are high­confidence rules, then the candidate rule {ad}  {bc} is generated by merging the consequents of both rules. • Then the candidate rules are checked for confidence.

9. Other concepts and algorithms • FP-Tree/FP-Growth See corresponding slide set and Assignment 2 solution. • Maximal Frequent Itemsets • Closed Itemset • Interest factor • Mining Sequences

10. Maximal Frequent Itemsets An itemset is maximal frequent if none of its immediate supersets is frequent Maximal Itemsets Maximal frequent itemsets form the smallest set of itemsets from which all frequent itemsets can be derived. Infrequent Itemsets Border

11. Closed Itemsets • Despite providing a compact representation, maximal frequent itemsets do not contain the support information of their subsets. • An additional pass over the data set is needed to determine the support counts of the non­maximal frequent itemsets. • It might be desirable to have a minimal representation of frequent itemsets that preserves the support information. • Such representation is the set of the closed frequent itemsets. • An itemset is closed if none of its immediate supersets has the same support as the itemset. • An itemset is a closed frequent itemset if it is closed and its support is greater than or equal to minsup.

12. Maximal vs Closed Frequent Itemsets Closed but not maximal Transaction Ids Minimum support = 2 Closed and maximal # Closed = 9 # Maximal = 4 Not supported by any transactions

13. ACD Deriving Frequent Itemsets From Closed Frequent Itemsets • E.g., consider the frequent itemset {a, d}. Because the itemset is not closed, its support count must be identical to one of its immediate supersets. • The key is to determine which superset among {a, b, d}, {a, c, d}, or {a, d, e} has exactly the same support count as {a, d}. • By the Apriori principle the support for {a, d} must be equal to the largest support among its supersets. • So, the support for {a, d} must be identical to the support for {a, c, d}. null 124 123 1234 245 345 A B C D E 12 124 24 123 4 2 3 24 34 45 AB AC AD AE BC BD BE CD CE DE 12 24 2 2 4 4 3 4 ABD ABE ACE ADE BCD BCE BDE CDE ABC 4 2 ABCD ABCE ABDE ACDE BCDE ABCDE

14. Support counting using closed frequent itemsets Let C denote the set of closed frequent itemsets Let kmax denote the maximum length of closed frequent itemsets Fkmax ={f | fC, | f | = kmax } {Frequent itemsets of size kmax} fork = kmax – 1 downto 1 do Set Fk to be all sub-itemsets of length k from the frequent itemsets in Fk+1 for eachfFkdo iffCthen f.support = max{f’.support | f’Fk+1, f f’} end if end for end for

15. f11: support of X and Yf10: support of X and Yf01: support of X and Yf00: support of X and Y Contingency Table • Given a rule X  Y, the information needed to compute rule interestingness can be obtained from a contingency table Contingency table for X  Y

16. Pitfall of Confidence • Consider association rule:Tea  Coffee • Confidence= • P(Coffee,Tea)/P(Tea) = P(Coffee|Tea) = 150/200 = 0.75 (seems quite high) • But, P(Coffee) = 0.9 • Thus knowing that a person is a tea drinker actually decreases his/her probability of being a coffee drinker from 90% to 75%! • Although confidence is high, rule is misleading In fact P(Coffee|Tea) = P(Coffee, Tea)/P(Tea) = 750/900 = 0.83

17. Interest Factor • Measure that takes into account statistical dependence • f11/N is an estimate for the joint probability P(A,B) • f1+ /N and f+1 /N are the estimates for P(A) and P(B), respectively. • If A and B are statistically independent, • then P(A,B)=P(A)×P(B), thus the Interest is 1.

18. Example: Interest Association Rule: Tea  Coffee Interest = 150*1100 / (200*900)= 0.92 (< 1, therefore they are negatively correlated)

19. Cross­support patterns • They are patterns that relate a high­frequency item such as milk to a low­frequency item such as caviar. • Likely to be spurious because their correlations tend to be weak. • E.g. confidence of {caviar}{milk} is likely to be high, but still the pattern is spurious, since there isn’t probably any correlation between caviar and milk. • Observation: On the other hand, the confidence of {milk}{caviar} is very low. • Cross­support patterns can be detected and eliminated by examining the lowest confidence rule that can be extracted from a given itemset. • Such confidence should be above certain level for the pattern to not be cross-support one.

20. Finding lowest confidence • Recall the anti­monotone property of confidence: conf( {i1 ,i2}{i3,i4,…,ik} )  conf( {i1 ,i2 ,i3}{i4,…,ik} ) • This property suggests that confidence never increases as we shift more items from the left­ to the right­hand side of an association rule. • Hence, the lowest confidence rule that can be extracted from a frequent itemset contains only one item on its left­hand side.

21. Finding lowest confidence • Given a frequent itemset {i1,i2,i3,i4,…,ik}, the rule {ij}{i1 ,i2 ,i3, ij-1, ij+1, i4,…,ik} has the lowest confidence if s(ij) = max {s(i1), s(i2),…,s(ik)} • This follows directly from the definition of confidence as the ratio between the rule's support and the support of the rule antecedent.

22. Finding lowest confidence • Summarizing, the lowest confidence attainable from a frequent itemset {i1,i2,i3,i4,…,ik}, is • This is also known as the h-confidence measure or all-confidence measure. • Cross­support patterns can be eliminated by ensuring that the h­confidence values for the patterns exceed some user specified threshold hc. • h-confidence is anti­monotone, i.e., • h­confidence({i1,i2,…, ik})  h­confidence({i1,i2,…, ik+1 }) • and thus can be incorporated directly into the mining algorithm.

23. Examples of Sequence • Web sequence:  {Homepage} {Electronics} {Digital Cameras} {Canon Digital Camera} {Shopping Cart} {Order Confirmation} {Return to Shopping}  • Purchase history of a given customer {Java in a Nutshell, Intro to Servlets} {EJB Patterns},… • Sequence of classes taken by a computer science major:  {Algorithms and Data Structures, Introduction to Operating Systems} {Database Systems, Computer Architecture} {Computer Networks, Software Engineering} {Computer Graphics, Parallel Programming} …

24. Element (Transaction) Event (Item) E1E2 E1E3 E2 E3E4 E2 Sequence Formal Definition of a Sequence • A sequence is an ordered list of elements (transactions) s = < e1 e2 e3 … > • Each element contains a collection of events (items) ei = {i1, i2, …, ik} • Each element is attributed to a specific time or location • A k-sequence is a sequence that contains k events (items)

25. Formal Definition of a Subsequence • A sequence  a1 a2 … an  is contained in another sequence  b1 b2 … bm  (m ≥ n) if there exist integers i1 < i2 < … < in such that a1 bi1 , a2bi2, …, anbin • Support of a subsequence w is the fraction of data sequences that contain w • A sequential pattern is a frequent subsequence (i.e., a subsequence whose support is ≥ minsup)

26. APRIORI-like Algorithm • Make the first pass over the sequence database to yield all the 1-element frequent sequences • Repeat until no new frequent sequences are found Candidate Generation: • Merge pairs of frequent subsequences found in the (k-1)th pass to generate candidate sequences that contain k items Candidate Pruning: • Prune candidate k-sequences that contain infrequent (k-1)-subsequences Support Counting: • Make a new pass over the sequence database to find the support for these candidate sequences • Eliminate candidate k-sequences whose actual support is less than minsup

27. Candidate Generation • Base case (k=2): • Merging two frequent 1-sequences <{i1}> and <{i2}> will produce four candidate 2-sequences: • <{i1}, {i2}>, <{i2}, {i1}>, <{i1,i2}>, <{i2,i1}> • General case (k>2): • A frequent (k-1)-sequence w1 is merged with another frequent (k-1)-sequence w2 to produce a candidate k-sequence if the subsequence obtained by removing the first event in w1 is the same as the subsequence obtained by removing the last event in w2 • The resulting candidate after merging is given by the sequence w1 extended with the last event of w2. • If the last two events in w2 belong to the same element, then the last event in w2 becomes part of the last element in w1 • Otherwise, the last event in w2 becomes a separate element appended to the end of w1

28. Candidate Generation Examples • Merging the sequences w1=<{1} {2 3} {4}> and w2 =<{2 3} {4 5}> will produce the candidate sequence < {1} {2 3} {4 5}> because the last two events in w2(4 and 5) belong to the same element • Merging the sequences w1=<{1} {2 3} {4}> and w2 =<{2 3} {4} {5}> will produce the candidate sequence < {1} {2 3} {4} {5}> because the last two events in w2(4 and 5) do not belong to the same element • Finally, the sequences <{1}{2}{3}> and <{1}{2, 5}> don’t have to be merged (Why?) • Because removing the first event from the first sequence doesn’t give the same subsequence as removing the last event from the second sequence. • If <{1}{2,5}{3}> is a viable candidate, it will be generated by merging a different pair of sequences, <{1}{2,5}> and <{2,5}{3}>.

29. Example Frequent 3-sequences Candidate < {1} {2} {3} > Generation < {1} {2 5} > < {1} {5} {3} > Candidate < {1} {2} {3} {4} > < {2} {3} {4} > Pruning < {1} {2 5} {3} > < {2 5} {3} > < {1} {5} {3 4} > < {3} {4} {5} > < {2} {3} {4} {5} > < {5} {3 4} > < {1} {2 5} {3} > < {2 5} {3 4} >

30. {A B} {C} {D E} <= max-gap <= max-span Timing Constraints max-gap = 2, max-span= 4

31. Mining Sequential Patterns with Timing Constraints • Approach 1: • Mine sequential patterns without timing constraints • Postprocess the discovered patterns • Approach 2: • Modify algorithm to directly prune candidates that violate timing constraints • Question: • Does APRIORI principle still hold?

32. APRIORI Principle for Sequence Data Suppose: max-gap = 1 max-span = 5 <{2} {5}> support = 40% but <{2} {3} {5}> support = 60% !! (APRIORI doesn’t hold) Problem exists because of max-gap constraint This problem can avoided by using the concept of a contiguous subsequence.

33. Contiguous Subsequences • s is a contiguous subsequence of w = <e1, e2 ,…, ek>if any of the following conditions holds: • s is obtained from w by deleting an item from either e1 or ek • s is obtained from w by deleting an item from any element ei that contains at least 2 items • s is a contiguous subsequence of s’ and s’ is a contiguous subsequence of w (recursive definition) • Examples: s = < {1} {2} > • is a contiguous subsequence of < {1} {2 3}>, < {1 2} {2} {3}>, and < {3 4} {1 2} {2 3} {4} > • is not a contiguous subsequence of< {1} {3} {2}> and < {2} {1} {3} {2}>

34. Modified Candidate Pruning Step • Modified APRIORI Principle • If a k-sequence is frequent, then all of its contiguous (k-1)-subsequences must also be frequent • Candidate generation doesn’t change. Only pruning changes. • Without maxgap constraint: • A candidate k-sequence is pruned if at least one of its (k-1)-subsequences is infrequent • With maxgap constraint: • A candidate k-sequence is pruned if at least one of its contiguous(k-1)-subsequences is infrequent

35. Inter-cluster distances are maximized Intra-cluster distances are minimized Cluster Analysis • Find groups of objects such that the objects in a group will be similar (or related) to one another and different from (or unrelated to) the objects in other groups

36. K-means Clustering • Partitional clustering approach • Each cluster is associated with a centroid (center point) • typically the mean of the points in the cluster. • Each point is assigned to the cluster with the closest centroid • Number of clusters, K, must be specified • The basic algorithm is very simple

37. Importance of Choosing Initial Centroids

38. Importance of Choosing Initial Centroids

39. Solutions to Initial Centroids Problem • Multiple runs • Helps, but probability is not on your side • Bisecting K-means • Not as susceptible to initialization issues

40. Bisecting K­means Straightforward extension of the basic K­means algorithm. Simple idea: To obtain K clusters, split the set of points into two clusters, select one of these clusters to split, and so on, until K clusters have been produced. Algorithm Initialize the list of clusters to contain the cluster consisting of all points. repeat Remove a cluster from the list of clusters. //Perform several ``trial'' bisections of the chosen cluster. fori = 1 to number of trials do Bisect the selected cluster using basic K­means (i.e. 2-means). end for Select the two clusters from the bisection with the lowest total SSE. Add these two clusters to the list of clusters. until the list of clusters contains K clusters.

41. Bisecting K-means Example

42. Limitations of K-means • K-means has problems when clusters are of differing • Sizes • Densities • Non-globular shapes • K-means has problems when the data contains outliers.

43. Exercise • For each figure, could you use K-means to find the patterns represented by the nose, eyes, and mouth? • Only for (b) and (d). • For (b), K-means would find the nose, eyes, and mouth, but the lower density points would also be included. • For (d), K-means would find the nose, eyes, and mouth straightforwardly as long as the number of clusters was set to 4. • What limitation does clustering have in detecting all the patterns formed by the points in figure c? • Clustering techniques can only find patterns of points, not of empty spaces.

44. Agglomerative Clustering Algorithm Compute the proximity matrix Let each data point be a cluster Repeat Merge the two closest clusters Update the proximity matrix Until only a single cluster remains • Key operation is the computation of the proximity of two clusters • Different approaches to defining the distance between clusters distinguish the different algorithms

45. Cluster Similarity: MIN • Similarity of two clusters is based on the two most similar (closest) points in the different clusters • Determined by one pair of points

46. 5 1 3 5 2 1 2 3 6 4 4 Hierarchical Clustering: MIN Nested Clusters Dendrogram

47. Two Clusters Strength of MIN Original Points Can handle non-globular shapes

48. Original Points Four clusters Three clusters: The yellow points got wrongly merged with the red ones, as opposed to the green one. Limitations of MIN Sensitive to noise and outliers

49. Cluster Similarity: MAX • Similarity of two clusters is based on the two least similar (most distant) points in the different clusters • Determined by all pairs of points in the two clusters

50. 4 1 2 5 5 2 3 6 3 1 4 Hierarchical Clustering: MAX Nested Clusters Dendrogram