Association Rule Mining: Apriori Algorithm

1. Association Rule Mining:Apriori Algorithm CIT366: Data Mining & Data Warehousing Bajuna Salehe The Institute of Finance Management: Computing and IT Dept.

2. Brief About Association Rule Mining • The results of Market Basket Analysis allowed companies to more fully understand purchasing behaviour and, as a result better target market audiences. • Association mining is user-centric as the objective is the elicitation of useful (or interesting) rules from which new knowledge can be derived.

3. Brief About Association Rule Mining • Association mining applications have been applied to many different domains including market basket and risk analysis in commercial environments, epidemiology, clinical medicine, fluid dynamics, astrophysics, crime prevention, and counter-terrorism—all areas in which the relationship between objects can provide useful knowledge.

4. Example of Association Rule • For example, an insurance company, by finding a strong correlation between two policies A and B, of the form A -> B, indicating that customers that held policy A were also likely to hold policy B, could more efficiently target the marketing of policy B through marketing to those clients that held policy A but not B.

5. Brief About Association Rule Mining • Association mining analysis is a two part process. • First, the identification of sets of items or itemsets within the dataset. • Second, the subsequent derivation of inferences from these itemsets

6. Why Use Support and Confidence? • Support reflects the statistical significance of a rule. Rules that have very low support are rarely observed, and thus, are more likely to occur by chance. For example, the rule A → B may not be significant because both items are present together only once in the previous Table in the last week lecture.

7. Why Use Support and Confidence? • Additionally, low support rules may not be actionable from a marketing perspective because it is not profitable to promote items that are seldom bought together by customers. • For these reasons, support is often used as a filter to eliminate uninteresting rules.

8. Why Use Support and Confidence? • Confidence is another useful metric because it measures how reliable is the inference made by a rule. • For a given rule A→ B , the higher the confidence, the more likely it is for itemset B to be present in transactions that contain A. In a sense, confidence provides an estimate of the conditional probability for B given A.

9. Causality & Association Rule • Finally, it is worth noting that the inference made by an association rule does not necessarily imply causality. • Instead, the implication indicates a strong co-occurrence relationship between items in the antecedent and consequent of the rule.

10. Causality & A. Rule • Causality, on the other hand, requires a distinction between the causal and effect attributes of the data and typically involves relationships occurring over time (e.g., ozone depletion leads to global warming).

11. More About Support and Confidence • The support for the following candidate rules • {Bread, Cheese} → {Milk}, {Bread,Milk} → {Cheese}, {Cheese,Milk} → {Bread}, {Bread} → {Cheese,Milk}, {Milk} → {Bread,Cheese}, {Cheese} → {Bread,Milk} are identical since they correspond to the same itemset, {Bread, Cheese, Milk}. • If the itemset is infrequent, then all six candidate rules can be immediately pruned without having to compute their confidence values.

12. More About Support and Confidence • Therefore, a common strategy adopted by many association rule mining algorithms is to decompose the problem into two major subtasks: • Frequent Itemset Generation. Find all itemsets that satisfy the minsupthreshold. These itemsets are called frequent itemsets. • Rule Generation. Extract high confidence association rules from the frequent • Itemsets found in the previous step. These rules are called strong rules.

13. Frequent Itemset Generation • A lattice structure can be used to enumerate the list of possible itemsets. • For example, the figure below illustrates all itemsets derivable from the set {A,B,C,D}.

14. Frequent Itemset Generation

15. Frequent Itemset Generation • In general, a data set that contains d items may generate up to 2d (raise to power ‘d’) − 1 possible itemsets, excluding the null set. • Because d can be very large in many commercial databases, frequent itemset generation is an exponentially expensive task.

16. Frequent Itemset Generation • A naıve approach for finding frequent itemsets is to determine the support count for every candidate itemset in the lattice structure. • To do this, we need to match each candidate against every transaction.

17. Apriori Algorithm • This algorithm is among of the algorithms that are grouped into candidate generation algorithms, used to identify candidate itemsets. • The common data structure that is used in apriori algorithm is tree data structures. • Two common type of tree data structures used in apriori are:- • Enumeration Set Tree • Prefix Tree

18. Data Structure for Apriori Algorithm

19. Apriori Algorithm • Frequent itemsets (also called as large itemsets), are those itemsets whose support is greater than minSupp (Minimum Support). • The apriori property (downward closure property) says that any subsets of any frequent itemset are also frequent itemsets • The use of support for pruning candidate itemsets is guided by the following principle (Apriori Principle). • If an itemset is frequent, then all of its subsets

20. Reminder: Steps of Association Rule Mining • The major steps in association rule mining are: • Frequent Itemset generation • Rules derivation

21. Apriori Algorithm • Any subset of a frequent itemset must be frequent • If {beer, nappy, nuts} is frequent, so is {beer, nappy} • Every transaction having {beer, nappy, nuts} also contains {beer, nappy} • Apriori pruning principle: If there is any itemset which is infrequent, its superset should not be generated/tested!

22. Apriori Algorithm • The APRIORI algorithm uses the downward closure property, to prune unnecessary branches for further consideration. It needs two parameters, minSupp and minConf. The minSupp is used for generating frequent itemsets and minConf is used for rule derivation.

23. The Apriori Algorithm: An Example Database TDB L1 C1 1st scan C2 C2 L2 2nd scan L3 C3 3rd scan

24. Important Details Of The Apriori Algorithm • There are two crucial questions in implementing the Apriori algorithm: • How to generate candidates? • How to count supports of candidates?

25. Generating Candidates • There are 2 steps to generating candidates: • Step 1: Self-joining Lk • Step 2: Pruning • Example of Candidate-generation • L3={abc, abd, acd, ace, bcd} • Self-joining: L3*L3 • abcd from abc and abd • acde from acd and ace • Pruning: • acde is removed because ade is not in L3 • C4={abcd}

26. Apriori Algorithm • k = 1. • Fk = { i | i ∈ I ∧ σ({i})/ N ≥ minsup}. {Find all frequent 1-itemsets} • repeat • k = k + 1. • Ck = apriori-gen(Fk−1). {Generate candidate itemsets} • for each transaction t ∈ T do • Ct = subset(Ck, t). {Identify all candidates that belong to t} • for each candidate itemset c ∈ Ct do • σ(c) = σ(c) + 1. {Increment support count} • end for • end for • Fk = { c | c ∈ Ck ∧ σ(c) /N ≥ minsup}. {Extract the frequent k-itemsets} • until Fk = ∅ • Result = _Fk

27. How to Count Supports Of Candidates? • Why counting supports of candidates a problem? • The total number of candidates can be huge • One transaction may contain many candidates • Method: • Candidate itemsets are stored in a hash-tree • Leaf node of hash-tree contains a list of itemsets and counts • Interior node contains a hash table • Subset function: finds all the candidates contained in a transaction

28. Generating Association Rules • Once all frequent itemsets have been found association rules can be generated • Strong association rules from a frequent itemset are generated by calculating the confidence in each possible rule arising from that itemset and testing it against a minimum confidence threshold

29. Example

30. Example

31. Challenges Of Frequent Pattern Mining • Challenges • Multiple scans of transaction database • Huge number of candidates • Tedious workload of support counting for candidates • Improving Apriori: general ideas • Reduce passes of transaction database scans • Shrink number of candidates • Facilitate support counting of candida

32. Bottleneck Of Frequent-Pattern Mining • Multiple database scans are costly • Mining long patterns needs many passes of scanning and generates lots of candidates • To find frequent itemset i1i2…i100 • # of scans: 100 • # of Candidates: + + … + = 2100-1 = 1.27*1030 • Bottleneck: candidate-generation-and-test

33. Mining Frequent Patterns Without Candidate Generation • Techniques for mining frequent itemsets which avoid candidate generation include: • FP-growth • Grow long patterns from short ones using local frequent items • ECLAT (Equivalence CLASS Transformation) algorithm • Uses a data representation in which transactions are associated with items, rather than the other way around (vertical data format) • These methods can be much faster than the Apriori algorithm