Classification and Prediction (cont.) Pertemuan 09

2. Classification and Prediction (cont.)Pertemuan 09 Matakuliah : M0614 / Data Mining & OLAP Tahun : Feb - 2010

3. Learning Outcomes Pada akhir pertemuan ini, diharapkan mahasiswa akan mampu : • Mahasiswa dapat menggunakan teknik analisis classification by decision tree induction, Bayesian classification, classification by back propagation, dan lazy learners pada data mining. (C3) 3

4. Acknowledgments These slides have been adapted from Han, J., Kamber, M., & Pei, Y. Data Mining: Concepts and Technique and Tan, P.-N., Steinbach, M., & Kumar, V. Introduction to Data Mining. Bina Nusantara

5. Rule-based classification Classification by back propagation Lazy learners (or learning from your neighbours) Outline Materi 5 Bina Nusantara

6. Rule-Based Classifier Classify records by using a collection of “if…then…” rules Rule: (Condition)  y where Condition is a conjunctions of attributes y is the class label LHS: rule antecedent or condition RHS: rule consequent Examples of classification rules: (Blood Type=Warm)  (Lay Eggs=Yes)  Birds (Taxable Income < 50K)  (Refund=Yes)  Evade=No

7. Rule-based Classifier (Example) R1: (Give Birth = no)  (Can Fly = yes)  Birds R2: (Give Birth = no)  (Live in Water = yes)  Fishes R3: (Give Birth = yes)  (Blood Type = warm)  Mammals R4: (Give Birth = no)  (Can Fly = no)  Reptiles R5: (Live in Water = sometimes)  Amphibians

8. Example: Rule extraction from our buys_computer decision-tree IF age = young AND student = no THEN buys_computer = no IF age = young AND student = yes THEN buys_computer = yes IF age = mid-age THEN buys_computer = yes IF age = old AND credit_rating = excellent THEN buys_computer = yes IF age = young AND credit_rating = fair THEN buys_computer = no Rule Extraction from a Decision Tree age? <=30 >40 31..40 student? credit rating? yes excellent fair no yes no yes no yes • Rules are easier to understand than large trees • One rule is created for each path from the root to a leaf • Each attribute-value pair along a path forms a conjunction: the leaf holds the class prediction • Rules are mutually exclusive and exhaustive September 14, 2014 Data Mining: Concepts and Techniques 8

9. Application of Rule-Based Classifier A rule rcovers an instance x if the attributes of the instance satisfy the condition of the rule R1: (Give Birth = no)  (Can Fly = yes)  Birds R2: (Give Birth = no)  (Live in Water = yes)  Fishes R3: (Give Birth = yes)  (Blood Type = warm)  Mammals R4: (Give Birth = no)  (Can Fly = no)  Reptiles R5: (Live in Water = sometimes)  Amphibians The rule R1 covers a hawk => Bird The rule R3 covers the grizzly bear => Mammal

10. How does Rule-based Classifier Work? R1: (Give Birth = no)  (Can Fly = yes)  Birds R2: (Give Birth = no)  (Live in Water = yes)  Fishes R3: (Give Birth = yes)  (Blood Type = warm)  Mammals R4: (Give Birth = no)  (Can Fly = no)  Reptiles R5: (Live in Water = sometimes)  Amphibians A lemur triggers rule R3, so it is classified as a mammal A turtle triggers both R4 and R5 A dogfish shark triggers none of the rules

11. Rule Coverage and Accuracy Coverage of a rule: Fraction of records that satisfy the antecedent of a rule Accuracy of a rule: Fraction of records that satisfy both the antecedent and consequent of a rule (Status=Single)  No Coverage = 40%, Accuracy = 50%

12. Characteristics of Rule-Based Classifier Mutually exclusive rules Classifier contains mutually exclusive rules if the rules are independent of each other Every record is covered by at most one rule Exhaustive rules Classifier has exhaustive coverage if it accounts for every possible combination of attribute values Each record is covered by at least one rule

13. From Decision Trees To Rules Rules are mutually exclusive and exhaustive Rule set contains as much information as the tree

14. Rules Can Be Simplified Initial Rule: (Refund=No)  (Status=Married)  No Simplified Rule: (Status=Married)  No

15. Effect of Rule Simplification Rules are no longer mutually exclusive A record may trigger more than one rule Solution? Ordered rule set Unordered rule set – use voting schemes Rules are no longer exhaustive A record may not trigger any rules Solution? Use a default class

16. Ordered Rule Set Rules are rank ordered according to their priority An ordered rule set is known as a decision list When a test record is presented to the classifier It is assigned to the class label of the highest ranked rule it has triggered If none of the rules fired, it is assigned to the default class R1: (Give Birth = no)  (Can Fly = yes)  Birds R2: (Give Birth = no)  (Live in Water = yes)  Fishes R3: (Give Birth = yes)  (Blood Type = warm)  Mammals R4: (Give Birth = no)  (Can Fly = no)  Reptiles R5: (Live in Water = sometimes)  Amphibians

17. Rule Ordering Schemes Rule-based ordering Individual rules are ranked based on their quality Class-based ordering Rules that belong to the same class appear together

18. Building Classification Rules Direct Method: Extract rules directly from data e.g.: RIPPER, CN2, Holte’s 1R Indirect Method: Extract rules from other classification models (e.g. decision trees, neural networks, etc). e.g: C4.5rules

19. Rule Induction: Sequential Covering Method Sequential covering algorithm: Extracts rules directly from training data Typical sequential covering algorithms: FOIL, AQ, CN2, RIPPER Rules are learned sequentially, each for a given class Ci will cover many tuples of Ci but none (or few) of the tuples of other classes Steps: Rules are learned one at a time Each time a rule is learned, the tuples covered by the rules are removed The process repeats on the remaining tuples unless termination condition, e.g., when no more training examples or when the quality of a rule returned is below a user-specified threshold Comp. w. decision-tree induction: learning a set of rules simultaneously September 14, 2014 Data Mining: Concepts and Techniques 19

20. Sequential Covering Algorithm while (enough target tuples left) generate a rule remove positive target tuples satisfying this rule Examples covered by Rule 2 Examples covered by Rule 1 Examples covered by Rule 3 Positive examples September 14, 2014 Data Mining: Concepts and Techniques 20

21. How to Learn-One-Rule? Star with the most general rule possible: condition = empty Adding new attributes by adopting a greedy depth-first strategy Picks the one that most improves the rule quality Rule-Quality measures: consider both coverage and accuracy Foil-gain (in FOIL & RIPPER): assesses info_gain by extending condition It favors rules that have high accuracy and cover many positive tuples Rule pruning based on an independent set of test tuples Pos/neg are # of positive/negative tuples covered by R. If FOIL_Prune is higher for the pruned version of R, prune R September 14, 2014 Data Mining: Concepts and Techniques 21

22. Rule Generation To generate a rule while(true) find the best predicate p if foil-gain(p) > threshold then add p to current rule else break A3=1 A3=1&&A1=2 A3=1&&A1=2 &&A8=5 Positive examples Negative examples September 14, 2014 Data Mining: Concepts and Techniques 22

23. Aspects of Sequential Covering Rule Growing Instance Elimination Rule Evaluation Stopping Criterion Rule Pruning

24. Rule Growing Two common strategies

25. Rule Growing (Examples) CN2 Algorithm: Start from an empty conjunct: {} Add conjuncts that minimizes the entropy measure: {A}, {A,B}, … Determine the rule consequent by taking majority class of instances covered by the rule RIPPER Algorithm: Start from an empty rule: {} => class Add conjuncts that maximizes FOIL’s information gain measure: R0: {} => class (initial rule) R1: {A} => class (rule after adding conjunct) Gain(R0, R1) = t [ log (p1/(p1+n1)) – log (p0/(p0 + n0)) ] where t: number of positive instances covered by both R0 and R1 p0: number of positive instances covered by R0 n0: number of negative instances covered by R0 p1: number of positive instances covered by R1 n1: number of negative instances covered by R1

26. Rule Evaluation Metrics: Accuracy Laplace M-estimate n : Number of instances covered by rule nc : Number of instances covered by rule k : Number of classes p : Prior probability

27. Stopping Criterion and Rule Pruning Stopping criterion Compute the gain If gain is not significant, discard the new rule Rule Pruning Similar to post-pruning of decision trees Reduced Error Pruning: Remove one of the conjuncts in the rule Compare error rate on validation set before and after pruning If error improves, prune the conjunct

28. Summary of Direct Method Grow a single rule Remove Instances from rule Prune the rule (if necessary) Add rule to Current Rule Set Repeat

29. Direct Method: RIPPER For 2-class problem, choose one of the classes as positive class, and the other as negative class Learn rules for positive class Negative class will be default class For multi-class problem Order the classes according to increasing class prevalence (fraction of instances that belong to a particular class) Learn the rule set for smallest class first, treat the rest as negative class Repeat with next smallest class as positive class

30. Direct Method: RIPPER Growing a rule: Start from empty rule Add conjuncts as long as they improve FOIL’s information gain Stop when rule no longer covers negative examples Prune the rule immediately using incremental reduced error pruning Measure for pruning: v = (p-n)/(p+n) p: number of positive examples covered by the rule in the validation set n: number of negative examples covered by the rule in the validation set Pruning method: delete any final sequence of conditions that maximizes v

31. Direct Method: RIPPER Building a Rule Set: Use sequential covering algorithm Finds the best rule that covers the current set of positive examples Eliminate both positive and negative examples covered by the rule Each time a rule is added to the rule set, compute the new description length stop adding new rules when the new description length is d bits longer than the smallest description length obtained so far

32. Direct Method: RIPPER Optimize the rule set: For each rule r in the rule set R Consider 2 alternative rules: Replacement rule (r*): grow new rule from scratch Revised rule(r’): add conjuncts to extend the rule r Compare the rule set for r against the rule set for r* and r’ Choose rule set that minimizes MDL principle Repeat rule generation and rule optimization for the remaining positive examples

33. Indirect Methods

34. Indirect Method: C4.5rules Extract rules from an unpruned decision tree For each rule, r: A  y, consider an alternative rule r’: A’  y where A’ is obtained by removing one of the conjuncts in A Compare the pessimistic error rate for r against all r’s Prune if one of the r’s has lower pessimistic error rate Repeat until we can no longer improve generalization error

35. Indirect Method: C4.5rules Instead of ordering the rules, order subsets of rules (class ordering) Each subset is a collection of rules with the same rule consequent (class) Compute description length of each subset Description length = L(error) + g L(model) g is a parameter that takes into account the presence of redundant attributes in a rule set (default value = 0.5)

36. Example

37. C4.5 versus C4.5rules versus RIPPER C4.5rules: (Give Birth=No, Can Fly=Yes)  Birds (Give Birth=No, Live in Water=Yes)  Fishes (Give Birth=Yes)  Mammals (Give Birth=No, Can Fly=No, Live in Water=No)  Reptiles ( )  Amphibians RIPPER: (Live in Water=Yes)  Fishes (Have Legs=No)  Reptiles (Give Birth=No, Can Fly=No, Live In Water=No)  Reptiles (Can Fly=Yes,Give Birth=No)  Birds ()  Mammals

38. C4.5 versus C4.5rules versus RIPPER C4.5 and C4.5rules: RIPPER:

39. Advantages of Rule-Based Classifiers As highly expressive as decision trees Easy to interpret Easy to generate Can classify new instances rapidly Performance comparable to decision trees

40. Classification by Backpropagation Backpropagation: A neural network learning algorithm Started by psychologists and neurobiologists to develop and test computational analogues of neurons A neural network: A set of connected input/output units where each connection has a weight associated with it During the learning phase, the network learns by adjusting the weights so as to be able to predict the correct class label of the input tuples Also referred to as connectionist learning due to the connections between units September 14, 2014 Data Mining: Concepts and Techniques 40

41. Classification by Backpropagation Backpropagation: A neural network learning algorithm Started by psychologists and neurobiologists to develop and test computational analogues of neurons A neural network: A set of connected input/output units where each connection has a weight associated with it During the learning phase, the network learns by adjusting the weights so as to be able to predict the correct class label of the input tuples Also referred to as connectionist learning due to the connections between units September 14, 2014 Data Mining: Concepts and Techniques 41

42. Neural Network as a Classifier Weakness Long training time Require a number of parameters typically best determined empirically, e.g., the network topology or “structure.” Poor interpretability: Difficult to interpret the symbolic meaning behind the learned weights and of “hidden units” in the network Strength High tolerance to noisy data Ability to classify untrained patterns Well-suited for continuous-valued inputs and outputs Successful on a wide array of real-world data Algorithms are inherently parallel Techniques have recently been developed for the extraction of rules from trained neural networks September 14, 2014 Data Mining: Concepts and Techniques 42

43. A Neuron (= a perceptron) The n-dimensional input vector x is mapped into variable y by means of the scalar product and a nonlinear function mapping - mk x0 w0 x1 w1 f å output y xn wn Input vector x weight vector w weighted sum Activation function September 14, 2014 Data Mining: Concepts and Techniques 43

44. A Multi-Layer Feed-Forward Neural Network Output vector Output layer Hidden layer wij Input layer Input vector: X September 14, 2014 Data Mining: Concepts and Techniques 44

45. How A Multi-Layer Neural Network Works? The inputs to the network correspond to the attributes measured for each training tuple Inputs are fed simultaneously into the units making up the input layer They are then weighted and fed simultaneously to a hidden layer The number of hidden layers is arbitrary, although usually only one The weighted outputs of the last hidden layer are input to units making up the output layer, which emits the network's prediction The network is feed-forward in that none of the weights cycles back to an input unit or to an output unit of a previous layer From a statistical point of view, networks perform nonlinear regression: Given enough hidden units and enough training samples, they can closely approximate any function September 14, 2014 Data Mining: Concepts and Techniques 45

46. Defining a Network Topology First decide the network topology: # of units in the input layer, # of hidden layers (if > 1), # of units in each hidden layer, and # of units in the output layer Normalizing the input values for each attribute measured in the training tuples to [0.0—1.0] One input unit per domain value, each initialized to 0 Output, if for classification and more than two classes, one output unit per class is used Once a network has been trained and its accuracy is unacceptable, repeat the training process with a different network topology or a different set of initial weights September 14, 2014 Data Mining: Concepts and Techniques 46

47. Backpropagation Iteratively process a set of training tuples & compare the network's prediction with the actual known target value For each training tuple, the weights are modified to minimize the mean squared error between the network's prediction and the actual target value Modifications are made in the “backwards” direction: from the output layer, through each hidden layer down to the first hidden layer, hence “backpropagation” Steps Initialize weights (to small random #s) and biases in the network Propagate the inputs forward (by applying activation function) Backpropagate the error (by updating weights and biases) Terminating condition (when error is very small, etc.) September 14, 2014 Data Mining: Concepts and Techniques 47

48. Lazy vs. Eager Learning Lazy vs. eager learning Lazy learning (e.g., instance-based learning): Simply stores training data (or only minor processing) and waits until it is given a test tuple Eager learning (the above discussed methods): Given a set of training set, constructs a classification model before receiving new (e.g., test) data to classify Lazy: less time in training but more time in predicting Accuracy Lazy method effectively uses a richer hypothesis space since it uses many local linear functions to form its implicit global approximation to the target function Eager: must commit to a single hypothesis that covers the entire instance space September 14, 2014 Data Mining: Concepts and Techniques 48

49. Lazy Learner: Instance-Based Methods Instance-based learning: Store training examples and delay the processing (“lazy evaluation”) until a new instance must be classified Typical approaches k-nearest neighbor approach Instances represented as points in a Euclidean space. Locally weighted regression Constructs local approximation Case-based reasoning Uses symbolic representations and knowledge-based inference September 14, 2014 Data Mining: Concepts and Techniques 49

50. Nearest Neighbor Classifiers Basic idea: If it walks like a duck, quacks like a duck, then it’s probably a duck Compute Distance Test Record Training Records Choose k of the “nearest” records