Decision-Tree Induction & Decision-Rule Induction

1. Decision-Tree Induction & Decision-Rule Induction Evgueni Smirnov

2. Overview • Instances, Concepts, Languages • Types of Tasks • Decision Trees • Decision Rules • Evaluation Techniques • Intro to Weka

3. Instances, Concepts, Languages A concept is a set of objects in a world that are unified by a reason. A reason may be a similar appearance, structure or function. friendly robots Example. The set: {children, photos, cat, diplomas} can be viewed as a concept “Most important things to take out of your apartment when it catches fire”.

4. Li Instances, Concepts, Languages head = square body = round smiling = yes holding = flag color = yellow friendly robots

5. Lc M Li Instances, Concepts, Languages smiling = yes  friendly robots head = square body = round smiling = yes holding = flag color = yellow friendly robots

6. Lc M   I+:   Li I-: The Concept Learning Task <Li, Lc, M, <I+, I->>

7. Lc M Li The Clustering Task <Li, Lc, M, I>

8. Lc M Li The Association Learning Task <Li, Lc, M, I>

9. Decision Trees • Decision trees • Appropriate problems for decision trees • Entropy and Information Gain • The ID3 algorithm • Avoiding Overfitting via Pruning • Handling Continuous-Valued Attributes • Handling Missing Attribute Values

10. Outlook Sunny Overcast Rainy Humidity yes Windy High Normal False True no yes yes no Decision Trees • Definition: A decision tree is a tree s.t.: • Each internal node tests an attribute • Each branch corresponds to attribute value • Each leaf node assigns a classification

11. Data Set for Playing Tennis

12. Decision Tree For Playing Tennis Outlook Sunny Overcast Rainy Humidity yes Windy High Normal False True no yes yes no

13. When to Consider Decision Trees • Each instance consists of an attribute with discrete values (e.g. outlook/sunny, etc..) • The classification is over discrete values (e.g. yes/no ) • It is okay to have disjunctive descriptions – each path in the tree represents a disjunction of attribute combinations. Any Boolean function can be represented! • It is okay for the training data to contain errors – decision trees are robust to classification errors in the training data. • It is okay for the training data to contain missing values – decision trees can be used even if instances have missing attributes.

14. Outlook Sunny Overcast Rainy Humidity yes Windy High Normal False True no yes yes no Rules in Decision Trees If Outlook = Sunny & Humidity = High then Play = no If Outlook = Sunny & Humidity = Normal then Play = yes If Outlook = Overcast then Play = yes If Outlook = Rainy & Windy = False then Play = yes If Outlook = Rainy & Windy = True then Play = no

15. Decision Tree Induction Basic Algorithm: 1. A  the “best" decision attribute for a node N. 2. Assign A as decision attribute for the node N. 3. For each value of A, create new descendant of the node N. 4. Sort training examples to leaf nodes. 5. IF training examples perfectly classified, THEN STOP. ELSE iterate over new leaf nodes

16. Decision Tree Induction Outlook Sunny Rain Overcast ____________________________________ Outlook Temp Hum Wind Play ------------------------------------------------------- Sunny Hot High Weak no Sunny Hot High Strong no Sunny Mild High Weak no Sunny Cool Normal Weak yes Sunny Mild Normal Strong yes _____________________________________ Outlook Temp Hum Wind Play --------------------------------------------------------- Overcast Hot High Weak yes Overcast Cool Normal Strong yes _____________________________________ Outlook Temp Hum Wind Play --------------------------------------------------------- Rain Mild High Weak yes Rain Cool Normal Weak yes Rain Cool Normal Strong no Rain Mild Normal Weak yes Rain Mild High Strong no

17. Entropy Let S be a sample of training examples, and p+ is the proportion of positive examples in S and p- is the proportion of negative examples in S. Then:  entropy measures the impurity of S: E( S) = - p+ log2 p+ – p- log2p-

18. Entropy Example from the Dataset

19. Information Gain Information Gain is the expected reduction in entropy caused by partitioning the instances according to a given attribute. Gain(S, A) = E(S) - where SV = { sS | A(s) = V}

20. Example Outlook Sunny Rain Overcast ____________________________________ Outlook Temp Hum Wind Play ------------------------------------------------------- Sunny Hot High False No Sunny Hot High True No Sunny Mild High False No Sunny Cool Normal False Yes Sunny Mild Normal True Yes _____________________________________ Outlook Temp Hum Wind Play --------------------------------------------------------- Overcast Hot High Weak Yes Overcast Cool Normal Strong Yes _____________________________________ Outlook Temp Hum Windy Play --------------------------------------------------------- Rain Mild High False Yes Rain Cool Normal False Yes Rain Cool Normal True No Rain Mild Normal False Yes Rain Mild High True No Which attribute should be tested here? Gain (Ssunny , Humidity) = = .970 - (3/5) 0.0 - (2/5) 0.0 = .970 Gain (Ssunny , Temperature) = .970 - (2/5) 0.0 - (2/5) 1.0 - (1/5) 0.0 = .570 Gain (Ssunny , Wind) = .970 - (2/5) 1.0 - (3/5) .918 = .019

21. ID3 Algorithm Informally: • Determine the attribute with the highest information gain on the training set. • Use this attribute as the root, create a branch for each of the values the attribute can have. • For each branch, repeat the process with subset of the training set that is classified by that branch.

22. Hypothesis Space Search in ID3 • The hypothesis space is the set of all decision trees defined over the given set of attributes. • ID3’s hypothesis space is a compete space; i.e., the target description is there! • ID3 performs a simple-to-complex, hill climbing search through this space.

23. Hypothesis Space Search in ID3 • The evaluation function is the information gain. • ID3 maintains only a single current decision tree. • ID3 performs no backtracking in its search. • ID3 uses all training instances at each step of the search.

24. Overfitting Definition: Given a hypothesis space H, a hypothesis h  H is said to overfit the training data if there exists some hypothesis h’  H, such that h has smaller error that h’ over the training instances, but h’ has a smaller error that h over the entire distribution of instances.

25. Reasons for Overfitting Outlook sunny overcast rainy Humidity yes Windy high normal false true no yes yes no • Noisy training instances. Consider an noisy training example: • Outlook = Sunny; Temp = Hot; Humidity = Normal; Wind = True; PlayTennis= No • This instance affects the training instances: • Outlook = Sunny; Temp = Cool; Humidity = Normal; Wind = False; PlayTennis= Yes • Outlook = Sunny; Temp = Mild; Humidity = Normal; Wind = True; PlayTennis= Yes

26. Reasons for Overfitting Outlook sunny overcast rainy Humidity yes Windy high normal false true Windy no yes no false true Outlook = Sunny; Temp = Hot; Humidity = Normal; Wind = True; PlayTennis= No Outlook = Sunny; Temp = Cool; Humidity = Normal; Wind = False; PlayTennis= Yes Outlook = Sunny; Temp = Mild; Humidity = Normal; Wind = True; PlayTennis= Yes yes Temp mild high cool yes no ?

27. area with probably wrong predictions Reasons for Overfitting • Small number of instances are associated with leaf nodes. In this case it is possible that for coincidental regularities to occur that are unrelated to the actual target concept. - + + + - + - + - + - + - - + - - - - - - - - - - - -

28. Approaches to Avoiding Overfitting • Pre-pruning: stop growing the tree earlier, before it reaches the point where it perfectly classifies the training data • Post-pruning: Allow the tree to overfit the data, and then post-prune the tree.

29. Validation Set • Validation set is a set of instances used to evaluate the utility of nodes in decision trees. The validation set has to be chosen so that it is unlikely to suffer from same errors or fluctuations as the training set. • Usually before pruning the training data is split randomly into a growing set and a validation set.

30. Reduced-ErrorPruning Split data into growing and validation sets. Pruning a decision node d consists of: • removing the subtree rooted at d. • making d a leaf node. • assigning d the most common classification of the training instances associated with d. Outlook sunny overcast rainy Humidity yes Windy high normal false true no yes yes no 3 instances 2 instances Accuracy of the tree on the validation set is 90%.

31. Reduced-Error Pruning Split data into growing and validation sets. Pruning a decision node d consists of: • removing the subtree rooted at d. • making d a leaf node. • assigning d the most common classification of the training instances associated with d. Outlook sunny overcast rainy no yes Windy false true yes no Accuracy of the tree on the validation set is 92.4%.

32. Reduced-Error Pruning Split data into growing and validation sets. Pruning a decision node d consists of: • removing the subtree rooted at d. • making d a leaf node. • assigning d the most common classification of the training instances associated with d. Do until further pruning is harmful: • Evaluate impact on validation set of pruning each possible node (plus those below it). • Greedily remove the one that most improves validation set accuracy. Outlook sunny overcast rainy no yes Windy false true yes no Accuracy of the tree on the validation set is 92.4%.

33. Reduced Error Pruning Example

34. Rule Post-Pruning • Convert tree to equivalent set of rules. • Prune each rule independently of others. • Sort final rules by their estimated accuracy, and consider them in this sequence when classifying subsequent instances. Outlook IF (Outlook = Sunny) & (Humidity = High) THEN PlayTennis = No IF (Outlook = Sunny) & (Humidity = Normal) THEN PlayTennis = Yes ………. sunny overcast rainy Humidity yes Windy normal false true no yes yes no

35. Continuous Valued Attributes • Create a set of discrete attributes to test continuous. • Apply Information Gain in order to choose the best attribute. Temperature: 40 48 60 72 80 90 PlayTennis: No No Yes Yes Yes No Temp>54 Tem>85

36. Missing Attribute Values • Strategies: • Assign most common value of A among other instances belonging to the same concept. • If node n tests the attribute A, assign most common value of A among other instances sorted to node n. • If node n tests the attribute A, assign a probability to each of possible values of A. These probabilities are estimated based on the observed frequencies of the values of A. These probabilities are used in the information gain measure (via entropy). (E( S) = - p+ log2 p+ – p- log2 p-)

37. Summary Points • Decision tree learning provides a practical method for concept learning. • ID3-like algorithms search complete hypothesis space. • The inductive bias of decision trees is preference (search) bias. • Overfitting the training data is an important issue in decision tree learning. • A large number of extensions of the ID3 algorithm have been proposed for overfitting avoidance, handling missing attributes, handling numerical attributes, etc.

38. Learning Decision Rules • Decision Rules • Basic Sequential Covering Algorithm • Learn-One-Rule Procedure • Pruning

39. Definition of Decision Rules Definition: Decision rules are rules with the following form: if <conditions> then concept C. Example: If you run the Prism algorithm from Weka on the weather data you will get the following set of decision rules: if outlook = overcast then PlayTennis = yes if humidity = normal and windy = FALSE then PlayTennis = yes if temperature = mild and humidity = normal then PlayTennis = yes if outlook = rainy and windy = FALSE then PlayTennis = yes if outlook = sunny and humidity = high then PlayTennis = no if outlook = rainy and windy = TRUE then PlayTennis = no

40. X 1 0 Z Y 1 0 1 0 0 W 1 Z 1 0 1 0 0 1 0 W 1 0 0 1 Why Decision Rules? • Decision rules are more compact. • Decision rules are more understandable. Example: Let X{0,1}, Y{0,1}, Z {0,1}, W{0,1}. The rules are: ifX=1 and Y=1 then 1 ifZ=1 and W=1 then 1 Otherwise 0;

41. Decision boundaries of decision trees - - + - - + - + + + + + + + + + - - - + - - - - - Decision boundaries of decision rules - - + - - + - + + + + + + + + + - - - + - - - - - Why Decision Rules?

42. How to Learn Decision Rules? • We can convert trees to rules • We can use specific rule-learning methods

43. Sequential Covering Algorithms function LearnRuleSet(Target, Attrs, Examples, Threshold): LearnedRules :=  Rule := LearnOneRule(Target, Attrs, Examples) while performance(Rule,Examples) > Threshold, do LearnedRules := LearnedRules  {Rule} Examples := Examples \ {examples covered by Rule} Rule := LearnOneRule(Target, Attrs, Examples) sort LearnedRules according to performance return LearnedRules

44. IF true THEN pos Illustration - - + - - + - + + + + + + + + + - - - + - - - - -

45. IF true THEN pos IF A THEN pos Illustration - - + - - + - + + + + + + + + + - - - + - - - - -

46. IF true THEN pos IF A THEN pos IF A & B THEN pos Illustration - - + - - + - + + + + + + + + + - - - + - - - - -

47. IF true THEN pos Illustration - - + - - + - + + + + + + + + + - - - + - - - - - IF A & B THEN pos

48. IF true THEN pos IF C THEN pos Illustration - - + - - + - + + + + + + + + + - - - + - - - - - IF A & B THEN pos

49. Illustration - - + - - + - + + + + + + + + + - - - + - - - - - IF A & B THEN pos IF C THEN pos IF C & D THEN pos IF true THEN pos

50. Learning One Rule To learn one rule we use one of the strategies below: • Top-down: • Start with maximally general rule • Add literals one by one • Bottom-up: • Start with maximally specific rule • Remove literals one by one • Combination of top-down and bottom-up: • Candidate-elimination algorithm.