1 / 58

Decision Trees

Decision Trees. continuous. categorical. categorical. class. Example of a Decision Tree. Splitting Attributes. Refund. Yes. No. NO. MarSt. Married. Single, Divorced. TaxInc. NO. < 80K. > 80K. YES. NO. Model: Decision Tree. Training Data. NO. Another Example of Decision Tree.

ivrit
Download Presentation

Decision Trees

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Decision Trees

  2. continuous categorical categorical class Example of a Decision Tree Splitting Attributes Refund Yes No NO MarSt Married Single, Divorced TaxInc NO < 80K > 80K YES NO Model: Decision Tree Training Data

  3. NO Another Example of Decision Tree continuous categorical categorical class Single, Divorced MarSt Married NO Refund No Yes TaxInc < 80K > 80K YES NO There could be more than one tree that fits the same data!

  4. Decision Tree Classification Task Decision Tree

  5. Decision Tree Induction is often based on Information Theory

  6. Information

  7. DT Induction • When all the marbles in the bowl are mixed up, little information is given. • When the marbles in the bowl are all from one class and those in the other two classes are on either side, more information is given. Use this approach with DT Induction !

  8. Information/Entropy • Given probabilitites p1, p2, .., ps whose sum is 1, Entropyis defined as: • Entropy measures the amount of randomness or surprise or uncertainty. • Goal in classification • no surprise • entropy = 0

  9. Measure of Impurity: GINI • Gini Index for a given node t : (NOTE: p( j | t) is the relative frequency of class j at node t). • Maximum (1 - 1/nc) when records are equally distributed among all classes, implying least interesting information • Minimum (0.0) when all records belong to one class, implying most interesting information

  10. Examples for computing GINI P(C1) = 0/6 = 0 P(C2) = 6/6 = 1 Gini = 1 – P(C1)2 – P(C2)2 = 1 – 0 – 1 = 0 P(C1) = 1/6 P(C2) = 5/6 Gini = 1 – (1/6)2 – (5/6)2 = 0.278 P(C1) = 2/6 P(C2) = 4/6 Gini = 1 – (2/6)2 – (4/6)2 = 0.444

  11. Splitting Based on GINI • Used in CART, SLIQ, SPRINT. • When a node p is split into k partitions (children), the quality of split is computed as, where, ni = number of records at child i, n = number of records at node p.

  12. CART • Create Binary Tree • Uses entropy • Formula to choose split point, s, for node t: • PL,PR : probability that a tuple in the training set will be on the left or right side of the tree.

  13. Stopping Criteria for Tree Induction • Stop expanding a node when all the records belong to the same class • Stop expanding a node when all the records have similar attribute values • Early termination

  14. Notes on Overfitting • Overfitting results in decision trees that are more complex than necessary • Training error no longer provides a good estimate of how well the tree will perform on previously unseen records • Need new ways for estimating errors

  15. How to Address Overfitting • Pre-Pruning (Early Stopping Rule) • Stop the algorithm before it becomes a fully-grown tree • Typical stopping conditions for a node: • Stop if all instances belong to the same class • Stop if all the attribute values are the same • More restrictive conditions: • Stop if number of instances is less than some user-specified threshold • Stop if expanding the current node does not improve impurity measures (e.g., Gini or information gain) • …

  16. How to Address Overfitting… • Post-pruning • Grow decision tree to its entirety • Trim the nodes of the decision tree in a bottom-up fashion • If generalization error improves after trimming, replace sub-tree by a leaf node. • Class label of leaf node is determined from majority class of instances in the sub-tree • Can use MDL for post-pruning

  17. Ensemble Methods

More Related