1 / 27

Split Selection for Decision Tree

Split Selection for Decision Tree. We already learned on selection method Information Gain (entropy) Are there any other selection criterion? Yes How many are they? A Lot Intuition Formalism Various Selection Methods. How to determine the Best Split. Greedy approach:

paulpeterson
Download Presentation

Split Selection for Decision Tree

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Split Selection for Decision Tree We already learned on selection method • Information Gain (entropy) • Are there any other selection criterion? • Yes • How many are they? • A Lot • Intuition • Formalism • Various Selection Methods CAP5610

  2. CAP5610

  3. How to determine the Best Split • Greedy approach: • Nodes with homogeneous class distribution are preferred • Need a measure of node impurity: Non-homogeneous, High degree of impurity Homogeneous, Low degree of impurity CAP5610

  4. M0 M2 M3 M4 M1 M12 M34 How to Find the Best Split Before Splitting: A? B? Yes No Yes No Node N1 Node N2 Node N3 Node N4 Gain = M0 – M12 vs M0 – M34

  5. CAP5610

  6. CAP5610

  7. CAP5610

  8. CAP5610

  9. CAP5610

  10. CAP5610

  11. CAP5610

  12. CAP5610

  13. CAP5610

  14. CAP5610

  15. CAP5610

  16. CAP5610

  17. CAP5610

  18. CAP5610

  19. Remedy: Concavity • Use impurity functions that are concave • Phi’’ <0 • Example impurity functions • Entropy • Gini-Index CAP5610

  20. Entropy as Impurity Measure • Entropy at a given node t: (NOTE: p( j | t) is the relative frequency of class j at node t). • Measures homogeneity of a node. • Maximum (log nc) when records are equally distributed among all classes implying least information • Minimum (0.0) when all records belong to one class, implying most information • Entropy based computations are similar to the GINI index computations

  21. Measure of Impurity: GINI • Gini Index for a given node t : (NOTE: p( j | t) is the relative frequency of class j at node t). • Maximum (1 - 1/nc) when records are equally distributed among all classes, implying least interesting information • Minimum (0.0) when all records belong to one class, implying most interesting information

  22. Misclassification Error / Resubstitution Error • Classification error at a node t : • Measures misclassification error made by a node. • Maximum (1 - 1/nc) when records are equally distributed among all classes, implying least interesting information • Minimum (0.0) when all records belong to one class, implying most interesting information

  23. Comparison among Splitting Criteria For a 2-class problem:

  24. CAP5610

  25. CAP5610

  26. CAP5610

  27. CAP5610

More Related