1 / 22

Decision Tree Learning

Decision Tree Learning. Presented by Ping Zhang Nov. 26th, 2007. Introduction. Decision tree learning is one of the most widely used and practical method for inductive inference

ramya
Download Presentation

Decision Tree Learning

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Decision Tree Learning Presented by Ping Zhang Nov. 26th, 2007

  2. Introduction • Decision tree learning is one of the most widely used and practical method for inductive inference • Decision tree learning is a method for approximating discrete-valued target functions, in which the learned function is represented by a decision tree • Decision tree learning is robust to noisy data and capable of learning disjunctive expressions

  3. Decision tree representation • Decision tree classify instances by sorting them down the tree from the root to some leaf node, which provides the classification of the instance • Each node in the tree specifies a test of some attribute of the instance, and each branch descending from that node corresponds to one of the possible values for this attributes

  4. Decision Tree for PlayTennis

  5. When to Consider Decision Trees • Instances describable by attribute-value pairs • Target function is discrete valued • Disjunctive hypothesis may be required • Possibly noisy training data Examples (Classification problems): • Equipment or medical diagnosis • Credit risk analysis

  6. Top-Down Induction of Decision Trees

  7. Entropy (1)

  8. Entropy (2)

  9. Information Gain

  10. Training Examples

  11. Selecting the Next Attribute

  12. Which attribute should be tested here?

  13. Hypothesis Space Search by ID3 • Hypothesis space is complete Target function surely in there • Only outputs a single hypothesis • No back tracking Local minima • Statically-based search choices Robust to noisy data • Inductive bias: “prefer shortest tree”

  14. From ID3 to C4.5 C4.5 made a number of improvements to ID3. Some of these are: • Handling both continuous and discrete attributes • Handling training data with missing attribute value • Handling attributes with differing costs • Pruning trees after creation

  15. Overfitting in Decision Trees

  16. Reduced-Error Pruning

  17. Rule Post-Pruning • Convert tree to equivalent set of rules • Prune each rule by removing any preconditions that result in improving its estimated accuracy • Sort the pruned rules by their estimated accuracy, and consider them in this sequence when classifying subsequent instance • Perhaps most frequently used method

  18. Continuous Valued Attributes • Create a discrete attribute to test continuous • There are two candidate thresholds • The information gain can be computed for each of the candidate attributes, Temperature>54 and Temperature>85, and the best can be selected(Temperature>54)

  19. Attributes with many Values Problems: • If attribute has many values, Gain will select it • Imagine using the attribute Data. It would have the highest information gain of any of attributes. But the decision tree is not useful.

  20. Missing Attribute Values

  21. Attributes with Costs • Consider Medical diagnosis, BloodTset has cost 150 dallors • How to learn a consistent tree with low expected cost?

  22. Conclusion Decision Tree Learning is • Simple to understand and interpret • Requires little data preparation • Able to handle both numerical and categorical data • Use a white box model • Possible to validate a model using statistical tests • Robust, perform well with large data in a short time

More Related