1 / 38

Decision Trees

Decision Trees. categorical. categorical. continuous. class. Example of a Decision Tree. Splitting Attributes. Refund. Yes. No. NO. MarSt. Married. Single, Divorced. TaxInc. NO. < 80K. > 80K. YES. NO. Model: Decision Tree. Training Data. NO. Another Example of Decision Tree.

jaimetaylor
Download Presentation

Decision Trees

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Decision Trees

  2. categorical categorical continuous class Example of a Decision Tree Splitting Attributes Refund Yes No NO MarSt Married Single, Divorced TaxInc NO < 80K > 80K YES NO Model: Decision Tree Training Data

  3. NO Another Example of Decision Tree categorical categorical continuous class Single, Divorced MarSt Married NO Refund No Yes TaxInc < 80K > 80K YES NO There could be more than one tree that fits the same data!

  4. Refund Yes No NO MarSt Married Single, Divorced TaxInc NO < 80K > 80K YES NO Apply Model to Test Data Test Data Start from the root of tree.

  5. Refund Yes No NO MarSt Married Single, Divorced TaxInc NO < 80K > 80K YES NO Apply Model to Test Data Test Data

  6. Apply Model to Test Data Test Data Refund Yes No NO MarSt Married Single, Divorced TaxInc NO < 80K > 80K YES NO

  7. Apply Model to Test Data Test Data Refund Yes No NO MarSt Married Single, Divorced TaxInc NO < 80K > 80K YES NO

  8. Apply Model to Test Data Test Data Refund Yes No NO MarSt Married Single, Divorced TaxInc NO < 80K > 80K YES NO

  9. Apply Model to Test Data Test Data Refund Yes No NO MarSt Assign Cheat to “No” Married Single, Divorced TaxInc NO < 80K > 80K YES NO

  10. Digression: Entropy

  11. Bits • We are watching a set of independent random samples of X • We see that X has four possible values • So we might see: BAACBADCDADDDA… • We transmit data over a binary serial link. We can encode each reading with two bits (e.g. A=00, B=01, C=10, D = 11) 0100001001001110110011111100…

  12. Fewer Bits • Someone tells us that the probabilities are not equal • It’s possible… …to invent a coding for your transmission that only uses 1.75 bits on average per symbol. Here is one.

  13. General Case • Suppose X can have one of m values… • What’s the smallest possible number of bits, on average, per symbol, needed to transmit a stream of symbols drawn from X’s distribution? It’s • Well, Shannon got to this formula by setting down several desirable properties for uncertainty, and then finding it.

  14. Back to Decision Trees

  15. Constructing decision trees (ID3) • Normal procedure: top down in a recursive divide-and-conquer fashion • First: an attribute is selected for root node and a branch is created for each possible attribute value • Then: the instances are split into subsets (one for each branch extending from the node) • Finally: the same procedure is repeated recursively for each branch, using only instances that reach the branch • Process stops if all instances have the same class

  16. Weather data

  17. Which attribute to select? (b) (a) (c) (d)

  18. A criterion for attribute selection • Which is the best attribute? • The one which will result in the smallest tree • Heuristic: choose the attribute that produces the “purest” nodes • Popular impurity criterion: entropy of nodes • Lower the entropy purer the node. • Strategy: choose attribute that results in lowest entropy of the children nodes.

  19. Attribute “Outlook” outlook=sunny info([2,3]) = entropy(2/5,3/5) = -2/5*log(2/5) -3/5*log(3/5) = .971 outlook=overcast info([4,0]) = entropy(4/4,0/4) = -1*log(1) -0*log(0) = 0 outlook=rainy info([3,2]) = entropy(3/5,2/5) = -3/5*log(3/5)-2/5*log(2/5) = .971 Expected info: .971*(5/14) + 0*(4/14) + .971*(5/14) = .693 0*log(0) is normally not defined.

  20. Attribute “Temperature” temperature=hot info([2,2]) = entropy(2/4,2/4) = -2/4*log(2/4) -2/4*log(2/4) = 1 temperature=mild info([4,2]) = entropy(4/6,2/6) = -4/6*log(1) -2/6*log(2/6) = .528 temperature=cool info([3,1]) = entropy(3/4,1/4) = -3/4*log(3/4)-1/4*log(1/4) = .811 Expected info: 1*(4/14) + .528*(6/14) + .811*(4/14) = .744

  21. Attribute “Humidity” humidity=high info([3,4]) = entropy(3/7,4/7) = -3/7*log(3/7) -4/7*log(4/7) = .985 humidity=normal info([6,1]) = entropy(6/7,1/7) = -6/7*log(6/7) -1/7*log(1/7) = .592 Expected info: .985*(7/14) + .592*(7/14) = .788

  22. Attribute “Windy” windy=false info([6,2]) = entropy(6/8,2/8) = -6/8*log(6/8) -2/8*log(2/8) = .811 humidity=true info([3,3]) = entropy(3/6,3/6) = -3/6*log(3/6) -3/6*log(3/6) = 1 Expected info: .811*(8/14) + 1*(6/14) = .892

  23. And the winner is... "Outlook" ...So, the root will be "Outlook" Outlook

  24. Continuing to split (for Outlook="Sunny") Which one to choose?

  25. Continuing to split (for Outlook="Sunny") temperature=hot: info([2,0]) = entropy(2/2,0/2) = 0 temperature=mild: info([1,1]) = entropy(1/2,1/2) = 1 temperature=cool: info([1,0]) = entropy(1/1,0/1) = 0 Expected info: 0*(2/5) + 1*(2/5) + 0*(1/5) = .4 humidity=high: info([3,0]) = 0 humidity=normal: info([2,0]) = 0 Expected info: 0 windy=false: info([1,2]) = entropy(1/3,2/3) = -1/3*log(1/3) -2/3*log(2/3) = .918 humidity=true: info([1,1]) = entropy(1/2,1/2) = 1 Expected info: .918*(3/5) + 1*(2/5) = .951 Winner is "humidity"

  26. Tree so far

  27. Tree so far Continuing to split (for Outlook="Overcast") • Nothing to split here, "play" is always "yes".

  28. Continuing to split (for Outlook="Rainy") • We can easily see that "Windy" is the one to choose. (Why?)

  29. The final decision tree • Note: not all leaves need to be pure; sometimes identical instances have different classes Þ Splitting stops when data can’t be split any further

  30. Information gain • Sometimes people don’t use directly the entropy of a node. Rather the information gain is being used. • Clearly, greater the information gain better the purity of a node. So, we choose “Outlook” for the root.

  31. Highly-branching attributes • The weather data with ID code

  32. Tree stump for ID code attribute

  33. Highly-branching attributes So, • Subsets are more likely to be pure if there is a large number of values • Information gain is biased towards choosing attributes with a large number of values • This may result in overfitting (selection of an attribute that is non-optimal for prediction)

  34. The gain ratio • Gain ratio: a modification of the information gain that reduces its bias • Gain ratio takes number and size of branches into account when choosing an attribute • It corrects the information gain by taking the intrinsic information of a split into account • Intrinsic information: entropy (with respect to the attribute on focus) of node to be split.

  35. Computing the gain ratio

  36. Gain ratios for weather data

  37. More on the gain ratio • “Outlook” still comes out top but “Humidity” is now a much closer contender because it splits the data into two subsets instead of three. • However: “ID code” has still greater gain ratio. But its advantage is greatly reduced. • Problem with gain ratio: it may overcompensate • May choose an attribute just because its intrinsic information is very low • Standard fix: choose an attribute that maximizes the gain ratio, provided the information gain for that attribute is at least as great as the average information gain for all the attributes examined.

  38. Discussion • Algorithm for top-down induction of decision trees (“ID3”) was developed by Ross Quinlan (University of Sydney Australia) • Gain ratio is just one modification of this basic algorithm • Led to development of C4.5, which can deal with numeric attributes, missing values, and noisy data • There are many other attribute selection criteria! (But almost no difference in accuracy of result.)

More Related