1 / 39

Decision trees

Decision trees. David Kauchak CS 451 – Fall 2013. Admin. Assignment 1 available and is due on Friday (printed out at the beginning of class) Door code for MBH632 Keep up with the reading Videos. Quick refresher from last time…. Representing examples. examples. What is an example?

aretha-fernandez
Download Presentation

Decision trees

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Decision trees David KauchakCS 451 – Fall 2013

  2. Admin Assignment 1 available and is due on Friday (printed out at the beginning of class) Door code for MBH632 Keep up with the reading Videos

  3. Quick refresher from last time…

  4. Representing examples examples What is an example? How is it represented?

  5. Features examples features How our algorithms actually “view” the data Features are the questions we can ask about the examples f1, f2, f3, …, fn f1, f2, f3, …, fn f1, f2, f3, …, fn f1, f2, f3, …, fn

  6. Features examples features How our algorithms actually “view” the data Features are the questions we can ask about the examples red, round, leaf, 3oz, … green, round, no leaf, 4oz, … yellow, curved, no leaf, 4oz, … green, curved, no leaf, 5oz, …

  7. Classification revisited label examples red, round, leaf, 3oz, … apple learn green, round, no leaf, 4oz, … apple model/ classifier yellow, curved, no leaf, 4oz, … banana banana green, curved, no leaf, 5oz, … During learning/training/induction, learn a model of what distinguishes apples and bananas based on the features

  8. Classification revisited predict Apple or banana? model/ classifier red, round, no leaf, 4oz, … The model can then classify a new example based on the features

  9. Classification revisited predict model/ classifier Apple red, round, no leaf, 4oz, … Why? The model can then classify a new example based on the features

  10. Classification revisited Training data Test set label examples red, round, leaf, 3oz, … apple ? green, round, no leaf, 4oz, … apple red, round, no leaf, 4oz, … yellow, curved, no leaf, 4oz, … banana banana green, curved, no leaf, 5oz, …

  11. Classification revisited Training data Test set label examples red, round, leaf, 3oz, … apple ? green, round, no leaf, 4oz, … apple red, round, no leaf, 4oz, … yellow, curved, no leaf, 4oz, … banana Learning is about generalizing from the training data banana green, curved, no leaf, 5oz, … What does this assume about the training and test set?

  12. Past predicts future Training data Test set

  13. Past predicts future Training data Test set Not always the case, but we’ll often assume it is!

  14. Past predicts future Training data Test set Not always the case, but we’ll often assume it is!

  15. More technically… We are going to use the probabilistic model of learning There is some probability distribution over example/label pairs called the data generating distribution Both the training data and the test set are generated based on this distribution What is a probability distribution?

  16. Probability distribution Describes how likely (i.e. probable) certain events are

  17. Probability distribution Training data High probability Low probability round apples curved bananas apples with leaves … curved apples red bananas yellow apples …

  18. data generating distribution Training data Test set data generating distribution

  19. data generating distribution Training data Test set data generating distribution

  20. data generating distribution Training data Test set data generating distribution

  21. A sample data set 8 AM, Rainy, Yes, No? 10 AM, Rainy, No, No? Can you describe a “model” that could be used to make decisions in general?

  22. Decision trees Leave At Tree with internal nodes labeled by features Branches are labeled by tests on that feature Leaves labeled with classes 10 AM 9 AM 8 AM Stall? Accident? No Yes Long No Yes Short Long Short Long

  23. Decision trees Leave At Tree with internal nodes labeled by features Branches are labeled by tests on that feature Leaves labeled with classes 10 AM 9 AM 8 AM Stall? Accident? No Yes Long No Yes Short Long Short Long Leave = 8 AM Weather = Rainy Accident = Yes Stall = No

  24. Decision trees Leave At Tree with internal nodes labeled by features Branches are labeled by tests on that feature Leaves labeled with classes 10 AM 9 AM 8 AM Stall? Accident? No Yes Long No Yes Short Long Short Long Leave = 8 AM Weather = Rainy Accident = Yes Stall = No

  25. Decision trees Leave At Tree with internal nodes labeled by features Branches are labeled by tests on that feature Leaves labeled with classes 10 AM 9 AM 8 AM Stall? Accident? No Yes Long No Yes Short Long Short Long Leave = 10 AM Weather = Rainy Accident = No Stall = No

  26. Decision trees Leave At Tree with internal nodes labeled by features Branches are labeled by tests on that feature Leaves labeled with classes 10 AM 9 AM 8 AM Stall? Accident? No Yes Long No Yes Short Long Short Long Leave = 10 AM Weather = Rainy Accident = No Stall = No

  27. To ride or not to ride, that is the question… Build a decision tree

  28. Recursive approach Base case: If all data belong to the same class, create a leaf node with that label Otherwise: • calculate the “score” for each feature if we used it to split the data • pick the feature with the highest score, partition the data based on that data value and call recursively

  29. Partitioning the data Terrain Trail Road YES: 4 NO: 1 YES: 2 NO: 3 Unicycle Normal Mountain YES: 4 NO: 0 YES: 2 NO: 4 Weather Sunny Rainy Snowy YES: 2 NO: 1 YES: 2 NO: 2 YES: 2 NO: 1

  30. Partitioning the data Terrain Trail Road YES: 4 NO: 1 YES: 2 NO: 3 Unicycle calculate the “score” for each feature if we used it to split the data Normal Mountain YES: 4 NO: 0 YES: 2 NO: 4 What score should we use? If we just stopped here, which tree would be best? How could we make these into decision trees? Weather Sunny Rainy Snowy YES: 2 NO: 1 YES: 2 NO: 2 YES: 2 NO: 1

  31. Decision trees Terrain Trail Road YES: 4 NO: 1 YES: 2 NO: 3 Unicycle Normal Mountain How could we make these into decision trees? YES: 4 NO: 0 YES: 2 NO: 4 Weather Sunny Rainy Snowy YES: 2 NO: 1 YES: 2 NO: 2 YES: 2 NO: 1

  32. Decision trees Terrain Trail Road YES: 4 NO: 1 YES: 2 NO: 3 Unicycle Normal Mountain YES: 4 NO: 0 YES: 2 NO: 4 Weather Sunny Rainy Snowy YES: 2 NO: 1 YES: 2 NO: 2 YES: 2 NO: 1

  33. Decision trees Terrain Trail Road YES: 4 NO: 1 YES: 2 NO: 3 Unicycle Normal Mountain Training error: the average error over the training set YES: 4 NO: 0 YES: 2 NO: 4 For classification, the most common “error” is the number of mistakes Weather Sunny Rainy Snowy Training error for each of these? YES: 2 NO: 1 YES: 2 NO: 2 YES: 2 NO: 1

  34. Decision trees Terrain Trail Road YES: 4 NO: 1 YES: 2 NO: 3 Unicycle 3/10 2/10 4/10 Normal Mountain YES: 4 NO: 0 YES: 2 NO: 4 Weather Training error: the average error over the training set Sunny Rainy Snowy YES: 2 NO: 1 YES: 2 NO: 2 YES: 2 NO: 1

  35. Recurse Unicycle Normal Mountain YES: 4 NO: 0 YES: 2 NO: 4

  36. Recurse Terrain Trail Road YES: 2 NO: 1 YES: 0 NO: 3 Unicycle Normal Mountain Weather Sunny Rainy YES: 4 NO: 0 YES: 2 NO: 4 Snowy YES: 1 NO: 1 YES: 0 NO: 2 YES: 1 NO: 1

  37. Recurse Terrain Trail Road YES: 2 NO: 1 YES: 0 NO: 3 1/6 Unicycle Normal Mountain Weather Sunny Rainy YES: 4 NO: 0 YES: 2 NO: 4 Snowy YES: 1 NO: 1 YES: 0 NO: 2 YES: 1 NO: 1 2/6

  38. Recurse Unicycle Normal Mountain Terrain YES: 4 NO: 0 Trail Road YES: 2 NO: 1 YES: 0 NO: 3

  39. Recurse Unicycle Normal Mountain Terrain YES: 4 NO: 0 Trail Road Weather YES: 0 NO: 3 Sunny Rainy Snowy YES: 1 NO: 1 YES: 0 NO: 1 YES: 1 NO: 0

More Related