1 / 48

Inductive Learning (1/2) Decision Tree Method

Inductive Learning (1/2) Decision Tree Method. Russell and Norvig: Chapter 18, Sections 18.1 through 18.4 Chapter 18, Sections 18.1 through 18.3 CS121 – Winter 2003. Quotes.

lperryman
Download Presentation

Inductive Learning (1/2) Decision Tree Method

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Inductive Learning (1/2)Decision Tree Method Russell and Norvig: Chapter 18, Sections 18.1 through 18.4 Chapter 18, Sections 18.1 through 18.3 CS121 – Winter 2003

  2. Quotes • “Our experience of the world is specific, yet we are able to formulate general theories that account for the past and predict the future” Genesereth and Nilsson, Logical Foundations of AI, 1987 • “Entities are not to be multiplied without necessity”Ockham, 1285-1349

  3. sensors environment ? agent actuators Learning Agent Critic Percepts Learning element Problem solver KB Actions

  4. Contents • Introduction to inductive learning • Logic-based inductive learning: • Decision tree method • Version space method • Function-based inductive learning • Neural nets

  5. Contents • Introduction to inductive learning • Logic-based inductive learning: • Decision tree method • Version space method + why inductive learning works • Function-based inductive learning • Neural nets 1 2 3

  6. Inductive Learning Frameworks • Function-learning formulation • Logic-inference formulation

  7. f(x) x Function-Learning Formulation • Goal function f • Training set: (xi, f(xi)), i = 1,…,n • Inductive inference: Find a function h that fits the point well  Neural nets

  8. Usually, not a sound inference Logic-Inference Formulation • Background knowledge KB • Training set D (observed knowledge) such that KB D and {KB, D}is satisfiable • Inductive inference:Find h(inductive hypothesis) such that • {KB, h} is satisfiable • KB,hD h = D is a trivial,but uninteresting solution (data caching)

  9. Rewarded Card Example • Deck of cards, with each card designated by [r,s], its rank and suit, and some cards “rewarded” • Background knowledge KB:((r=1) v … v (r=10))  NUM(r)((r=J) v (r=Q) v (r=K))  FACE(r)((s=S) v (s=C))  BLACK(s)((s=D) v (s=H))  RED(s) • Training set D:REWARD([4,C])  REWARD([7,C])  REWARD([2,S])  REWARD([5,H])  REWARD([J,S])

  10. Rewarded Card Example • Background knowledge KB:((r=1) v … v (r=10))  NUM(r)((r=J) v (r=Q) v (r=K))  FACE(r)((s=S) v (s=C))  BLACK(s)((s=D) v (s=H))  RED(s) • Training set D:REWARD([4,C])  REWARD([7,C])  REWARD([2,S])  REWARD([5,H])  REWARD([J,S]) • Possible inductive hypothesis:h (NUM(r)  BLACK(s)  REWARD([r,s])) There are several possible inductive hypotheses

  11. Learning a Predicate • Set E of objects (e.g., cards) • Goal predicate CONCEPT(x), where x is an object in E, that takes the value True or False (e.g., REWARD) • Example: CONCEPT describes the precondition of an action, e.g., Unstack(C,A) • E is the set of states • CONCEPT(x)  HANDEMPTYx, BLOCK(C) x, BLOCK(A) x, CLEAR(C) x, ON(C,A) x • Learning CONCEPT is a step toward learning the action

  12. Learning a Predicate • Set E of objects (e.g., cards) • Goal predicate CONCEPT(x), where x is an object in E, that takes the value True or False (e.g., REWARD) • Observable predicates A(x), B(X), … (e.g., NUM, RED) • Training set: values of CONCEPT for some combinations of values of the observable predicates

  13. A Possible Training Set Note that the training set does not say whether an observable predicate A, …, E is pertinent or not

  14. Learning a Predicate • Set E of objects (e.g., cards) • Goal predicate CONCEPT(x), where x is an object in E, that takes the value True or False (e.g., REWARD) • Observable predicates A(x), B(X), … (e.g., NUM, RED) • Training set: values of CONCEPT for some combinations of values of the observable predicates • Find a representation of CONCEPT in the form: CONCEPT(x) S(A,B, …)where S(A,B,…) is a sentence built with the observable predicates, e.g.:CONCEPT(x)  A(x) (B(x) v C(x))

  15. These objects are arches: (positive examples) These aren’t: (negative examples) Learning the concept of an Arch ARCH(x)  HAS-PART(x,b1)  HAS-PART(x,b2) HAS-PART(x,b3)  IS-A(b1,BRICK) IS-A(b2,BRICK) MEET(b1,b2)  (IS-A(b3,BRICK) v IS-A(b3,WEDGE))  SUPPORTED(b3,b1)  SUPPORTED(b3,b2)

  16. a small one! Example set • An example consists of the values of CONCEPT and the observable predicates for some object x • A example is positive if CONCEPT is True, else it is negative • The set E of all examples is the example set • The training set is a subset of E

  17. It is called a space because it has some internal structure Hypothesis Space • An hypothesis is any sentence h of the form: CONCEPT(x) S(A,B, …)where S(A,B,…) is a sentence built with the observable predicates • The set of all hypotheses is called the hypothesis space H • An hypothesis h agrees with an example if it gives the correct value of CONCEPT

  18. Inductivehypothesis h Training set D - - + - + - - - - + + + + - - + + + + - - - + + Hypothesis space H {[CONCEPT(x)  S(A,B, …)]} Example set X {[A, B, …, CONCEPT]} Inductive Learning Scheme

  19. 2n 2 Size of Hypothesis Space • n observable predicates • 2n entries in truth table • In the absence of any restriction (bias), there are hypotheses to choose from • n = 6  2x1019 hypotheses!

  20. Multiple Inductive Hypotheses Need for a system of preferences – called a bias – to compare possible hypotheses h1  NUM(x)  BLACK(x)  REWARD(x) h2  BLACK([r,s]) (r=J)  REWARD([r,s]) h3  ([r,s]=[4,C])  ([r,s]=[7,C])  [r,s]=[2,S])  REWARD([r,s]) h3  ([r,s]=[5,H])  ([r,s]=[J,S])  REWARD([r,s]) agree with all the examples in the training set

  21. Keep-It-Simple (KIS) Bias • Motivation • If an hypothesis is too complex it may not be worth learning it (data caching might just do the job as well) • There are much fewer simple hypotheses than complex ones, hence the hypothesis space is smaller • Examples: • Use much fewer observable predicates than suggested by the training set • Constrain the learnt predicate, e.g., to use only “high-level” observable predicates such as NUM, FACE, BLACK, and RED and/or to have simple syntax (e.g., conjunction of literals) If the bias allows only sentences S that are conjunctions of k << n predicates picked fromthe n observable predicates, then the size of H is O(nk)

  22. Testset Object set yes Evaluation no Example set X Goal predicate Training set D Inducedhypothesis h Learningprocedure L Hypothesisspace H Bias Observable predicates Putting Things Together

  23. Predicate-Learning Methods • Decision tree • Version space

  24. A? True False B? False False True C? True True False True False Predicate as a Decision Tree The predicate CONCEPT(x)  A(x) (B(x) v C(x)) can be represented by the following decision tree: • Example:A mushroom is poisonous iffit is yellow and small, or yellow, • big and spotted • x is a mushroom • CONCEPT = POISONOUS • A = YELLOW • B = BIG • C = SPOTTED

  25. A? True False B? False False True C? True True False True False Predicate as a Decision Tree The predicate CONCEPT(x)  A(x) (B(x) v C(x)) can be represented by the following decision tree: • Example:A mushroom is poisonous iffit is yellow and small, or yellow, • big and spotted • x is a mushroom • CONCEPT = POISONOUS • A = YELLOW • B = BIG • C = SPOTTED • D = FUNNEL-CAP • E = BULKY

  26. Training Set

  27. D E B A A T F C T F T F T E A F T T F Possible Decision Tree

  28. D E B A A T F C A? CONCEPT  A (B v C) True False T F B? False False T F T True E C? True False A True False True F T T F Possible Decision Tree CONCEPT  (D  (E v A)) v (C  (B v ((E A) v A))) KIS bias  Build smallest decision tree Computationally intractable problem greedy algorithm

  29. Getting Started The distribution of the training set is: True: 6, 7, 8, 9, 10,13 False: 1, 2, 3, 4, 5, 11, 12

  30. Getting Started The distribution of training set is: True: 6, 7, 8, 9, 10,13 False: 1, 2, 3, 4, 5, 11, 12 Without testing any observable predicate, we could report that CONCEPT is False (majority rule) with an estimated probability of error P(E) = 6/13

  31. Getting Started The distribution of training set is: True: 6, 7, 8, 9, 10,13 False: 1, 2, 3, 4, 5, 11, 12 Without testing any observable predicate, we could report that CONCEPT is False (majority rule)with an estimated probability of error P(E) = 6/13 Assuming that we will only include one observable predicate in the decision tree, which predicateshould we test to minimize the probability or error?

  32. A F T 6, 7, 8, 9, 10, 13 11, 12 True: False: 1, 2, 3, 4, 5 If we test only A, we will report that CONCEPT is Trueif A is True (majority rule) and False otherwise The estimated probability of error is: Pr(E) = (8/13)x(2/8) + (5/8)x0 = 2/13 Assume It’s A

  33. B F T 9, 10 2, 3, 11, 12 True: False: 6, 7, 8, 13 1, 4, 5 If we test only B, we will report that CONCEPT is Falseif B is True and True otherwise The estimated probability of error is: Pr(E) = (6/13)x(2/6) + (7/13)x(3/7) = 5/13 Assume It’s B

  34. C F T 6, 8, 9, 10, 13 1, 3, 4 True: False: 7 1, 5, 11, 12 If we test only C, we will report that CONCEPT is Trueif C is True and False otherwise The estimated probability of error is: Pr(E) = (8/13)x(3/8) + (5/13)x(1/5) = 4/13 Assume It’s C

  35. D F T 7, 10, 13 3, 5 True: False: 6, 8, 9 1, 2, 4, 11, 12 If we test only D, we will report that CONCEPT is Trueif D is True and False otherwise The estimated probability of error is: Pr(E) = (5/13)x(2/5) + (8/13)x(3/8) = 5/13 Assume It’s D

  36. E F T 8, 9, 10, 13 1, 3, 5, 12 True: False: 6, 7 2, 4, 11 If we test only E we will report that CONCEPT is False, independent of the outcome The estimated probability of error is unchanged: Pr(E) = (8/13)x(4/8) + (5/13)x(2/5) = 6/13 Assume It’s E So, the best predicate to test is A

  37. 6, 8, 9, 10, 13 True: False: 7 11, 12 Choice of Second Predicate A F T False C F T The majority rule gives the probability of error Pr(E|A) = 1/8and Pr(E) = 1/13

  38. 11,12 True: False: 7 Choice of Third Predicate A F T False C F T True B T F

  39. A True False A? C False False True True False B? False True B False True True False C? True False True True False False True Final Tree L  CONCEPT  A (C v B)

  40. Noise in training set! May return majority rule,instead of failure Subset of examples that satisfy A Learning a Decision Tree DTL(D,Predicates) • If all examples in D are positive then return True • If all examples in D are negative then return False • If Predicates is empty then return failure • A  most discriminating predicate in Predicates • Return the tree whose: - root is A, - left branch is DTL(D+A,Predicates-A), - right branch is DTL(D-A,Predicates-A)

  41. Using Information Theory • Rather than minimizing the probability of error, most existing learning procedures try to minimize the expected number of questions needed to decide if an object x satisfies CONCEPT • This minimization is based on a measure of the “quantity of information” that is contained in the truth value of an observable predicate

  42. 100 % correct on test set size of training set Typical learning curve Miscellaneous Issues • Assessing performance: • Training set and test set • Learning curve

  43. Risk of using irrelevantobservable predicates togenerate an hypothesisthat agrees with all examplesin the training set The resulting decision tree + majority rule may not classify correctly all examples in the training set Terminate recursion when information gain is too small Miscellaneous Issues • Assessing performance: • Training set and test set • Learning curve • Overfitting • Tree pruning

  44. The value of an observablepredicate P is unknown foran example x. Then construct a decision tree for both valuesof P and select the value that ends up classifying x in the largest class Miscellaneous Issues • Assessing performance: • Training set and test set • Learning curve • Overfitting • Tree pruning • Cross-validation • Missing data

  45. Select threshold that maximizesinformation gain Miscellaneous Issues • Assessing performance: • Training set and test set • Learning curve • Overfitting • Tree pruning • Cross-validation • Missing data • Multi-valued and continuous attributes These issues occur with virtually any learning method

  46. Patrons? None Full Some Hungry? True False Yes No False Type? Burger Italian Thai False FriSat? True Multi-valued attributes False True Multi-Valued Attributes WillWait predicate (Russell and Norvig)

  47. Applications of Decision Tree • Medical diagnostic / Drug design • Evaluation of geological systems for assessing gas and oil basins • Early detection of problems (e.g., jamming) during oil drilling operations • Automatic generation of rules in expert systems

  48. Summary • Inductive learning frameworks • Logic inference formulation • Hypothesis space and KIS bias • Inductive learning of decision trees • Assessing performance • Overfitting

More Related