Concept Learning Machine Learning by T. Mitchell (McGraw-Hill) Chp. 2

1. Concept Learning Machine Learning by T. Mitchell (McGraw-Hill) Chp. 2

2. Much of learning involves acquiring general concepts from specific training examples e.g. what is a bird? what is a chair? Concept learning: Inferring a boolean-valued function from training examples of its input and output.

3. A Concept Learning Task Example • Target Concept “Days on which my friend Aldo enjoys his favorite water sport” (you may find it more intuitive to think of “Days on which the beach will be crowded” concept) • Task Learn to predict the value of EnjoySport/Crowded for an arbitrary day • Training Examples for the Target Concept • 6 attributes (Nominal-valued (symbolic) attributes): Sky (SUNNY, RAİNY, CLOUDY), Temp (WARM,COLD), Humidity (NORMAL, HIGH), Wind (STRONG, WEAK), Water (WARM, COOL), Forecast (SAME, CHANGE)

4. A Learning Problem x1 Unknown Function x2 y = f (x1,x2,x3, x4 ) x3 x4 Hypothesis Space (H): Set of all possible hypotheses that the learner may consider during learning the target concept. How many?

5. Hypothesis Space:Unrestricted Case • | A  B | = | B | | A | • |H4 H | = | {0,1} {0,1}{0,1}{0,1}  {0,1} | = 224 = 65536 function values • After 7 examples, still have 29 = 512 possibilities (out of 65536) for f • Is learning possible without any assumptions?

6. A Concept Learning Task In this notation, a conjunction Such as <warm,?,stong> indicaties A1=Warm AND A3=strong You dont need to understand the use of  (no value allowed), but understand the above and know that there may be more specific or more general hypotheses. • Hypothesis h: Conjunction of Constraints on Attributes Constraint Values: • Specific value (e.g., Water = Warm) • All values allowedfor that attribute (e.g., “Water = ?”) • No value allowed for that attribute (e.g., “Water = Ø”) • Hypothesis Representation – • Example Hypothesis for EnjoySport: ‘’Aldo enjoys his favorite sport only on sunny days with Strong wind’’: Sky AirTemp Humidity Wind Water Forecast <Sunny ? ?Strong ? ?> • The most general hypothesis – every day is a positive example of this concept • <?, ?, ?, ?, ?, ?> • The most specific possible hypothesis – no day is a positive ex. of this concept • <  ,, , , , > • Is this hypothesis consistent with the training examples? • What are some hypotheses that are consistent with the examples?

7. A Concept Learning Task(2) The instance space,X(book uses ‘’set of instances’’) all possible days represented by attributes Sky, AirTemp,... Target concept, c : • Any boolean-valued function defined over the instance space X • c : X  {0, 1} (I.e. if EnjoySport = Yes, then c(x) = 1 ) Training Examples(denoted by D): ordered pair <x, c(x)> • Positive example: member of the target concept, c(x) = 1 • Negative example: nonmember of the target concept, c(x) = 0 • Assumption: no missing X values • No noise in values of c (contradictory labels). Hypotheses Space, H: • Often picked by the designer, for instance only conjunctions (e.g. A1  A5) • H is the set of Boolean valued functions defined over X • Or you may narrow it down to conjunction of constraints on attributes The goal of the learner • Hypothesize or estimate c: • Find a hypothesis h such that h(x) = c(x)for all x in X

8. A Concept Learning Task(3) • Although the learning task is to determine a hypothesis h identical to c, over the entire set of instances X, the only information available about c is its value over the training instances D. • Inductive Learning Hypothesis Any hypothesis found to approximate the target function well over a sufficiently large set of training examples will also approximate the target function well over other unobserved examples.

9. Concept Learning As Search • Concept Learning As Search Concept learning can be viewed as the task of searching through a large space of hypotheses implicitly defined by the hypothesis representation. The goal of this search is to find the hypothesis that (best) fits the training examples. SkyAirTempHumidity Wind Water Forecast <Sunny/Rainy/Cloudy Warm/Cold Normal/High Weak/Strong Warm/Cold Change/Same> • EnjoySport Learning Task • Size of the instance space X 3  2  2  2  2  2 = 96 • Syntactically distinct hypotheses (including ?, Ø) 5  4  4  4  4  4 = 5120 • Semantically distinct hypotheses (Ø anywhere means the empty set of instances and classifies each possible instance as a negative example) 1 + (4  3  3  3  3  3) = 973 • Often much larger, sometimes infinite hypotheses spaces

10. Version Space • Version space: The version space, denoted VSH,D, with respect to hypothesis space H and training example D, is the subset of hypotheses fromHwhich are consistent with the training examples in D. VSH,D {h H | Consistent(h, D)}

11. Inductive Bias Mitchell-Chp. 2

12. What Justifies This Inductive Leap? • Example: Inductive Generalization Positive example: <Sunny, Warm, Normal, Strong, Cool, Change, Yes> Positive example: <Sunny, Warm, Normal, Light, Warm, Same, Yes> So far we may think that the right concept is: <Sunny, Warm, Normal, ?, ?, ?> • But why we believe we can classify the unseen(with the above hypothesis, we would say this is a Yes)? e.g., <Sunny, Warm, Normal, Strong, Warm, Same>

13. Inductive Bias • A biased Hypothesis space EnjoySport example • Restriction : only conjunctions of attribute values. • No representation for a disjunctive target concept • (Sky = Sunny) or (Wind = Weak) • Potential problem • We biased the learner (inductive bias) to consider only conjunctive hypotheses • But the concept may require a more expressive hypothesis space

14. UnBiased Learner • An Unbiased Learner Obvious solution: Provide a hypothesis space capable of representing every teachable concept = every possible subset of the instance space X • The set of all subsets of a set X is called the power set of X • EnjoySport Example • |Instance space| : 96 • |Power set of X| : 296 = 79228162514264337593543950336 • |Conjunctive hypothesis space| : 973 • Very biased hypothesis space indeed!!

15. Need for Inductive Bias • An Unbiased Learner Reformulate the EnjoySport learning task in an unbiased way • Defining a new hypothesis space H’ that can represent every subset of X • Allow arbitrary disjunctions, conjunctions, and negations • Example: “Sky = Sunny or Wind = Weak” • <Sunny, ?, ?, ?, ?, ?> OR <?, ?, ?, Weak, ?, ?> • New problem: We will be completely unable to generalize beyond the observed samples! • This is explained without using the representation in class: If we allow full flexibility for the concept (conjunctions, disjunctions,…) , then even if one unseen sample is left, then there are still 2 hypotheses left that are consistent with the data (i.e. You still havent precisely found the right concept, eliminating all those that are inconsistent with the data you have seen!).

16. Need For Inductive Bias • Fundamental property of inductive inference: A learner that makes no a priori assumptions regarding the identity of the target concept has no rational basis for classifying any unseen instances