Concept Learnin g

1. Concept Learning C.C. Fang Reference: Mitchell Machine Learning, Chapter 2

2. Outline • Concept learning • Learning from instances (examples) • General-to-specific ordering over hypotheses • Version spaces and candidate-elimination algorithm • The role of bias

3. The definition of Concept Learning • Inferring Boolean-valued functions from training examples; • Concept learning is learning a function which has a Boolean-valued output • f: X → {0,1} • Many machine learning approaches us this simplistic binary view of the world • Example: What is the general concept?

4. Representing Hypotheses Many possible representations... Here, h is conjunction of constraints on attributes. Each constraint can be • a specfic value (e.g., “Water = Warm”) • don’t care (e.g., “Water =?”) • no value allowed (e.g.,“Water=ψ ;”)

5. A Concept Learning Task

6. The Inductive Learning Hypothesis • Note: the only information available about c is c(x) for each <x, c(x)> in D. • Any hypothesis found to approximate the target function well over a sufficiently large set of training examples will also approximate the target function well over other observed example. • Assumption of Inductive Learning • • The best hypothesis regarding the unseen instances is thehypothesis that best fits the observed training data

7. Viewing Learning As a Search Problem (1/2)

8. Viewing Learning As a Search Problem (2/2) • Study of learning algorithms that examine different strategies for searching the hypothesis space, e.g., • Find-S Algorithm • List-Then-Eliminate Algorithm • Candidate Elimination Algorithm • • How to exploit the naturally occurring structure in the hypothesis space ? • Relations among hypotheses , e.g., • General-to-Specific-Ordering

9. General-to-Specific-Ordering of Hypothesis (1/3)

10. General-to-Specific-Ordering of Hypothesis (2/3)

11. General-to-Specific-Ordering of Hypothesis (3/3)

12. Find-S Algorithm (1/3)

13. Find-S Algorithm (2/3)

14. Find-S Algorithm (3/3)

15. Complaints about Find-S • Can not tell whether it has learned concept (Output only one. Many other consistent hypotheses may exist!) • Picks a maximally specific h (why?) (Find a most specific hypothesis consistent with the training data) • Can not tell when training data inconsistent – What if there are noises or errors contained in training examples • Depending on H, there might be several !

16. Consistence of Hypotheses

17. Version Space

18. List-Then-Eliminate Algorithm 1.VersionSpace ← a list containing all hypotheses in H 2.For each training example, <x, c(x)> remove from VersionSpace any hypothesis h for which h(x)≠c(x) – i.e., eliminate hypotheses inconsistent with any training examples – The VersionSpace shrinks as more examples are observed 3.Output the list of hypotheses in VersionSpace

19. Drawbacks of List-Then-Eliminate • The algorithm requires exhaustively enumerating all hypotheses in H • An unrealistic approach ! (full search) • If insufficient (training) data is available, the algorithm will output a huge set of hypotheses consistent with the observed data

20. Example Version Space

21. Representing Version Space

22. Candidate Elimination Algorithm (1/3)

23. Candidate Elimination Algorithm (2/3)

24. Candidate Elimination Algorithm (3/3)

25. Example Trace (1/5)

26. Example Trace (2/5)

27. Example Trace (3/5)

28. Example Trace (4/5)

29. Example Trace (5/5)

30. Overview Example Trace

31. What Next Training Example

32. How Should These Be Classified ?

33. Ranking Inductive Learners according to their Biases Weak • Rote-Learner: This system simply memorizes the training data and their classification--- No generalization is involved. • Candidate-Elimination: New instances are classified only if all the hypotheses in the version space agree on the classification • Find-S: New instances are classified using the most specific hypothesis consistent with the training data Bias Strength Strong

34. Biased Hypothesis Space

35. Summary Points • Concept learning as search through H • General-to-specific ordering over H • Version space candidate elimination algorithm –S and G boundaries characterize learners uncertainty • Learner can generate useful queries