Machine Learning

1. Machine Learning CPSC 315 – Programming Studio Spring 2009 Project 2, Lecture 5

2. Forms of Learning • Supervised • Learns from examples which provide desired outputs for given inputs • Unsupervised • Learns patterns in input data when no specific output values are given • Reinforcement • Learns by an indication of correctness at end of some reasoning

3. Supervised Learning • Must have training data including • Inputs (features) to be considered in decision • Outputs (correct decisions) for those inputs • Inductive reasoning • Given a collection of examples of function f, return a function h that approximates f • Difficulty: many functions h may be possible • Hope to pick function h that generalizes well • Tradeoff between the complexity of the hypothesis and the degree of fit to the data • Consider data modeling

4. Evaluating Supervised Learning Algorithms • Collect a large set of examples (input/output pairs) • Divide into two disjoint sets • Training data • Testing data • Apply learning algorithm to training data, generating a hypothesis h • Measure % of examples in the testing data that are successfully classified by h (or amount of error for continuously valued outputs) • Repeat above steps for different sizes of training sets and different randomly selected training sets

5. Decision Trees • Map features of situation to decision • Example from a classification of unsafe acts:

6. Decision Trees • Relation to rule-based reasoning • Features of element used to classify element • Features of situation used to select action • Used as the basis for many “how to” books • How to identify type of snake? • Observable features of snake • How to fix an automobile? • Features related to problem and state of automobile • If features are understandable, the decision tree can be used to explain decision

7. Learning Decision Trees • Types of Decision Trees • Learning a discrete-valued function is classification learning • Learning a continuous-valued function is regression • Assumption: use of Ockham’s razor will result in more general function • Want the smallest decision tree, but that is not tractable • Will be satisfied with smallish tree

8. Algorithm for Decision Tree Learning • Basic idea • Recursively select feature that splits data (most) unevenly • No need to use all features • Heuristic approach • Compare features for their ability to meaningfully split data • Feature-value = greatest difference in average output value(s) * size of smaller subset • Avoids splitting out individuals too early

9. Unsupervised Learning • Used to characterize/explain the key features of a set of data • No notion of desired output • Example: identifying fast-food vs. fine-dining restaurants when classes are not known ahead of time • Techniques • Clustering (k means, HAC) • Self-Organizing Maps • Gaussian Mixture Models • More on this topic in Project 3

10. Reinforcement Learning • Many large problems do not have desired outputs that can be used as training data • Process • Agent (system) performs a set of actions • Agent occasionally receives a reward to indicate something went right or penalty to indicate something went wrong • Agent has to learn relationship between the model of the situation, the chosen actions, and the rewards/penalties

11. Analogy to Animal Training • We cannot tell our pets what is right and wrong in (preconditions, action) pairs • Instead we reward good behavior (giving treats) and penalize bad behavior (spraying water or loud noise) • Pet has to learn when and where what is appropriate • Can result in incorrect interpretations (go in corner vs. go outside) • Difficulty: what of the prior/recent actions caused the positive/negative outcome • Clicker training for animals is meant to help this

12. Reinforcement Learning in Games • Simplest reinforcements • Winning or losing • Requires lots of games/time to learn • Other potential reinforcements • Opponent’s action selection • Did they minimize your goodness value • Modify goodness function to better match their moves • Potential to learn an individual’s values/strategy • Predicted goodness value vs. observed goodness value • This can be used in small (a few moves) or large (a game) time scales • Similar to person reflecting on when things went wrong • Need to be careful in implementation or else goodness function will return a constant (thus being totally consistent)

13. Modifying a Goodness Function • Consider the game of chess • Presume goodness function has three linear components • BoardControl • the difference between the number of board positions that Player1 and Player2 can get a piece to in one move • Threatened • the difference between the number of opponents pieces threatened (can be taken in one move) between Player1 and Player2 • Pieces • the difference in the sum of the values of pieces left for Player1 and Player2 where Queen = 10, Rook = 6, Bishop = 3, Knight = 3, Pawn = 1

14. Modifying a Goodness Function • G(s) = a*BoardControl + b*Pieces + c*Threatened • Modify coefficients to learn appropriate weighting of terms • Quantity of overall modification should relate to difference between predicted goodness and observed goodness • Direction of modification to each linear component should be related to whether they are consistent with or disagree with outcome • Could modify coefficients using fixed values (e.g. +/- .1) or with values a function of their effect on overall G for the state being considered • In theory, such a computer player could recognize that BoardControl is more important early in a game, Pieces is more important mid-game, and Threatened is more important for the end game.