Class Project

1. Class Project • Due at end of finals week • Essentially anything you want, so long as it’s AI related and I approve • Any programming language you want • In pairs or individual • Email me by Wednesday, November 3

2. Projects • Implementing Knn to Classify Bedform Stability Fields • Blackjack Using Genetic Algorithms • Computer game players:Go, Checkers, Connect Four, Chess, Poker • Computer puzzle solvers: Minesweeper, mazes • Pac-Man with intelligent monsters • Genetic algorithms: • blackjack strategy • Automated 20-questions player • Paper on planning • Neural network spam filter • Learning neural networks via GAs

3. Projects • Solving neural networks via backprop • Code decryptor using Gas • Box pushing agent (competing against an opponent)

4. What didn’t work as well • Too complicated games: Risk, Yahtzee, Chess, Scrabble, Battle Simulation • Got too focused in making game work • I sometimes had trouble running the game • Game was often incomplete • Didn’t have time to do enough AI • Problems that were too vague • Simulated ant colonies / genetic algorithms • Bugs swarming for heat (emergent intelligence never happened) • Finding paths through snow • AdaBoost on protein folding data • Couldn’t get boosting working right, needed more time on small datasets (spent lots of time parsing protein data)

5. Reinforcement Learning • Game playing: So far, we have told the agent the value of a given board position. • How can agent learn which positions are important? • Play whole bunch of games, and receive reward at end (+ or -) • How to determine utility of states that aren’t ending states?

6. The setup: Possible game states • Terminal states have reward • Mission: Estimate utility of all possible game states

7. What is a state? • For chess: state is a combination of position on board and location of opponents • Half of your transitions are controlled by you (your moves) • Other half of your transitions are probabilistic (depend on opponent) • For now, we assume all moves are probabilistic (probabilities unknown)

8. Passive Learning • Agent learns by “watching” • Fixed probability of moving from one state to another

9. Sample Results

10. Technique #1: Naive Updating • Also known as Least Mean Squares (LMS) approach • Starting at home, obtain sequence of states to terminal state • Utility of terminal state = reward • loop back over all other states • utility for state i = running average of all rewards seen for state i

11. Naive Updating Analysis • Works, but converges slowly • Must play lots of games • Ignores that utility of a state should depend on successor

12. Technique #2: Adaptive Dynamic Programming • Utility of a state depends entirely on the successor state • If a state has one successor, utility should be the same • If a state has multiple successors, utility should be expected value of successors

13. Finding the utilities • To find all utilities, just solve equations • Set of linear equations, solveable • Changes each iteration as you learn probabilities • Completely intractable for large problems: • For a real game, it means finding actual utilities of all states

14. Technique 3: Temporal Difference Learning • Want utility to depend on successors, but want to solve iteratively • Whenever you observe a transition from i to j: • a = learning rate • difference between successive states = temporal difference • Converges faster than Naive updating

15. Active Learning • Probability of going from one state to another now depends on action • ADP equations are now:

16. Active Learning • Active Learning with Temporal Difference Learning: works the same way (assuming you know where you’re going) • Also need to learn probabilities to eventually make decision on where to go

17. Exploration: where should agent go to learn utilities? • Suppose you’re trying to learn optimal game playing strategies • Do you follow best utility, in order to win? • Do you move around at random, hoping to learn more (and losing lots in the process)? • Following best utility all the time can get you stuck at an imperfect solution • Following random moves can lose a lot

18. Where should agent go to learn utilities? • f(u,n) = exploration function • depends on utility of move (u), and number of times that agent has tried it (n) • One possibility: instead of using utility to decide where to go, use • Try a move a bunch of times, then eventually settle

19. Q-learning • Alternative approach for temporal difference learning • No need to learn probabilities: considered more desirable sometimes • Instead, looking for “quality” of (state, action) pair

20. Generalization in Reinforcement Learning • Maintaining utilities for all seen states in a real game is intractable. • Instead, treat it as a supervised learning problem • Training set consists of (state, utility) pairs • Or, alternatively, (state, action, q-value) triples • Learn to predict utility from state • This is a regression problem, not a classification problem • Radial basis function neural networks (hidden nodes are Gaussians instead of sigmoids) • Support vector machines for regression • Etc…

21. Other applications • Applies to any situation where something is to learn from reinforcement • Possible examples: • Toy robot dogs • Petz • That darn paperclip • “The only winning move is not to play”