A Scalable Machine Learning Approach to Go

1. A Scalable Machine Learning Approach to Go Pierre Baldi and Lin Wu UC Irvine

2. Contents • Introduction on Go • Existing approaches • Our approach • Results • Conclusion & Future work

3. What is Go?

4. What is Go? • Black & white play alternatively • Stones with zero liberty will be removed • The one who has more territory wins

5. Why is Go interested? • Go is a hard game for computer. • The best Go computer programs are easily defeated by an average human amateur • Board games have expert-level programs • Chess: Deep blue (1997) & FRITZ (2002) • Checker: Chinook (1994) • Othello (Reversi): Logistello (2002) • Backgammon: TD-GAMMON (1992)

6. Why is Go interested for AI? • Poses unique opportunities and challenges for AI and machine learning • Hard to build high quality evaluation function • Big branching factor, 200-300, compared with 35-40 for chess

7. Existing approaches • Hard-coded programs • Evaluate the next move by playing large number of random games • Use machine learning to learn the evaluation functions

8. Existing approaches── hard-coded programs • Hand-tailored pattern libraries • Hard-coded rules to choose among multiple hits • Tactical search (or reading) • E.g. “Many Faces of Go”, “GnuGo”

9. Existing approaches── hard-coded programs • Pros: • Good performance • Cons: • Intensive manual work • Pattern library is not complete • Hard to manage and improve

10. Existing approaches── Random games • Play huge number of random games from given position • Use the results of games to evaluate all the legal moves • Choose the legal move with best evaluation • E.g: Gobble, Go81

11. Existing approaches── Random games • Pros • Easy to implement • Reasonable performance • Cons • Small boards only, cannot scale to normal board

12. Existing approaches── Machine learning • Schraudolph et al., 1994 • TD0 • Neural Network • Graepel et al., 2001 • Condensed graph by common fate property • SVM • Stern, Graepel, and MacKay, 2005 • Conditional Markov random field

13. Existing approaches── Machine learning • Pros: • Learn automatically • Cons: • Poor performance

14. Out approach • Use scalable algorithms to learn high quality evaluation functions automatically • Imitate human evaluating process

15. Our approach── Human evaluating process • Three key components • The understanding of patterns • The ability to combine patterns • The ability to relate strategic rewards to tactical ones

16. Our approach── System components • 3x3 pattern library • Learn tactical patterns automatically • A structure-rich Recursive Neural Network • Propagate interaction between patterns • Learn the correlation between strategic rewards (Targets) and tactical reward (Inputs)

17. Our approach── RNN architecture • Six planes • One input plane • One output plane • Four Hidden Planes

18. Our approach── Update sequence

19. Our approach── Provide relevant inputs • For intersections • Intersection type: black, white, or empty • Influence: influence from the same & opposite color • Pattern stability: a statistical value calculated from 3x3 patterns • For groups • Number of eyes • Number of 1st, 2nd, 3rd, and 4th order liberties • Number of liberties of the 1st and 2nd weakest opponents

20. Our approach── Pattern stability (I) • 9x9 board is split into 10 unique locations for 3x3 patterns with mirror and rotation symmetries considered • Stability is measured for each intersection of each pattern within each unique location.

21. Our approach── Pattern stability (II) • Ten unique pattern locations

22. Our approach── Pattern stability (III)

23. Our approach── Pattern stability results (I)

24. Our approach── Pattern stability results (II)

25. Results── Validation error

26. Results── Results on move predictions

27. Results── Matched move (I)

28. Results── Matched move (II)

29. Conclusion & Future work