Learning Positional Features for Annotating Chess Games : A Case Study

1. CG 2008 LearningPositionalFeaturesforAnnotatingChessGames: A CaseStudy Matej Guid, Martin Mozina, Jana Krivec, Aleksander Sadikov and Ivan Bratko Faculty of Computer and Information Science University of Ljubljana, Slovenia

2. Enhanced Knowledge in Chess Programs • The improvements of Rybka 3 in evaluation function for chess middlegames: • understanding good and bad bishops, • much more advanced understanding of kings attacks, • importance of defending king by pieces, • “space” (previously addresed only indirectly) (Larry Kaufmann, designer of Rybka’s evaluation function, RybkaForum.net) • Knowledge is particularly important when annotating chess games… How to “teach” a program to recognise chess concepts?

3. The Concept of the Bad Bishop • Chess experts in general understand the concept of bad bishop. • Preciseformalisation of this concept is difficult. Traditional definition (John Watson, Secrets of Modern Chess Strategy, 1999) • A bishop that is on the same colour of squares as its own pawns is bad: • its mobility is restricted by its own pawns, • it does not defend the squares in front of these pawns. • Moreover, centralisation of these pawns is the main factor in deciding whether the bishop is bad or not.

4. CRAFTY's Features for Describing Bishops

5. CRAFTY's Features for Describing Bishops

6. CRAFTY's Features for Describing Bishops

7. Static Nature of PositionalFeatures • Positional features in evaluation functions are static in their nature • Heuristic search enables to fulfil their purpose – contributing to the program finding best moves

8. Static Nature of PositionalFeatures • Positional features in evaluation functions are static in their nature • Heuristic search enables to fulfil their purpose – contributing to the program finding best moves Is freeing move e6-e5 possible or not? • It is also desirable for positional features for annotating chess games to be of static nature! Heuristic search is important for instructive annotations. • see the paper Automated chess tutor.CG 2006, Turin, Italy, May 29-31, 2006.

9. Expert-Crafted Rules & Introducing Learning • It is extremely difficult for chess experts to define appropriate rules for determining bad bishops -> knowledge acquisition bottleneck • various expert-crafted rules performed rather poorly (only 59% of examples were correctly classified) -> motivation for introducing machine learning Chess experts: wGM Jana KrivecFM Matej Guid Data Set for Machine Learning • Data set: 200 middlegamepositionsfrom real chessgames: • bad: 78 bishops • not bad: 122 bishops • We randomly selected100 positions forlearning and 100 positions for testing. • CRAFTY’s positional feature valuesserved asattribute values for learning.

10. Bad Bishops from Static Point of View • Our goal: construct a static positional feature BAD_BISHOP • Assessing bishops “statically” is counter-intuitive from chess point of view • Special guidelines were chosen for determining “statically” bad bishops… Guidelines for Determining Bad Bishops from Static Point of View The bishop is bad from the static point of view in some position, if its improvement or exchange is desirable the pawn structurelimits thebishop’s chances for taking an active part in the game, its mobility in this position is limited or not important for the evaluation.

11. Standard Machine Learning Methods' Performance with CRAFTY's features only • Machine learning methods’ performance on initial dataset • The results were obtained on test data set. • The results obtained with CRAFTY’s positional features only aretoo inaccurate for commenting purposes… • additional information for describing bad bishops is necessary. Improved Method • see the paper Fighting Knowledge Acquisition Bottleneck with Argument Based Machine Learning.European Conference on Artificial Intelligence, Patras, Greece, July21-25, 2008.Available at: www.ailab.si/matej

12. First Critical Example • Rules obtained by ABML method ABCN2failed to classify this exampleas "not bad“. • The following question was given to the experts: “Why is the black bishop not bad?“ • The experts used their domain knowledge: “The black bishop is not bad, since its mobility is not seriously restricted by the pawns of both players.”

13. Introducing new attribute into the domain and adding argument to an example • Experts’ explanation could not be described with current domain attributes. • A new attribute,IMPROVED_BISHOP_MOBILITY, was included into the domain: • number of squares accessible to the bishop, taking into account only own and opponent’s pawn structure • The argument “BISHOP=“not bad” because IMPROVED_BISHOP_MOBILITY is high“ was added to the example.

14. Another Critical Example • The ABML-based knowledge elicitation process can be used to induce rules to determine both good (i.e., not bad) and bad bishops. • Now the following question was given to the experts: “Why is the black bishop bad?“ • The experts used their domain knowledge: “The black bishop is bad, since black pawns block its diagonals.”

15. Introducing new attribute into the domain and adding argument to an example • Experts’ explanation could not be described with current domain attributes. • A new attribute,BLACK_PAWN_BLOCKS_BISHOP_DIAGONAL, was included into the domain: • pawns on the color of the square of the bishop that block the bishop's (front) diagonals • weights of such pawns were taken into account (see the “Look-up table” that follows) • The argument “BISHOP=“bad” because BLACK_PAWN_BLOCKS_BISHOP_DIAGONAL is high“ was added to the example.

16. Counter example • Methodfailed to explain critical example with given argument. • Counter example was presented to experts: Critical example: “not bad”, BLACK_PAWN_BLOCKS_BISHOP_DIAGONAL is high. Counter example: “bad”, although BLACK_PAWN_BLOCKS_BISHOP_DIAGONAL is high. • "Why is the “green” bishop not bad, compared to the“red” one?" • Experts’ explanation: “The black bishop is not bad, since its mobility is not seriously restricted, taking the pawn structure into account.”

17. Improving Arguments with Counter Examples • The argument given to the critical example was extended to“BISHOP=“bad” because BLACK_PAWN_BLOCKS_BISHOP_DIAGONAL is high and IMPROVED_BISHOP_MOBILITY is low.” • With this argument the method could not find any counter examples anymore. • The new rule covering the critical example became: ifBAD_PAWNS_BLOCK_BISHOP_DIAGONAL > 16 andIMPROVED BISHOP MOBILITY< 4 thenBAD_BISHOP= “bad” class distribution [42,0]

18. Assesing “Bad” Pawns • Experts designed a look-up table (left) with predefined values for pawns that are on color of square of the bishop in order to assign weights to such pawns. BAD_PAWNS_AHEAD = 16 + 24 + 2 = 42

19. After the Final Iteration... • The whole process consisted of 8 iterations. • 7 argumentswereattached to automaticallyselectedcriticalexamples • 5 new attributeswere included into the domain

20. Classification Accuracy Through Iterations • The accuracies of all methods improvedby adding new attributes. • ABCN2(which alsoused the arguments)outperformed all others. Arguments suggested useful attributes AND lead to more accurate models.

21. The Final Model • Rule #2: ifBAD_PAWNS_AHEAD > 18 andIMPROVED_BISHOP_MOBILITY < 3 thenBISHOP= “bad” class distribution [46,0]

22. The Final Model • Possible comment: “Black bishop is bad, since black pawns on the same colour ofsquares ahead of it, and pawns of both opponents restrict its mobility.”

23. Conclusions In thiscasestudy, Argument BasedMachineLearningwasshown to be apowerful knowledge-elicitation methodfordefiningdeeppositional patterns. ABMLoffersthefollowingadvantagesforknowledgeelicitation: easier for experts to articulate knowledge explain single example expert provides only relevant knowledge critical examples hypotheses are consistent with expert knowledge arguments constrain learning