Rating Systems Vs Machine Learning on the context of sports

1. Rating Systems Vs Machine Learning on the context of sports George Kyriakides, Kyriacos Talattinis, George Stefanides Department of Applied Informatics, University Of Macedonia

2. Aim of the paper • Study the performance of linear algebra rating systems and machine learning methods. • Evaluate the accuracy of each method. • Evaluate the quality of the predictions.

3. Structure of the presentation • Clarify what is rating and ranking. • Explain linear algebra rating systems used in this paper. • Explain machine learning methods used in this paper. • Compare the rating systems and machine learning methods by predicting soccer games (English Premier League). • Conclusions. • Possible future work.

4. Related research • Nivard van Wijk uses a Pseudo Least-Squares Estimator to predict soccer matches. • Paul Kvam and Joel S. Sokol use logistic regression and markov chains to predict basketball matches. • Keeneth Massey uses linear algebra to rank basketball and football teams. • Search engines use machine learning to rank search results. • No comparison between machine learning and linear algebra has been made. • Studies on accuracy have been conducted for machine learning and linear algebra independently, but never for profitability of the methods.

5. Rating • The evaluation of an object, based on some desirable criteria. For example, a car may have a 1 to 5 stars NCAP safety rating.

6. Ranking • A relationship between a set of objects, such that for any two items, one is ranked higher than the other and is consequently better, assuming that the object ranked highest is the best. • Rating can be used to rank a set of objects. Sorting a vector containing ratings we are effectively ranking the elements.

7. Ranking

8. Rating Systems • Many systems have been proposed: • Massey • Colley • Markov • Keener • Google Page Rank • mHITS • Elo

9. Methods used in this paper • Massey • Colley • mHITS

10. Massey Method • Proposed by Kenneth Massey in 1997 to rank NCAA (National Collegiate Athletic Association)teams. • It uses a linear least squares regression to solve a system of linear equations. • A rating vector is calculated, where each entry corresponds to the rating of the team.

11. Massey Method (2) • First, the matrix M is generated , where Mij is the number of games teams i and j played, multiplied by -1 and Mii is the total number of games team i has played. • The second step is to calculate the vector p, where piis the total number of points scored by team i, minus the points that were scored against the team. • The final step is to solve the system of equations: Mr= p where r is the vector of the ratings, so if ri > rjteam i is better than team j.

12. Colley Method • Proposed by astrophysicist Dr.Wesley Colley in 2001. • Variation of a simple method used to rank teams, which calculated the win ratio of each team (wins divided by total games). • A system of linear equations is solved in order to find a rating vector.

13. Colley Method (2) • First the matrix C is computed as follows: Where nijis the total number of games played between teams i and j. • Second, the vector b is computed: -) Where wi are the total wins of team i and li are the total losses of team i. • Finally, the system of equations Cr=b is solved, where r is the vector with the rating for each team.

14. mHITS(Offence-Defense model) • It was proposed by Anjela Govan in 2009. • Generalization of HITS algorithm for ranking web pages. • It uses the offensive and defensive strength of the teams to calculate their overall rating. • The method calculates the rating of a team: Where ri is the rating of the team, oi is its offensive strength and di is its defensive strength.

15. mHITS(Offence-Defense model) (2) • Initialize vector d(0)=[d1 d2....dn] as a vector of ones. • Compute the matrix A, where aijis the score that team j generated against team i(0 if they did not play each other). • The third step of the initialization is to calculate • Continue to refine o and d, by continuously computing • High o values signify strong offence and low d values signify strong defense

16. Machine Learning

17. Methods used in this paper • Decision Trees • Artificial Neural Networks • Random Forests

18. Artificial Neural Networks • Networks of nodes which accept inputs and produce an output based on an activation function. • Nodes’ connections are weighted. • Usually organized in layers . • Each layer is a group of nodes not connected to any node of the same group. • Input data is presented to the input layer, “hidden” layers process the data and the presentation layer outputs the results.

20. Artificial Neural Networks • Multilayer Perceptron, which uses back propagation was used in the paper. • Learning rate:0.3 • Momentum:0.2 • Hidden layers:3 • Epochs:500 • 10-Fold Cross-Validation

21. Decision Tree Learning • Uses a decision tree to classify/predict. • Also known as classification/regression trees. • Leaves represent class labels. • Branches split the data into appropriate sets. • Different algorithms use different criteria to split the data.

22. C4.5 • Uses Information Gain to measure the quality of the split. • Confidence factor: 0.7 • Unpruned • 10-Fold Cross-Validation

23. Random Forest • A multitude of decision trees is generated • Each tree is trained independently, using a different subset of the data • Input data is presented to all trees • The class that the highest percentage of trees produce is the output

24. Soccer

25. Soccer • Ternary Result

26. Predictions • Hindsight • Foresight • Betting

27. Implementation • Weka • Open-source GNU GPL • Data mining software • Implemented in Java • Association rules, Classification, Clustering • Custom Java code • mHits • Colley • Massey • Jamma • Numerical linear algebra library

28. Hindsight Accuracy

29. Hindsight (2)

30. Confusion Matrices • Confusion matrices allow the visualization of the performance of an algorithm • It indicates the quality of the predictions • 95% Accuracy:

31. Confusion Matrices (Hindsight) Random Forest Decision Tree ANN 94.74% 72.11% 56.32%

32. Foresight Accuracy

33. Foresight (2)

34. Confusion Matrices (Foresight) Random Forest Decision Tree ANN 37.89% 46.84% 46.84%

35. Making a profit Quantity VS Quality

36. Making a profit • Quantity. • 90% Accuracy – 1.05 average booking odds • In 100 games, betting 1 unit each time: • Win 90*0.05 = 4.5 • Lose 10 • Net profit = 10-4.5 = -5.5

37. Making a profit • Quality • 60% Accuracy – 2.5 average booking odds • In 100 games, betting 1 unit each time: • Win 60*1.5 = 90 • Lose 40 • Net profit = 90 - 40 = 50

38. Betting • The average odds of the 5 biggest online booking companies. • 1000 money units starting capital. • 50 units betted each time. • Each method chose the outcome it thought would occur.

39. Foresight Betting Money Sum

40. Foresight Betting Net Profits sum

41. Conclusions • Machine learning proved to be superior in hindsight predictions and prediction quality. • Draws are the most difficult to predict. • mHITS is the best in foresight prediction accuracy, but not in quality. • Neural Networks are the most profitable of all. • Random Forests constructed the best hindsight models.

42. Future Work • Other Machine Learning Methods • Portfolio management for betting section • Other sports, where a draw is much less likely to be the outcome

43. Thank you very much for your attention!