Estimation As A Data Mining Task

1. Estimation As A Data Mining Task Theoretical Understanding Vinh Ngo & Mike Ellis

2. Introduction • Estimation • Predicting values not in predetermined categories • Three main techniques • Regression • Decision trees • Neural networks

3. Regression • Linear regression • Method of Least Squares • Easy technique to use • Excel “Data Analysis” • Excel chart example

4. Excel Linear Regression Example

5. Other Regression Forms • Multiple regression • Polynomial equation • Define new variables • Equation becomes

6. Decision Trees • Regression tree • Leaves → average values • Model tree • Leaves → linear regression models • Discretizing the data • Convert continuous data into discrete partitions • Threshold values

7. Threshold Values • Entropy • Measure of purity • Information gain • Expected reduction in entropy due to partitioning • Maximize for best threshold

8. CART Algorithm • Classification and Regression Tree • Grow a tree that overfits data • Prune the tree • Select best subtree

9. Decision Trees • Strengths • Understandable • Which fields are most important • Weaknesses • Intended for discrete data • Time to grow and prune tree

10. Comparison Example

11. Linear Regression Result PRP = - 56.1 + 0.049 MYCT + 0.015 MMIN + 0.006 MMAX + 0.630 CACH - 0.270 CHMIN + 1.46 CHMAX

12. Regression Tree

13. Model Tree

14. Side-By-Side Linear regression Regression tree PRP = • - 56.1 • + 0.049 MYCT • + 0.015 MMIN • + 0.006 MMAX • + 0.630 CACH • - 0.270 CHMIN • + 1.46 CHMAX Model tree

15. Simple Neural Network

16. Building the Neural Net • Recursive process • Assign initial weights • Run training values through network • Compare results to actual value • Backpropagation • Pass errors back through net • Incorrect node gets less influence • Military metaphor • Recurrent networks • Genetic algorithms • Simulated annealing

17. Neural Networks • Strengths • Accurate • Fast to use • Handle missing or corrupt data well • Weaknesses • Not intuitive • Don’t handle large numbers of predictors well • Data preprocessing

18. Neural Net Example • Four years of 30-minute trading data, 1985-1988 • 1986 & 1987 for training • 1985 for testing • USD/CHF • Single layer model • Input nodes: 7 • Hidden nodes: 7 • Two layer model • Input nodes: 7 • Hidden nodes: 5/2 • Output • Value between -1.0 and 1.0 • Rise or fall?

19. Accuracy of Models • Overfitting • Applies to all three • Independent test data • Statistical measures

20. Statistical Measures

21. Any questions?

22. Special Bonus Slide