Neural Networks and Classical Linear Regression

1. Neural Networks and Classical Linear Regression Szu Hua Huang Jianjun Luo Texas Tech University 6/10/2014

2. Contents • Neural Networks and Classical Linear Regression • Introduction Neural Networks • Neural Networks VS Classical Linear Regression • Case study

3. Neural Network - Neuron

4. STRUCTURE OF A NEURAL NETWORK Multilayer feedforward networks

5. A simple MLP (multilayer perceptron) Hidden Layer Output Layer Input Layer Weights(Regression Coefficients) Back-propagation Model Updating (Learning) 3 W13 1 W36 Node W14 W15 6 W46 Output Input 4 W23 W56 W24 2 Data from True Function W25 5 Errors Transfer Function, g(s)

6. Transfer Function

7. Neural Network VS Classical Regression Model Neural Network Classical Regression Linear Normality Constant variability of error terms Independence Assumption • Non-linear

8. Neural Network VS Classical Regression Model Neural Network Classical Regression Regression coefficient shows the effect • Weight estimates(regression coefficient) do not tell you the effect • No guarantee the best linear combination of parameter estimates

9. Neural Network VS Classical Regression Model Multi-collinearity Outlier or Influential No effect to Neural Network ? It does hurt the Classical Regression • No effect to Neural Network ? • It does hurt the Classical Regression

10. Overfitting Neural Network Classical Regression Not very sensitive to the given data. • Sensitive to the given data, too much flexibility to the underlying distribution of data. • Big Sample size can help to solve the problem of overfitting.

11. Assessment Neural Network Classical Regression R square • Optimization plot based on the updated weight estimates at each iteration of the iterative grip search routine. • Using valid data

12. Optimization plot • 1. When the new weights are only incrementally different from those of the preceding iteration • 2. When the misclassification rate reaches a required threshold • 3. When the limit on the number of runs is reached ASE training validation 0 5 10 15 20 Iteration

13. Supplement-Other Optimization Algorithms: • Newton • Quasi-Newton method • Levenberg-Marquardt • Gauss-Newton Method • etc Stanford Open Course – Machine Learning

14. B: Case Study • Dataset • The School Children Data Set from Lewis & Taylor “Introduction to Experimental Ecology” (1967) • Includes 126 male records • Variables: • Age (months) • height (inches) • weight (pounds) • Purpose • Predicting the weight of male school children based on their age and height. • Comparing neural networks with OLS

15. Exploration of the dataset

16. Classical Linear Regression Model procreg data=men; model weight=height age; output out=regout p=pred r=resid; run;

17. Output of OLS

18. Neural Network Model INPUT OUTPUT HIDDEN COMBINATION w1+w2S_Height+w3S_Age = H11 TRANSFORMATION tanh(H11)) =A Standardization H11 Height S_Height Weight COMBINATION w7+w8A+w9B=Weight S_Age Age Standardization TRANSFORMATION tanh(H12) =B COMBINATION w4+w5S_Height+w6S_Age = H12 H12 18

19. Neural Network in SAS • The SAS neural network procedure • PROC NEURAL • SAS Enterprise Miner • A visual programming with a GUI interface

20. Neural Network Modeling using SAS Enterprise Miner • To save time, I recorded the following video to show how to build the Neural Network Model with SAS Enterprise Miner. • In case you are interested, I uploaded this video to YouTube: http://www.youtube.com/watch?v=Bb3K7xAcJbk&feature=youtu.be

21. Neural Network Weight Estimates

22. Neural network or classical linear regression?

23. Comparing Neural Network and Classical Linear Regression predicted values

24. Output: Observed and Predicted Values of Male's Weight against Age

25. References Eric Roberts. Neural networks. Available online at: http://www-cs-faculty.stanford.edu/~eroberts/courses/soco/projects/neural-networks/ Jim Georges, 2009. Applied analytics using SAS Enterprise Miner 6.1 Course Notes. SAS Institute Inc. Lewis, T. and Taylor, L.R. 1967. Introduction to Experimental Ecology, Academic Press, Inc. Randall Matignon, 2005. Neural Network Modeling using SAS Enterprise Miner. AuthorHouse SAS Institute, 1999. SAS/STAT User’s Guide Version 8. Available online at: http://ciser.cornell.edu/sasdoc/saspdf/common/mainpdf.htm Sue Walsh, 2002. Applying Data Mining Techniques Using Enterprise Miner Course Notes. SAS Institute Inc. Wikipedia. Neural network. Available online at: http://en.wikipedia.org/wiki/Neural_network

26. Thank You!