1 / 12

Neural Networks and SVM

Neural Networks and SVM. Stat 600. Neural Networks. History: started in the 50s and peaked in the 90s Idea: learning the way the brain does. Numerous applications Handwriting, face, speech recognition Vehicles that drive themselves Models of reading, sentence production, dreaming.

pabla
Download Presentation

Neural Networks and SVM

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Neural Networks and SVM Stat 600

  2. Neural Networks • History: started in the 50s and peaked in the 90s • Idea: learning the way the brain does. • Numerous applications • Handwriting, face, speech recognition • Vehicles that drive themselves • Models of reading, sentence production, dreaming

  3. Non-linear Regression • At the end this is a non-linear regression problem. • Let us consider our usual data set: • Y (response, numerical or categorical) • X1,…,Xp my predictors • In the linear model we model Y as: • Y=Xb + e • Here we say that Y is a function of • Y = g(Ha) + e • Where h = f(Xb) • So essentially it is non-linear because of the functions g or f. • The function g is generally chosen as the logit transform, [1+e-z]-1

  4. Model Form

  5. Parameter Estimation • In order to control the level of overfitting we use penalized least squares which penalize for the overfit using a Ridge regression like squared error penalty. • The penalty is imposed NOT on the number of parameters but on the MAGNITUDE of the parameters. The criterion is given by:

  6. Rcode for NNet #neural networks Library(nnet) nnetmodel=nnet(class~.,data=train.all,size=8,decay=.2,linout=FALSE, entropy=TRUE) nnetmodel nnetpred1=predict(nnetmodel,newdata=train.all,type="class") nnetpred2=predict(nnetmodel,newdata=test.all,type="raw") table(nnetpred1,test.all$class) library(devtools) source_url('https://gist.github.com/fawda123/7471137/raw/c720af2cea5f312717f020a09946800d55b8f45b/nnet_plot_update.r') plot.nnet(nnetmodel)

  7. Example: Apple data

  8. Fitting Neural Networks • Generally the gradient descent method is used to fit the models where: • The gr is the learning rate taken as a constant and can be optimized by a line search that minimizes error function at each update.

  9. Issues • Starting Values: Pick weights close to zero to start the process • Overfitting: Ridge or other penalties are used • Scaling inputs: good idea to scale weights • Number of hidden layers: better to have too many than too few

  10. Support Vector Machines • Highly flexible, powerful modeling methods • Remember in linear regression we seek parameter estimates that minimize SSE, and a drawback is that outliers affect this minimization. • In Robust regression we use HUBER weights to minimize the effect of influential observations. • SVM for regression uses a similar function to Huber but with a difference. • In SVM (given the threshold) set by the researcher, data points with residuals within the threshold DO NOT contribute to the regression fit, while data points with absolute difference greater than the threshold contribute a linear scale amount. • Samples that fit the model well have NO effect on the regression. • If threshold is set high, ONLY the outliers affect the regression.

  11. SVM Estimation • To estimate the model parameters SVM uses a user specified loss function Le but also adds a penalty. • The SVM coefficients minimize: • The cost penalty is specified by the user and penalizes for a LARGE residual (this is opposite of Ridge regression and Nnet, which puts the penalty for large betas).

  12. Svm PLOT FOR PROTEIN DATA X1 TO X7

More Related