290 likes | 529 Views
Polynomial Curve Fitting BITS C464/BITS F464. Navneet Goyal Department of Computer Science, BITS- Pilani , Pilani Campus, India. Polynomial Curve Fitting. Seems a very trivial concept!! Why are we discussing it in Machine Learning course? A simple regression problem!!
E N D
Polynomial Curve FittingBITS C464/BITS F464 NavneetGoyal Department of Computer Science, BITS-Pilani, Pilani Campus, India
Polynomial Curve Fitting • Seems a very trivial concept!! • Why are we discussing it in Machine Learning course? • A simple regression problem!! • It motivates a number of key concepts of ML!! • Let’s discover…
Polynomial Curve Fitting Observe Real-valued input variable x • Use x to predict value of target variable t • Synthetic data generated from sin(2π x) • Random noise in target values Target Variable Input Variable Reference: Christopher M Bishop: Pattern Recognition & Machine Leaning, 2006 Springer
Polynomial Curve Fitting N observations of x x = (x1,..,xN)T t = (t1,..,tN)T • Goal is to exploit training set to predict value of from x • Inherently a difficult problem Target Variable Data Generation: N = 10 Spaced uniformly in range [0,1] Generated from sin(2πx) by adding small Gaussian noise Noise typical due to unobserved variables Input Variable Reference: Christopher M Bishop: Pattern Recognition & Machine Leaning, 2006 Springer
Polynomial Curve Fitting • Where M is the order of the polynomial • Is higher value of M better? We’ll see shortly! • Coefficients w0 ,…wM are denoted by vector w • Nonlinear function of x, linear function of coefficients w • Called Linear Models Target Variable Input Variable Reference: Christopher M Bishop: Pattern Recognition & Machine Leaning, 2006 Springer
Sum-of-Squares Error Function Reference: Christopher M Bishop: Pattern Recognition & Machine Leaning, 2006 Springer
Polynomial curve fitting Reference: Christopher M Bishop: Pattern Recognition & Machine Leaning, 2006 Springer
Polynomial curve fitting • Choice of M?? • Called the model selection or model comparison Reference: Christopher M Bishop: Pattern Recognition & Machine Leaning, 2006 Springer
0th Order Polynomial Poor representations of sin(2πx) Reference: Christopher M Bishop: Pattern Recognition & Machine Leaning, 2006 Springer
1st Order Polynomial Poor representations of sin(2πx) Reference: Christopher M Bishop: Pattern Recognition & Machine Leaning, 2006 Springer
3rd Order Polynomial Best Fit to sin(2πx) Reference: Christopher M Bishop: Pattern Recognition & Machine Leaning, 2006 Springer
9th Order Polynomial Over Fit: Poor representation of sin(2πx) Reference: Christopher M Bishop: Pattern Recognition & Machine Leaning, 2006 Springer
Polynomial Curve Fitting • Good generalization is the objective • Dependence of generalization performance on M? • Consider a data set of 100 points • Calculate E(w*) for both training data & test data • Choose M which minimizes E(w*) • Root Mean Square Error (RMS) • Sometimes convenient to use as division by N allows us to compare different sizes of data sets on equal footing • Square root ensures ERMS is measure on the same scale ( and in same units) as the target variable t Reference: Christopher M Bishop: Pattern Recognition & Machine Leaning, 2006 Springer
Flexibility & Model Complexity • M=0, very rigid!! Only 1 parameter to play with!
Flexibility & Model Complexity • M=1, not so rigid!! 2 parameters to play with!
Flexibility & Model Complexity • So what value of M is most suitable? • Any Answers???
Over-fitting For small M(0,1,2) Inflexible to handle oscillations of sin(2πx) M(3-8) flexible enough to handle oscillations of sin(2πx) For M=9 Too flexible!! TE = 0 GE = high Why is it happening? Reference: Christopher M Bishop: Pattern Recognition & Machine Leaning, 2006 Springer
Polynomial Coefficients Reference: Christopher M Bishop: Pattern Recognition & Machine Leaning, 2006 Springer
Data Set Size M=9 - Larger the data set, the more complex model we can afford to fit to the data - No. of data pts should be no less than 5-10 times the no. of adaptive parameters in the model Reference: Christopher M Bishop: Pattern Recognition & Machine Leaning, 2006 Springer
Over-fitting Problem Should we limit the no. of parameters according to the available training set? Complexity of the model should depend only on the complexity of the problem! LSE represents a specific case of Maximum Likelihood Over-fitting is a general property of maximum likelihood Over-fitting Problem can be avoided using the Bayesian Approach!
Over-fitting Problem In Bayesian Approach, the effective number of parameters adapts automatically to the size of the data set In Bayesian Approach, models can have more parameters than the number of data points Reference: Christopher M Bishop: Pattern Recognition & Machine Leaning, 2006 Springer
Regularization Penalize large coefficient values Reference: Christopher M Bishop: Pattern Recognition & Machine Leaning, 2006 Springer
Regularization: Reference: Christopher M Bishop: Pattern Recognition & Machine Leaning, 2006 Springer
Regularization: Reference: Christopher M Bishop: Pattern Recognition & Machine Leaning, 2006 Springer
Regularization: vs. Reference: Christopher M Bishop: Pattern Recognition & Machine Leaning, 2006 Springer
Polynomial Coefficients Reference: Christopher M Bishop: Pattern Recognition & Machine Leaning, 2006 Springer
Take Away from Polynomial Curve Fitting • Concept of over-fitting • Model Complexity & Flexibility Will keep revisiting it from time to time…