Regression / Calibration

1. Regression / Calibration MLR, RR, PCR, PLS

2. Paul Geladi Head of Research NIRCE Unit of Biomass Technology and Chemistry Swedish University of Agricultural Sciences Umeå Technobothnia Vasa paul.geladi@btk.slu.sepaul.geladi@syh.fi

3. Univariate regression

4. y Slope a Offset x

5. y e y = a + bx + e Slope b a Offset a x

6. y x

7. Linear fit Underfit y x

8. y Overfit x

9. y Quadratic fit x

10. Multivariate linear regression

11. y = f(x) Works sometimes y = f(x) Works only for a few variables Measurement noise! ∞ possible functions

12. K X y I

13. y = f(x) y = f(x) Simplified by: y = b0 + b1x1 + b2x2 + ... + bKxK + f Linear approximation

14. Nomenclature y = b0 + b1x1 + b2x2 + ... + bKxK + f y : response xk : predictors bk : regression coefficients b0 : offset, constant f : residual

15. K X y I X, y mean-centered b0 out

16. y = b1x1 + b2x2 + ... + bKxK + f y = b1x1 + b2x2 + ... + bKxK + f y = b1x1 + b2x2 + ... + bKxK + f } I samples y = b1x1 + b2x2 + ... + bKxK + f y = b1x1 + b2x2 + ... + bKxK + f

17. y = b1x1 + b2x2 + ... + bKxK +f y = b1x1 + b2x2 + ... + bKxK +f y = b1x1 + b2x2 + ... + bKxK +f y = b1x1 + b2x2 + ... + bKxK +f y = b1x1 + b2x2 + ... + bKxK +f

18. K y X f + = b I y=Xb+f

19. X, yknown, measurableb,funknownNo solutionfmust be constrained

20. The MLR solutionMultiple Linear RegressionOrdinary Least Squares (OLS)

21. b= (X’X)-1X’y Least squares Problems?

22. 3b1 + 4b2 = 1 4b1 + 5b2 = 0 One solution

23. 3b1 + 4b2 = 1 4b1 + 5b2 = 0 b1 + b2 = 4 No solution

24. 3b1 + 4b2 + b3 = 1 4b1 + 5b2 +b3 = 0 ∞ solutions

25. b= (X’X)-1X’y -K > I ∞ solutions -I > K no solution -error in X -error in y -inverse may not exist -inverse may be unstable

26. 3b1 + 4b2 + e = 1 4b1 + 5b2 + e = 0 b1 + b2 + e = 4 Solution

27. Wanted solution • - I ≥ K • No inverse • No noise in X

28. Diagnostics y=Xb+f SS tot = SSmod + SSres R2 = SSmod / SStot = 1- SSres / SStot Coefficient of determination

29. Diagnostics y=Xb+f SSres = f’f RMSEC = [ SSres / (I-A) ] 1/2 Root Mean Squared Error of Calibration

30. Alternatives to MLR/OLS

31. Ridge Regression (RR) b= (X’X)-1X’y I easiest to invert b= (X’X + kI)-1X’y k (ridge constant) as small as possible

32. Problems - Choice of ridge constant - No diagnostics

33. Principal Component Regression (PCR) • I ≥ K • Easy inversion

34. Principal Component Regression (PCR) A K X T PCA • - A ≤ I • T orthogonal • Noise in X removed

35. Principal Component Regression (PCR) y=Td+f d = (T’T)-1T’y

36. Problem How many components used?

37. Advantage - PCA done on data - Outliers - Classes - Noise in X removed

38. Partial Least SquaresRegression

39. X t u Y

40. X t u Y w’ q’ Outer relationship

41. X t u Y w’ q’ Inner relationship

42. A A X t u Y w’ q’ A A p’

43. Advantages - X decomposed - Y decomposed - Noise in X left out - Noise in Y left out

44. PCR, PLS are one component at a time methodsAfter each component, a residual is calculatedThe next component is calculatedon the residual

45. Another view y=Xb+f y=XbRR+fRR y=XbPCR+fPCR y=XbPLS+fPLS

47. Prediction

48. K Xcal ycal I Xtest yhat ytest J

49. Prediction diagnostics yhat = Xtestb ftest = ytest -yhat PRESS = ftest’ftest RMSEP = [ PRESS / J ] 1/2 Root Mean Squared Error of Prediction

50. Prediction diagnostics yhat = Xtestb ftest = ytest -yhat R2test = Q2 = 1 - ftest’ftest/ytest’ytest