Friedman's SuperSmoother : Special Notes on Span

1. Friedman's SuperSmoother: Special Notes on Span IVAN MEJIA GUEVARA CONSEJO NACIONAL DE POBLACION CEPAL/IDRC PROJECT Honolulu, June 5 2008 National Transfer Accounts

2. Seoul’s Meeting: Smoothing • Some problems identified in the workshop: • Young ages. Many profiles are inherently not smooth at early ages • Old ages. The age profile estimates at old ages are often based on relatively few observations • Over smoothing.There is a natural tendency to over-smooth the data. The danger is that real and important fluctuations may be concealed. Smoothing too little is better than smoothing out real changes. National Transfer Accounts

3. Seoul’s Meeting: Smoothing Some problems identified in the workshop: National Transfer Accounts

4. Seoul’s Meeting • For most, but not all purposes, SUPSMU with a relatively narrow bandwidth (0.05 or less) appears to be a relatively reliable method National Transfer Accounts

5. Alternative Method using SUPSMU • This approach attempts to deal with the problem of selecting the most appropriate bandwidth (span) • The method might also be effective to deal with the problem of over smoothing • The method proposed relies on: • The use of statistical criteria • Semi-automatic selection of bandwidth National Transfer Accounts

6. Method Description • Definition of a metric must be defined in order to measure the difference between actual (unsmoothed) values (At) and output (smoothed) values (Ft) • Specification of span options • Compute the metric(s) defined in 1 • Apply the Majority Voting Scheme (MVS) as a selection criterion National Transfer Accounts

7. 1. Definition of a metric National Transfer Accounts

8. 2. Specification of span Options • Consider several span options from a list previously specified by the user. For example, the list {0.035, 0.04, 0.045, 0.05, 0.055, 0.06, 0.065, 0.07, 0.075, 0.08, 0.09, 0.1, ‘cv’} • Users are free to choose the range that they consider more appropriate for their particular situation National Transfer Accounts

9. 3. Compute the metric(s) defined in 1. • Compute the MSE, MAE and MAPE for every span in the list specified in 2. National Transfer Accounts

10. 4. Use of MVS as a selection criterion National Transfer Accounts

11. Implementation R code: ******SUPSMU******* aloha<-read.csv("earnings.csv", header=T) t1<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.035") t2<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.04") t3<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.045") t4<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.05") t5<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.055") t6<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.06") t7<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.065") t8<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.07") t9<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.075") t10<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.08") t11<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.09") t12<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.1") t13<-supsmu(aloha$age, aloha$income, aloha$sample, span = "cv") write.csv(c(t1,t2,t3,t4,t5,t6,t7,t8, t9,t10,t11,t12,t13), "smoothed_earnings") National Transfer Accounts

12. Implementation Excel Template: National Transfer Accounts

13. Discussion • Ad hoc is used to break ties where MVS cannot be applied • This method could be effective in practice for some profiles • A lot of time is saved by (semi) automatically selecting the span in this way • This method might be no appropriate when the researcher cares more about some parts of the age profile than others or when profiles present a lot of sharp elbows National Transfer Accounts

14. Example 1 National Transfer Accounts

15. Example 2 National Transfer Accounts

16. Wiki http://www.schemearts.com/proj/nta/web/nta/show/Documents/Smoothing/Friedman's%20SuperSmoother/Special%20Notes%20on%20Span National Transfer Accounts