1 / 48

Model Selection in Semiparametrics and Measurement Error Models

Model Selection in Semiparametrics and Measurement Error Models. Raymond J. Carroll Department of Statistics Faculty of Nutrition and Toxicology Texas A&M University http://stat.tamu.edu/~carroll. Outline. Problem motivating the work Measurement error in nutritional epidemiology

Download Presentation

Model Selection in Semiparametrics and Measurement Error Models

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Model Selection in Semiparametrics and Measurement Error Models Raymond J. Carroll Department of Statistics Faculty of Nutrition and Toxicology Texas A&M University http://stat.tamu.edu/~carroll

  2. Outline • Problem motivating the work • Measurement error in nutritional epidemiology • Power of model selection/averaging • Difficulties with BIC • Semiparametric formulation • The Hjort and Claeskens asymptotics • Major results on semiparametric efficiency

  3. Co-authors, Nutritional Epidemiology Laurence Freedman, Gertner Institute (Israel) and National Cancer Institute Victor Kipnis National Cancer Institute

  4. Co-author, Semiparametrics Gerda Claeskens, University of Leuvan

  5. Papers • Seemingly Unrelated Measurement Error Models, Biometrics, 2006, Victor Kipnis and Larry Freedman • Post-Model Selection Inference in Semiparametric Models, with Gerda Claeskens • This paper has been under review for 6 months, at a journal that just asked me to review a paper in 6 weeks. 

  6. Seemingly Unrelated Measurement Error Models (SUMEM) • Problem: Understand the properties of food frequency questionnaires (FFQ), the most used means of estimating nutrient intakes • Issue: True intake cannot be measured

  7. SUMEM • Notation: • Y = FFQ • X = true intake (Not Observable) • Z = other covariates • Goals: Estimate two important properties • For pre-study sample sizes • post-study relative risk and power

  8. The OPEN Study • I will use data from the OPEN Study • First large biomarker study for nutritional epidemiology • Biomarkers available: • Protein (urinary nitrogen) • Energy/Calories (Doubly labeled water) • Approximately 200 women in the study

  9. SUMEM: Model for Protein • Basic variance components model: • Error Model for Biomarker:

  10. SUMEM: Model for Protein • More Data: Along with Protein, we also measure Energy (calories) • Correlated: • Energy = Protein + Fat + Carbohydrates + Alcohol • Combine: We try to combine energy with protein

  11. SUMEM: Model for Protein • Variance components model with Energy: • Error Model for Biomarker: Note how this model predicts FFQ protein using true energy

  12. SUMEM: More pain, no gain • Variance components model with Energy: • No Gain: One can show that if you fit this measurement error model, • Marginal protein fit unaffected, identical estimates of correlation and attenuation Note how this model predicts FFQ protein using true energy

  13. SUMEM: More pain, no gain • Variance components model with Energy: • Intuition: To predict FFQ Protein: • If I tell you true protein intake, why should true energy intake matter? Note how this model predicts FFQ protein using true energy

  14. SUMEM: More pain, more gain? • Variance components model with Energy: • A reduced model: Full Model Reduced Model

  15. OPEN Biomarker Study Results • Variance Reduction: As if the sample size increased by a factor of 3 (s.e. from Burnham & Anderson)

  16. OPEN: Model Selection • AIC and Weights: Let be the likelihood ratio chi-squared statistic, full vs. reduced • Let dbe the difference in degrees of freedom • The AIC weight to the reduced model is • We can do a weighted average of the full and reduced model estimates

  17. OPEN: Model Selection • BIC and Weights: Let be the likelihood ratio chi-squared statistic, full vs. reduced • Let dbe the difference in degrees of freedom • The BIC weight to the reduced model is

  18. OPEN: Model Selection • In OPEN: the AIC weight to the reduced model is • We can do a weighted average of the full and reduced model estimates • In OPEN, this is obviously essentially the reduced model

  19. OPEN: Bootstrapping Bootstrap: resample the individuals Note how the bootstrap distribution of the AIC weights is nothing like what one would expect

  20. OPEN: Bootstrapping Note how the bootstrap distribution of the attenuation is much more variable than the Burnham and Anderson calculations suggest

  21. OPEN: Bootstrapping Note how the bootstrap distribution of the correlation is much more variable than the Burnham and Anderson calculations suggest

  22. OPEN Type Simulation • Setup: We used OPEN parameters, but put in a full model • n = 200, as in OPEN • BIC is supposed to be a consistent model selector, hence should have weights near 0. • AIC is not a consistent model selector

  23. Full Model Simulation The weights should be near zero, since the full model holds. Note how BIC is an utter disaster: it almost always selects the reduced model

  24. Full Model Simulation Correlation: Fitting the reduced model here results in a 30% bias This would mean a study would have only 55% of the sample size it needs to attain a desired power Bias here is important!

  25. Full Model Simulation Attenuation: Fitting the reduced model here results in a 30% bias

  26. Full Model Simulation Correlation: Fitting the reduced model here results in a 25% bias This would mean a study would have only 60% of the sample size it needs to attain a desired power Bias here is important!

  27. Full Model Simulation For AIC, the Burnham and Anderson standard error estimates are OK, but coverage is somewhat low For BIC, it is so badly biased that the coverage probabilities are < 50% in all cases

  28. Summary of Numerical Work • BIC: • seriously biased in the full model holds • Seems to love the reduced model • From the Biometrics article

  29. Oracle Estimators • BIC: • This is an example of an oracle estimator • Leeb and Poetscher state

  30. Oracle Estimators • Leeb and Poetscher go on to state • This seems to be too good to be true, and it is • It is a delusion to believe it carries statistical meaning • It is remarkable that some of the lessons learned from Hodges’ counterexample seem not to have been received in the model selection literature

  31. Asymptotic Framework • I will next describe some local-alternative asymptotics for semiparametric models • These go some way to understanding the problem with oracle estimators • Semiparametric generalization of the work of Hjort and Claeskens

  32. Semiparametric Framework • Likelihood Formulation • Example: Partially Linear Model

  33. Semiparametric Framework • Example: Heteroscedastic Linear Model

  34. Semiparametric Framework • Likelihood Formulation • Two models: • Full model • Reduced model

  35. Semiparametric Framework • Local misspecification assumption • Intuitively: even an oracle will have to think a bit to distinguish between the full and reduced models • Practically: the models are not extremely different

  36. Semiparametric Framework • Local misspecification assumption • Hjort and Claeskens: parametric models • We deal with semiparametric models

  37. Profile Methods • Likelihood Formulation • For every , do local likelihood nonparametric regression to obtain • Then maximize

  38. Semiparametric Information Bound • The semiparametric information is the semiparametric version of Fisher information • Many references, good one is Murphy and van der Vaart, JASA, 2002

  39. Semiparametric Information Bound • Standard Result: For any function , at the full model

  40. Semiparametric Information Bound • Non-Standard Result: we have to compute the limit distribution of the reduced model estimate at the full (local) model • Easy using contiguity arguments (LeCam’s 1st, 2nd and 3rd Lemma)

  41. Semiparametric Information Bound • Non-Standard Result: Note asymptotic bias

  42. Semiparametric Information Bound • Model Averaging Result: Consider a model averaged estimator with weights for the reduced model depending on the profile likelihood, e.g., BIC, AIC, etc. • Recall that • Let D be the limiting distribution of

  43. Semiparametric Information Bound • Model Averaging Result: Then the model average estimator has limiting distribution • This is exactly the same result as in Hjort and Claeskens, except their Fisher information matrix is replaced by the semiparametric information bound

  44. Semiparametric Information Bound • Inference: Hjort and Claeskens propose a method for inference that is asymptotically correct. • In numerous simulations and examples for AIC, we have found that it is essentially the same as full-model inference • BIC: In this local misspecification framework, BIC is an inconsistent model selector: it selects the reduced model with probability  1.

  45. Summary • SUMEM: Model averaging can, in real life, result in estimators that decrease variance by 2/3 while retaining robustness • Very important in measurement error problems, where information is sparse • Inference: The Burnham and Anderson standard error formulae are reasonably precise • Because of non-normal limit distributions, coverage probabilities are much less than nominal

  46. Summary • BIC and Oracles: Simply disastrous in our example • Lack of uniform consistency means the: “it sounds too good to be true” is correct • Bootstrap: Does not work • Not new, but striking numerics • Semiparametrics: • In the local framework, same as parametric methods with the semiparametric information bound replacing the Fisher information (I contiguity)

  47. Burnham and Anderson Formula • You want to estimate a standard errror for a model-average estimate of

  48. Bootstrap Problem • First two terms have same limit as the limit in the data world. • 3rd term is not  0, hence inconsistency

More Related