Model Selection in Semiparametrics and Measurement Error Models

1. Model Selection in Semiparametrics and Measurement Error Models Raymond J. Carroll Department of Statistics Faculty of Nutrition and Toxicology Texas A&M University http://stat.tamu.edu/~carroll

2. Outline • Problem motivating the work • Measurement error in nutritional epidemiology • Power of model selection/averaging • Difficulties with BIC • Semiparametric formulation • The Hjort and Claeskens asymptotics • Major results on semiparametric efficiency

3. Co-authors, Nutritional Epidemiology Laurence Freedman, Gertner Institute (Israel) and National Cancer Institute Victor Kipnis National Cancer Institute

4. Co-author, Semiparametrics Gerda Claeskens, University of Leuvan

5. Papers • Seemingly Unrelated Measurement Error Models, Biometrics, 2006, Victor Kipnis and Larry Freedman • Post-Model Selection Inference in Semiparametric Models, with Gerda Claeskens • This paper has been under review for 6 months, at a journal that just asked me to review a paper in 6 weeks. 

6. Seemingly Unrelated Measurement Error Models (SUMEM) • Problem: Understand the properties of food frequency questionnaires (FFQ), the most used means of estimating nutrient intakes • Issue: True intake cannot be measured

7. SUMEM • Notation: • Y = FFQ • X = true intake (Not Observable) • Z = other covariates • Goals: Estimate two important properties • For pre-study sample sizes • post-study relative risk and power

8. The OPEN Study • I will use data from the OPEN Study • First large biomarker study for nutritional epidemiology • Biomarkers available: • Protein (urinary nitrogen) • Energy/Calories (Doubly labeled water) • Approximately 200 women in the study

9. SUMEM: Model for Protein • Basic variance components model: • Error Model for Biomarker:

10. SUMEM: Model for Protein • More Data: Along with Protein, we also measure Energy (calories) • Correlated: • Energy = Protein + Fat + Carbohydrates + Alcohol • Combine: We try to combine energy with protein

11. SUMEM: Model for Protein • Variance components model with Energy: • Error Model for Biomarker: Note how this model predicts FFQ protein using true energy

12. SUMEM: More pain, no gain • Variance components model with Energy: • No Gain: One can show that if you fit this measurement error model, • Marginal protein fit unaffected, identical estimates of correlation and attenuation Note how this model predicts FFQ protein using true energy

13. SUMEM: More pain, no gain • Variance components model with Energy: • Intuition: To predict FFQ Protein: • If I tell you true protein intake, why should true energy intake matter? Note how this model predicts FFQ protein using true energy

14. SUMEM: More pain, more gain? • Variance components model with Energy: • A reduced model: Full Model Reduced Model

15. OPEN Biomarker Study Results • Variance Reduction: As if the sample size increased by a factor of 3 (s.e. from Burnham & Anderson)

16. OPEN: Model Selection • AIC and Weights: Let be the likelihood ratio chi-squared statistic, full vs. reduced • Let dbe the difference in degrees of freedom • The AIC weight to the reduced model is • We can do a weighted average of the full and reduced model estimates

17. OPEN: Model Selection • BIC and Weights: Let be the likelihood ratio chi-squared statistic, full vs. reduced • Let dbe the difference in degrees of freedom • The BIC weight to the reduced model is

18. OPEN: Model Selection • In OPEN: the AIC weight to the reduced model is • We can do a weighted average of the full and reduced model estimates • In OPEN, this is obviously essentially the reduced model

19. OPEN: Bootstrapping Bootstrap: resample the individuals Note how the bootstrap distribution of the AIC weights is nothing like what one would expect

20. OPEN: Bootstrapping Note how the bootstrap distribution of the attenuation is much more variable than the Burnham and Anderson calculations suggest

21. OPEN: Bootstrapping Note how the bootstrap distribution of the correlation is much more variable than the Burnham and Anderson calculations suggest

22. OPEN Type Simulation • Setup: We used OPEN parameters, but put in a full model • n = 200, as in OPEN • BIC is supposed to be a consistent model selector, hence should have weights near 0. • AIC is not a consistent model selector

23. Full Model Simulation The weights should be near zero, since the full model holds. Note how BIC is an utter disaster: it almost always selects the reduced model

24. Full Model Simulation Correlation: Fitting the reduced model here results in a 30% bias This would mean a study would have only 55% of the sample size it needs to attain a desired power Bias here is important!

25. Full Model Simulation Attenuation: Fitting the reduced model here results in a 30% bias

26. Full Model Simulation Correlation: Fitting the reduced model here results in a 25% bias This would mean a study would have only 60% of the sample size it needs to attain a desired power Bias here is important!

27. Full Model Simulation For AIC, the Burnham and Anderson standard error estimates are OK, but coverage is somewhat low For BIC, it is so badly biased that the coverage probabilities are < 50% in all cases

28. Summary of Numerical Work • BIC: • seriously biased in the full model holds • Seems to love the reduced model • From the Biometrics article

29. Oracle Estimators • BIC: • This is an example of an oracle estimator • Leeb and Poetscher state

30. Oracle Estimators • Leeb and Poetscher go on to state • This seems to be too good to be true, and it is • It is a delusion to believe it carries statistical meaning • It is remarkable that some of the lessons learned from Hodges’ counterexample seem not to have been received in the model selection literature

31. Asymptotic Framework • I will next describe some local-alternative asymptotics for semiparametric models • These go some way to understanding the problem with oracle estimators • Semiparametric generalization of the work of Hjort and Claeskens

32. Semiparametric Framework • Likelihood Formulation • Example: Partially Linear Model

33. Semiparametric Framework • Example: Heteroscedastic Linear Model

34. Semiparametric Framework • Likelihood Formulation • Two models: • Full model • Reduced model

35. Semiparametric Framework • Local misspecification assumption • Intuitively: even an oracle will have to think a bit to distinguish between the full and reduced models • Practically: the models are not extremely different

36. Semiparametric Framework • Local misspecification assumption • Hjort and Claeskens: parametric models • We deal with semiparametric models

37. Profile Methods • Likelihood Formulation • For every , do local likelihood nonparametric regression to obtain • Then maximize

38. Semiparametric Information Bound • The semiparametric information is the semiparametric version of Fisher information • Many references, good one is Murphy and van der Vaart, JASA, 2002

39. Semiparametric Information Bound • Standard Result: For any function , at the full model

40. Semiparametric Information Bound • Non-Standard Result: we have to compute the limit distribution of the reduced model estimate at the full (local) model • Easy using contiguity arguments (LeCam’s 1st, 2nd and 3rd Lemma)

41. Semiparametric Information Bound • Non-Standard Result: Note asymptotic bias

42. Semiparametric Information Bound • Model Averaging Result: Consider a model averaged estimator with weights for the reduced model depending on the profile likelihood, e.g., BIC, AIC, etc. • Recall that • Let D be the limiting distribution of

43. Semiparametric Information Bound • Model Averaging Result: Then the model average estimator has limiting distribution • This is exactly the same result as in Hjort and Claeskens, except their Fisher information matrix is replaced by the semiparametric information bound

44. Semiparametric Information Bound • Inference: Hjort and Claeskens propose a method for inference that is asymptotically correct. • In numerous simulations and examples for AIC, we have found that it is essentially the same as full-model inference • BIC: In this local misspecification framework, BIC is an inconsistent model selector: it selects the reduced model with probability  1.

45. Summary • SUMEM: Model averaging can, in real life, result in estimators that decrease variance by 2/3 while retaining robustness • Very important in measurement error problems, where information is sparse • Inference: The Burnham and Anderson standard error formulae are reasonably precise • Because of non-normal limit distributions, coverage probabilities are much less than nominal

46. Summary • BIC and Oracles: Simply disastrous in our example • Lack of uniform consistency means the: “it sounds too good to be true” is correct • Bootstrap: Does not work • Not new, but striking numerics • Semiparametrics: • In the local framework, same as parametric methods with the semiparametric information bound replacing the Fisher information (I contiguity)

47. Burnham and Anderson Formula • You want to estimate a standard errror for a model-average estimate of

48. Bootstrap Problem • First two terms have same limit as the limit in the data world. • 3rd term is not  0, hence inconsistency