1 / 21

Bootstrap for Goodness of Fit

Bootstrap for Goodness of Fit. G. Jogesh Babu Center for Astrostatistics http://astrostatistics.psu.edu. Astrophysical Inference from astronomical data. Fitting astronomical data Non-linear regression Density (shape) estimation Parametric modeling Parameter estimation of assumed model

fabrizio
Download Presentation

Bootstrap for Goodness of Fit

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Bootstrap for Goodness of Fit G. Jogesh Babu Center for Astrostatistics http://astrostatistics.psu.edu

  2. Astrophysical Inference from astronomical data Fitting astronomical data • Non-linear regression • Density (shape) estimation • Parametric modeling • Parameter estimation of assumed model • Model selection to evaluate different models • Nested (in quasar spectrum, should one add a broad absorption line BAL component to a power law continuum) • Non-nested (is the quasar emission process a mixture of blackbodies or a power law?) • Goodness of fit

  3. Chandra X-ray Observatory ACIS data COUP source # 410 in Orion Nebula with 468 photons Fitting to binned data using c2(XSPEC package) Thermal model with absorption, AV~1 mag

  4. Fitting to unbinned EDF Maximum likelihood (C-statistic) Thermal model with absorption

  5. Empirical Distribution Function

  6. Incorrect model family Power law model, absorption AV~1 mag Question : Can a power law model be excluded with 99% confidence?

  7. K-S Confidence bands F=Fn +/- Dn(a)

  8. Model fitting Find most parsimonious `best’ fit to answer: • Is the underlying nature of an X-ray stellar spectrum a non-thermal power law or a thermal gas with absorption? • Are the fluctuations in the cosmic microwave background best fit by Big Bang models with dark energy or with quintessence? • Are there interesting correlations among the properties of objects in any given class (e.g. the Fundamental Plane of elliptical galaxies), and what are the optimal analytical expressions of such correlations?

  9. Statistics Based on EDF Kolmogrov-Smirnov:supx |Fn(x) - F(x)|, supx (Fn(x) - F(x))+, supx (Fn(x) - F(x))- Cramer - van Mises: Anderson - Darling: All of these statistics are distribution free Nonparametric statistics. But they are no longer distribution free if the parameters are estimated or the data is multivariate.

  10. KS Probabilities are invalid when the model parameters are estimated from the data. Some astronomers use them incorrectly. (Lillifors 1964)

  11. Multivariate Case Warning: K-S does not work in multidimensions Example – Paul B. Simpson (1951) F(x,y) = ax2 y + (1 – a) y2 x, 0 < x, y < 1 (X1, Y1) data from F, F1EDF of(X1, Y1) P(| F1(x,y) - F(x,y)| < 0.72, for all x, y) is > 0.065ifa = 0,(F(x,y) = y2 x) < 0.058ifa = 0.5, (F(x,y) = xy(x+y)/2) Numerical Recipe’s treatment of a 2-dim KS test is mathematically invalid.

  12. Processes with estimated Parameters {F(.; q): q e Q} - a family of distributions X1, …, Xn sample from F Kolmogorov-Smirnov, Cramer-von Mises etc., when q is estimated from the data, are Continuous functionals of the empirical process Yn (x; qn) = (Fn (x) – F(x; qn))

  13. In the Gaussian case, q = (m,s2) and

  14. Bootstrap Gn is an estimator of F, based on X1, …, Xn X1*, …, Xn* i.i.d. from Gn qn*= qn(X1*, …, Xn*) F(.; q) is Gaussian with q = (m, s2) and , then Parametric bootstrap if Gn =F(.; qn) X1*, …, Xn* i.i.d. from F(.; qn) Nonparametric bootstrap if Gn =Fn (EDF)

  15. Parametric Bootstrap X1*, …, Xn*sample generatedfromF(.; qn). In Gaussian case . Both supx |Fn (x) – F(x; qn)|and supx |Fn* (x) – F(x; qn*)| have the same limiting distribution (In the XSPEC packages, the parametric bootstrap is command FAKEIT, which makes Monte Carlo simulation of specified spectral model)

  16. Nonparametric Bootstrap X1*, …, Xn*i.i.d. fromFn. A bias correction Bn(x) = Fn (x) – F(x; qn) is needed. supx |Fn (x) – F(x; qn)|and supx |Fn* (x) – F(x; qn*) - Bn (x) | have the same limiting distribution (XSPEC does not provide a nonparametric bootstrap capability)

  17. Chi-Squaretype statistics – (Babu, 1984, Statistics with linear combinations of chi-squares as weak limit. Sankhya, Series A, 46, 85-93.) U-statistics – (Arcones and Giné, 1992, On the bootstrap of U and V statistics. Ann. of Statist., 20, 655–674.)

  18. Confidence limits under misspecification of model family X1, …, Xn data from unknown H. H may or may not belong to the family {F(.; q): q e Q}. H is closest to F(.; q0), in Kullback - Leibler information h(x) log (h(x)/f(x; q)) dn(x) 0 h(x) |log (h(x)| dn(x) < h(x) log f(x; q0) dn(x) = maxq h(x) log f(x; q) dn(x)

  19. For any 0 < a < 1, P( supx |Fn (x) – F(x; qn) – (H(x) – F(x; q0)) | <Ca*) a Ca* is the a-th quantile of supx |Fn* (x) – F(x; qn*) – (Fn (x) – F(x; qn)) | This provide an estimate of the distance between the true distribution and the family of distributions under consideration.

  20. References • G. J. Babu and C. R. Rao (1993). Handbook of Statistics, Vol 9, Chapter 19. • G. J. Babu and C. R. Rao (2003).Confidence limits to the distance of the true distribution from a misspecified family by bootstrap.   J. Statist. Plann. Inference 115, 471-478. • G. J. Babu and C. R. Rao (2004). Goodness-of-fit tests when parameters are estimated.   Sankhya, Series A,66 (2004) no. 1, 63-74.

  21. The End

More Related