Goodness of fit, confidence intervals and limits

1. fourth lecture Goodness of fit, confidence intervals and limits Jorge Andre Swieca School Campos do Jordão, January,2003

2. References • Statistical Data Analysis, G. Cowan, Oxford, 1998 • Statistics, A guide to the Use of Statistical Methods in the Physical Sciences, R. Barlow, J. Wiley & Sons, 1989; • Particle Data Group (PDG) Review of Particle Physics, 2002 electronic edition. • Data Analysis, Statistical and Computational Methods for Scientists and Engineers, S. Brandt, Third Edition, Springer, 1999

3. Limits “Tens, como Hamlet, o pavor do desconhecido? Mas o que é conhecido? O que é que tu conheces, Para que chames desconhecido a qualquer coisa em especial?” Álvaro de Campos (Fernando Pessoa) “Se têm a verdade, guardem-na!” Lisbon Revisited, Álvaro de Campos

4. Statistical tests How well the data stand in agreement with given predicted probabilities – hypothesis. null hypothesis H0 alternative function of measured variables: test statistics error first kind significance level error second kind power to discriminate against H1 power =

5. Neyman-Pearson lemma H0 signal H1 background Where to place tcut? 1-D: efficiency (and purity) m-D: def. of acceptance region is not obvious Neyman-Pearson lemma: highest power (highest signal purity) for a given significance level α determined by the desired efficiency region of t-space such that

6. 10 17 18 19 20 0 1 2 3 Goodness of fit how well a given null hypothesis H0 is compatible with the observed data (no reference to other alternative hypothesis) coins: N tosses, nh , nt= N - nh coin “fair’? H and T equal? binomial distribution, p=0.5 test statistic: nh N=20, nh=17 E[nh]=Np=10

7. Goodness of fit P-value: probability P, under H0, obtain a result as compatible of less with H0 than the one actually observed. P=0.0026 P-value is a random variable, α is a constant specified before carrying out the test Bayesian statistics: use the Bayes theorem to assign a probability to H0 (specify the prior probability) P value is often interpreted incorrectly as a prob. to H0 P-value: fraction of times on would obtain data as compatible with H0 or less so if the experiment (20 coin tosses) were repeated under similar circunstances

8. Goodness of fit “optional stopping problem” Easy to identify the region of values of t with equal or less degree of compatibility with the hypothesis than the observed value (alternate hypothesis: p ≠ 0.5)

9. Significance of an observed signal Whether a discrepancy between data and expectation is sufficiently significant to merit a claim for a new discovery signal event ns, Poisson variable νS background event nb, Poisson variable νb prob. to observe n events: experiment: nobs events, quantify our degree of confidence in the discovery of a new effect (νS≠0) How likely is to find nobs events or more from background alone?

10. Significance of an observed signal P(n>nobs)=1.7x10-4 Ex: expect νb=0.5, nobs= 5 this is not the prob. of the hypothesis νS=0 ! this is the prob., under the hypothesis νS=0, of obtaining as many events as observed or more.

11. Significance of an observed signal How to report the measurement? estimate of ν : • only two std. deviations from zero • impression that νS is not very incompatible • with zero misleading: yes: prob. that a Poisson variable of mean νb will fluctuate up to nobs or higher no: prob. that a variable with mean nobs will fluctuate down to νb or lower

12. Pearson’s test ni νi histogram of x with N bins construct a statistic which reflects the level of agreement between observed and expected histograms data aprox. gaussian, Poisson distributed with follow a distribution for N degrees of freedom • regardless of the distribution of x • distribution free larger larger discrepancy between data and the hypothesis

13. Pearson’s test (rule of thumb for a good fit)

14. Pearson’s test

15. Pearson’s test Poisson variable with Before ni dist. as multinomial with prob. Set ntot = fixed Not testing the total number of expected and observed Events, but only the distribution of x. large number on entries in each bin pi known Follows a distribution for N-1 degrees of freedom In general, if m parameters estimated from data, nd = N - m

16. Standard deviation as stat. error n observations of x, hypothesis p.d.f f(x;θ) ML: estimator for θ measurement analytic method RCF bound Monte Carlo graphical standard deviation repeated estimates each based on n obs.: estimator dist. centered around true value θ and with true estimated by and Most practical estimators: becomes approx. Gaussian in the large sample limit.

17. prob. α prob.β Classical confidence intervals n obs. of x, evaluate an estimator for a param. θ obtained and its p.d.f. (for a given θ unknown)

18. Classical confidence intervals prob. for estimator to be inside the belt regardless of θ monotonic incresing functions of θ

19. a: hypothetical value of for which a fraction of the repeated estimt. would be higher than the obtain. Classical confidence intervals Usually: central confidence interval

20. Classical confidence intervals Relationship between a conf. interval and a test of goodness of fit: test the hypothesys using having equal or less agreement than the result obtained P-value = α (random variable) and θ = a is specified Confidence interval: α is specified first, a is a random quantity depending on the data

21. Classical confidence intervals Many experiments: the interval would include the true value in It does not mean that the probability that the true value of is in the fixed interval is Frequency interpretation: is not a random variable, but the interval fluctuates since it is constructed from data.

22. Gaussian distributed Simple and very important application Central limit theorem: any estimator linear function of sum of random variables becomes Gaussian in the large sample limit. known, experiment resulted in

23. Gaussian distributed

24. Gaussian distributed Choose quantile 1 0.6827 1 0.8413 2 0.9544 2 0.9772 3 0.9973 3 0.9987 Choose confidence level 0.90 1.645 0.90 1.282 0.95 1.960 0.95 1.645 0.99 2.576 0.99 2.326.