Comment on definition of likelihood ratio for limits

1. Comment on definition of likelihood ratio for limits ATLAS Statistics Forum CERN, 2 September, 2009 Glen Cowan Physics Department Royal Holloway, University of London g.cowan@rhul.ac.uk www.pp.rhul.ac.uk/~cowan Comment on use of LR for limits

2. Introduction At the Statistics Forum on 8.7.09 N. Andari presented a study (Orsay & Wisconsin) showing that a modified definition for the likelihood ratio leads to a sampling distribution that accurately follows the half-chi-square distribution. This offered the possibility to increase greatly the ease and accuracy with which we can compute exclusion limits, even for small samples. At the time some (at least GC and EG) did not fully understand how this could work, so we thought through a simple example. Conclusions agree with approach of Andari et al. Purpose of present talk is to present this example; see also attached note. Comment on use of LR for limits

3. The simple problem Suppose outcome of measurement is Gaussian distributed x with expectation value and variance s and b are contributions from signal and background, take s as a known constant; m is strength parameter. The likelihood function for the parameter of interest m is Suppose goal is to set an upper limit on m given a measurement x. Comment on use of LR for limits

4. The likelihood ratio To test a value of m, construct likelihood ratio: Suppose on physical grounds m should be positive, then the maximum of L(m) from the allowed range of m is from Usually use logarithmic equivalent -2 ln l(m): Comment on use of LR for limits

5. Likelihood ratio for upper limit For an upper limit on m one uses the test statistic which, putting together the ingredients, becomes Comment on use of LR for limits

6. p-value for exclusion To quantify level of agreement between observed x and hypothesized m, calculate p-value 95% CL upper limit on m is value for which pm = 0.05. Note if x ~ Gaussian(ms+b, s), then the quantity follows a chi-square pdf for 1 d.o.f. But the distribution of qm is more complicated (not chi-square), Comment on use of LR for limits

7. ^ Likelihood ratio without constraint on m Andari et al. propose to define an unphysical estimator which goes negative if x < b. Using this define then a corresponding test statistic for upper limits: Comment on use of LR for limits

8. Exclusion significance from qm′ From the definition of qm′ one can see its pdf must be a half-chi-square distribution, i.e., a delta function at zero when x > ms + b, and a chi-square pdf for x ≤ ms + b. Therefore (see CSC note), the significance from an observed value qm′ is given by the simple relation p-value of 0.05 corresponds to Z = 1.64. Comment on use of LR for limits

9. Comparison of test variables Both qm and qm′ are shown here as a function of x for m=1, s=10, b=20, s2=20. Note they are equal for b ≤ x ≤ ms + b. Comment on use of LR for limits

10. Equivalence of qm and qm′ It is easy to see that the two test variables qm and qm′ are monotonically related: and therefore they represent equivalent tests of m. Comment on use of LR for limits

11. Relation to study by Andari et al. (Eilam) Andari et al. presented the following table of the fraction of toy experiments with values of the test statistic below certain levels: Chi-square and “exact” (counting) formulae give same fractions for “median”, but not median +1s, +2s. This is because in this example it corresponds to having x < b, i.e., this where qm and qm′ are different. But for example, median - 1s or -2s would correspond to the region where qm and qm′ are equal, and the two methods there would agree. Comment on use of LR for limits

12. Conclusion on qm′ Both test variables, qm and qm′ , give equivalent tests, because of their monotonic relation. If one were to work out (with difficulty) the exact sampling pdf of qm, and to compute from it the p-value, and from it the significance Z, then it would be the same as from the simple formula Z = √ qm′ using the same value of the observation x. We still regard only positive m as physical, but allow its estimator to go negative effectively as a mathematical trick to get the desired p-value. Also easy to show that this likelihood ratio gives a test equivalent to the ratio used for the LEP analyses (see attached note): Comment on use of LR for limits

13. Extra slides Comment on use of LR for limits

14. Example from CSC book E.g. H → gg from the CSC combination chapter used the statistic qm (as did all other channels). MC studies show that distribution of qm departs significantly from half-chi-square form. If we had used qm′, the agreement with half-chi-square would be much closer, even for low luminosity. Comment on use of LR for limits