Proposal for alternative to CLs: Power-Constrained Limits

1. Proposal for alternative to CLs:Power-Constrained Limits ATLAS Statistics Forum CERN, 25 May, 2010 Glen Cowan, RHUL Kyle Cranmer, NYU Eilam Gross, Ofer Vitells, Weizmann Inst. Alternative to CLs

2. The “CLs” issue When the cross section for the signal process becomes small (e.g., large Higgs mass), the distribution of the test variable used in a search becomes the same under both the b and s+b hypotheses: f (q| b) f (q| s+b) In such a case we will reject the signal hypothesis with a probability approaching a = 1 – CL (i.e. 5%) assuming no signal. Alternative to CLs

3. The CLs solution The CLs solution (A. Read et al.) is to base the test not on the usual p-value (CLs+b), but rather to divide this by CLb (one minus the background of the b-only hypothesis, i.e., Define: f (q| b) f (q| s+b) q Reject signal hypothesis if: Reduces “effective” p-value when the two distributions become close (prevents exclusion if sensitivity is low). Alternative to CLs

4. Alternative proposal – basic idea CLs method reduces the p-value according to: where m = strength parameter, proportional to cross section. Statistics community does not smile upon ratio of p-values; would prefer to regard parameter m as excluded if: (a) p-value of m < 0.05 (b) power of test of m with respect to background-only > some threshold Requiring (a) alone gives the standard frequentist interval, (CLs+b method) which has the correct coverage. Requiring ANDed combination of (a) and (b) is more conservative; end effect is similar to CLs, but makes more explicit the minimum the role of minimum sensitivity (as quantified by power). Alternative to CLs

5. Similar to…. Feldman and Cousins touched on the same idea in connection with FC limits: We propose to make this more explicit using the power of the test of a given strength parameter m with respect to the alternative background-only hypothesis. Alternative to CLs

6. Formalizing the problem In the context of tests based on the likelihood ratio l(m), the p-value can be written Estimator for strength parameter Standard normal cumulative dist. The upper limit is found by setting pm = a and solving for m, Alternative to CLs

7. False exclusion rate for no sensitivity Excluding m if pm < a gives right coverage, but this means that probability to exclude m in case of no sensitivity is a. To see this note, probability to exclude m assuming m = 0 is “No sensitivity” means m /s « . In this limit, the false exclusion probability becomes Alternative to CLs

8. Power of test of m relative to m = 0 The power of a test of m relative to the alternative m = 0 is or equivalently in terms of the distribution of mup, Alternative to CLs

9. Criterion for rejecting m We formulate the criterion for rejecting a hypothesized m by Requiring pm < a and also that the power be greater than a minimum threshold 1 – b′. (i.e. Type-II error rate < b′ ). The power-constrained limit is thus where mb ′is the m for which the power is b′. The requirement implies so the minimum power requirement can be expressed or equivalently Alternative to CLs

10. Choice of minimum power Note that if the minimum power 1 – b′ = a (typically 0.05), then mb ′= 0, and then mpc = mup always. Normally would choose a < 1 – b′ ≤ 0.5. Convention must be discussed (also with CMS). Coverage of power-constrained interval is well defined: 95% for mpc = mup 100% for mpc < mup Alternative to CLs

11. Solution in terms of median p-value Because of the monotonic relation between the p-value and estimator for m, the median of pm assuming m = 0 is: In addition to the median (50% quantile) we can also find the quantiles corresponding to the +/-Ns deviations of muHat: Alternative to CLs

12. Extra slides Alternative to CLs

13. A possible experimental outcome Suppose a given experiment gave the following p-value versus m: Here data have clearly fluctuation low. Alternative to CLs

14. Choice of likelihood ratio statistic Ongoing discussion as to whether best to use LEP-style likleihood ratio or and in both cases how to deal with the nuisance parameters. In simple cases one obtains the same test from both statistics. Alternative to CLs