Why do Wouter (and ATLAS) put asymmetric errors on data points ?

1. What is involved in the CLs exclusion method and what do the colours/lines mean ? Why do Wouter (and ATLAS) put asymmetric errors on data points ? ATLAS J/Ψ peak (muons) Excluding SM Higgs masses LEP exclusion Tevatron exclusion

2. Why do you put an error on a data-point anyway ? ATLAS J/Ψ peak (muons) Estimate of underlying truth (model value)

3. Poisson distribution Probability to observe n eventswhen λare expected Poisson distribution Number of observed events λ=4.90 #observed Lambda hypothesis varying fixed

4. Poisson distribution: properties Poisson distribution http://www.nikhef.nl/~ivov/Statistics/Poisson.pdf properties (1) Mean: (2) Variance: (3) Most likely value: first integer ≤λ  the famous √N

5. Lambda known  expected # events λ=0.00 λ=1.00 λ=5.00 λ=4.90

6. Large number of events λ=40.0 Unfortunately this is not what you wanted to know … What you have: What you want:

7. From data to theory Likelihood: Poisson distribution “what can I say about the measurement (Number of observed events) given an expectation from an underlying theory ?” This is what you want to know: “what can I say about the underlying theory given my observation of a given number of events ?”

8. Nobs known (4)  information on lambda “Given a number of observed events (4):  what is the most likely / average / mean underlying true vanue of λ ?” Likelihood: P(Nobs=4|λ) λ (hypothesis) #observed Lambda hypothesis Normally you plot -2log(Likelihood) fixed varying

9. Properties of P(λ|N) for flat P(λ) http://www.nikhef.nl/~ivov/Statistics/Poisson.pdf Assuming P(λ) is flat properties (1) Mean: (2) Variance: (3) Most likely value: λmost likely = x

10. This is normally presented as likelihood curve Pdf for λ P(Nobs=4|λ) 68.4% λ (hypothesis) -2Log(Prob) -1.68 +2.35 Likelihood -2Log(P(Nobs=4|λ)) ΔL=+1 sigma: ΔL=+1 2.32 4.00 6.35

11. So, if you have observed 4 eventsyour best estimate for λ is … : ATLAS J/Ψ peak (muons)

12. CLS method http://www.nikhef.nl/~ivov/Statistics/thesis_I_v_Vulpen.pdf Chapter 7.4

13. Your Higgs analysis Scaled to correct cross-sections and 100 pb-1 SM+Higgs Higgs SM Higgs SM Discriminant variable Discriminant variable Hebben we nou de Higgs gezien of niet ? Can also be an invariant mass plot

14. Approach 1: counting Experiment 1 Experiment 2 tellen tellen Discriminant variable Discriminant variable

15. Expectations If the Higgs is there: On average 17.2 events If the Higgs is NOT there: On average 12.2 events SM SM + Higgs Experiment 1:11 events observed Experiment 2:17 events observed

16. Discovery - Only look at what you expect from Standard Model background • Given the SM expectation: if probability to observe as many events you have observed (or more) is smaller than 5.7 10-7  SM hypothesis is very unlikely  reject SM  discovery !

17. Test hypotheses: rules for discovery Integrate this plot SM SM + Higgs In the hypothesis that there is NO Higgs (SM hypothesis): What is the probability to observe as many events as I have observed …OR EVEN MORE If P < 5.7 10-7 reject SM P(N≥33|12.2) = 6.35 10-7 P(N≥34|12.2) = 2.24 10-7

18. Question 1: did you make a discovery ? See previous slide: Yes No Discovery No discovery

19. Question 2: did you expect to make a discovery: If the Higgs is there: On average 17.2 events If the Higgs is NOT there: On average 12.2 events If you observe exactly the number of events you expect (assuming the Higgs is there), it is not unlikely enough to be explained by the SM NO discovery expected SM SM + Higgs

20. Question 3: At what luminositydo you expect to make a discovery ? Lumi x 1 NSM = 12.2 NHiggs = 5.1 no SM + Higgs SM Lumi x 10 NSM = 122.0 NHiggs = 51.0 no SM SM + Higgs NSM = 152.5 NHiggs = 63.75 yes Lumi x 12.5

21. Discovery or not It is not likely you get exactly the number of events you expect.  You can be lucky … or unlucky.

22. From simple counting to the real thing in 3 steps 1) Introduce X (Likelihood ratio) test statistic 2) From simple counting to weighted counting (a real analysis) 3) Toy Monte-Carlo (fake experiments)

23. From simple counting to the real thing in 3 steps 1) Introduce X (Likelihood ratio) test statistic 2) From simple counting to weighted counting (a real analysis) 3) Toy Monte-Carlo (fake experiments)

24. Hypothesis testing: likelihood ratio Hypothesis 1: the Standard Model without the Higgs boson Hypothesis 2:the Standard Model with the Higgs boson Definieer een statistic (= variabele) die onderscheid maakt tussen de 2 hypotheses. Note: kan vanalles zijn: # events of Neural net output.  frequently used: X=-2ln(Q) Likelihood ratio Ex: counting experiment

25. Likelihood ratio: counting 14 events observed Counting experiment N events left after some a selection of cut on discriminant Variabele transformatie More SM+Higgs like More SM like Used in plots: 100.000 SM experiments Note: X = 0 means hypoteses equally likely 100.000 SM + Higgs experiments

26. Likelihood ratio: counting Counting experiment 15 events observed 14 events observed N events left after some a selection of cut on discriminant More SM+Higgs like More SM like Used in plots: 100.000 SM experiments Note: X = 0 means hypoteses equally likely 100.000 SM + Higgs experiments

27. From simple counting to the real thing in 3 steps 1) Introduce X (Likelihood ratio) test statistic 2) From simple counting to weighted counting (a real analysis) 3) Toy Monte-Carlo (fake experiments)

28. Likelihood ratio Counting experiment Weighted counting experiment Eveny event has a weight according to a NN output or discriminant called pi : Signal: S(pi) and Background B(pi) N events left after some a selection of cut on discriminant tellen B(pi) S(pi)+B(pi)

29. From simple counting to the real thing in 3 steps 1) Introduce X (Likelihood ratio) test statistic 2) From simple counting to weighted counting (a real analysis) 3) Toy Monte-Carlo (fake experiments)

30. Many possible experiments Experiment 1 Experiment 2 tellen tellen Discriminant variable Discriminant variable 1) Experiment condensed in 1 variable Note: Each experiment (read ATLAS) yields only ONE value of Q see 2 slides ago for counting example 2) Do Toy-MC experiments to study distribution of Q Note: Two distributions: for SM and SM+Higgs hypothesis

31. Toy Monte Carlo experiment λSM(i)+ λSM+Higgs(i) λSM(i) SM toy experiment: Draw for each bin i a random number from Poisson with μ= λSM (i) SM+Higgs toy experiment: Draw for each bin i a random number from Poisson with μ= λSM(i)+ λSM+Higgs(i)

32. The Higgs does not exist: 100,000 toy-experiments (SM) The Higgs exists: 100,000 toy-experiments (SM+Higgs)

33. With 1 and 2 sigma bands for SM hypothesis Note (again): each experiment will produce 1 (one) number in this plot

34. Different masses … different cross-sections Small Higgs cross-section Large Higgs cross-section Two hypotheses are more apart if: 1) cross-section of Higgs is larger 2) Higgs is more different from SM

35. dummy LEP plots Cross-section drops as function of mass LEP paper Fig 1 dummy dummy

36. Expectation for Q or -2ln (Q): toy experiments Clb = confidence level in the background Probability that background results in the numer observed or less SM SM+Higgs Probability that background resultsin the numer observed or (even) more If 1-CLb < 5.7 10-7 we can say we reject the SM hypothesis  discovery ! The famous 5 sigma

37. Discovery

38. Do you expect to discover Higgs with at this mass ? Average SM+Higgs experiment: 1-CLb = 2 10^-7 So yes, you expect to make a discovery IF 10xSM

39. The one 2-sigma is not the other 2-sigma 2.X sigma discrepancy at mh ~ 97 GeV Far away form what you expect from Higgs 1.X sigma away at mh = 114 GeV Exactly what you expect from Higgs No 5 sigma discovery  what Higgs hypotheses can we reject

40. No discovery No 5 sigma deviation found … what now ? Trying to say something on the hypothesis that the Higgs exists  exclusion

41. Exclusion - Look at what you expect from Standard Model +Higgs - Given the SM + Higgs expectation: if probability to observe as many events you have observed (or less) is smaller than 5%  SM+Higgs hypothesis is not very likely  reject SM+Higgs

42. Expectation for Q or -2ln (Q): toy experiments SM Probability that signal hypothesis results in the numer observed or less SM+Higgs Extra Normalisation: This is why it is called modified frequentist Cls = confidence level in the signal If CLs < 0.05 we are allowed to rejectthe SM+Higgs at 95% confidence level The famous 95% confidence level

43. Question 2: did you expect to be able to exclude ? CLs mean SM-only expeciment is 0.13  > 0.05 so NO !

44. Question 3: At what luminositydo you expect to make a discovery ? Lumi = 1x normal lumi CLs = 0.13  no exclusion for average SM-only experiment #SM = 100 #H = 10 Lumi = 2x normal lumi CLs = 0.034  exclusion for average SM-only experiment #SM = 200 #H = 20

45. A scan: 2 sigma up CLs = 0.66 CLs = 0.13 CLs = 0.046 1 sigma down Si: If you would have a 1 sigma downward fluctuation, i.e. you see less events than you expect there is less room for a SM+Higgs hypothesis. In this case you would have been able to exclude it. CLs CLs = 0.05 Luminosity / nominal luminosity You expect to be able to exclude at Lumi / Lumi nominal = 1.70

46. Question 4: At what Higgs xsdo you expect to make a discovery ? Higgs XS = 1x normal Higgs XS CLs = 0.13  no exclusion for average SM-only experiment #SM = 100 #H = 10 Higgs XS = 2x normal Higgs XS CLs = 0.006  exclusion for average SM-only experiment #SM = 100 #H = 20

47. A scan: 2 sigma up CLs = 0.66 CLs = 0.13 CLs = 0.046 1 sigma down CLs CLs = 0.05 Higgs XS / nominal Higgs XS You expect to be able to exclude at Higgs XS / Higgs XS nominal = 1.40

48. A projection along the CLs = 0.05 line At what Higgs XS scale factordo you expect to be able to exclude the Higgs hypothesis ? SM only (2 sigma up) SM only (1 sigma up) 1.4 SM only (mean) Higgs XS / nominal Higgs XS SM only (1 sigma down) SM only (2 sigma down) Nominal luminosity

49. You can now scan over Higgs masses 1.4 Higgs XS / nominal Higgs XS The important thing is of course what you actually measured

50. Finito!