1 / 20

News from the Statistics Forum

News from the Statistics Forum. ATLAS Week Plenary Meeting CERN, 2 February, 2012. Glen Cowan (RHUL) Alex Read (Oslo). Overview. Summarize recent progress (meetings 8.11.11, 19.12.11, 9.1.12, plus very active email/hypernews discussions).

blair-cain
Download Presentation

News from the Statistics Forum

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. News from the Statistics Forum ATLAS Week Plenary Meeting CERN, 2 February, 2012 Glen Cowan (RHUL) Alex Read (Oslo) News from the Statistics Forum / 2 February, 2012

  2. Overview Summarize recent progress (meetings 8.11.11, 19.12.11, 9.1.12, plus very active email/hypernews discussions). Next Statistics Forum meeting is today (2.2.12), 17:00, Salle Dirac Main topics for today: Refinement for frequentist tests (“uncapped”p-values) Progress on Bayesian reference analyses ABCD method revisited News from the Statistics Forum / 2 February, 2012

  3. Aaron Armbruster, Tim Ayde Refinement for Frequentist Tests The usual statistic we define to test, e.g., the background-only hypothesis reflects disagreement with the hypothesis only if the data fluctuate higher than the expected background: This means that the p-value of the background-only hypothesis, under assumption of background only, is 0.5 ~half the time. Bug or feature? The interesting cases are when the p-value is small, e.g., p0 < 2.9 × 10-7 means we reject background-only hypothesis at 5σ level. But the plot of p0 versus mH comes out “capped” at 50%. News from the Statistics Forum / 2 February, 2012

  4. Aaron Armbruster, Tim Ayde Capped p-values Here data lower than expected background, p0 capped at 0.5 News from the Statistics Forum / 2 February, 2012

  5. Aaron Armbruster, Tim Ayde Similar capping for CLs News from the Statistics Forum / 2 February, 2012

  6. Aaron Armbruster, Tim Ayde Redefinition of test statistic Replace q0 by News from the Statistics Forum / 2 February, 2012

  7. Aaron Armbruster, Tim Ayde Uncapped p-values News from the Statistics Forum / 2 February, 2012

  8. Aaron Armbruster, Tim Ayde Summary on (un)capped p-values By modifying the test statistic as proposed, there is no change in cases where the p-value is less then 0.5, so no important conclusions change. But if the data fluctuate away from discovery or exclusion, the new statistic makes this easily visible. Some technical issues concerning asymptotic properties of q0′ still under study, but providing these are resolved we will propose to ATLAS (and CMS) that the “uncapped” method be adopted as part of the standard frequentist procedure. News from the Statistics Forum / 2 February, 2012

  9. Progress on Bayesian methods Although main emphasis has been on frequentist methods, also important progress on Bayesian techniques: E.g., many contributions to Statistics Forum on Bayesian Analysis Toolkit (BAT) and Bayesian tools within RooStats. News from the Statistics Forum / 2 February, 2012

  10. Bayesian priors For various reasons (historical,simplicity, conservatism,...) the prior for the mean number of signal events, s, is usually taken as constant: This has many well-known features: very conservative for limits, not invariant under reparametrization,... In the statistics community there is wide use of so-called “reference” priors. Recently Diego has developed a method to construct the reference prior for the signal parameter s in counting experiments: D. Casadei, Reference analysis of the signal + background model in counting experiments, JINST 7 (2012) P01012; arXiv:1108.4270 News from the Statistics Forum / 2 February, 2012

  11. Reference analysis à la Diego Find marginal model for the number of events, k, given a mean signal s and background b, by assuming b has a gamma prior: Use this model to compute the Fisher information: And then use this to find the reference prior for s: (same as the Jeffreys prior for single parameter): For details see Diego’s paper! News from the Statistics Forum / 2 February, 2012

  12. Example of reference priors π(s) for different gamma priors used for the background D. Casadei, JINST 7 (2012) P01012; arXiv:1108.4270 News from the Statistics Forum / 2 February, 2012

  13. Status of Bayesian Recommendation The current ATLAS recommendation on limits is to use (frequentist) CLs, except for analyses with a history of using Bayesian limits (and if CMS also follows this), in which case a constant prior is used for the signal rate. The reference prior has a number of important advantages: fast convergence of limit’s coverage probability to the (Bayesian) probability content; invariance under reparametrization,... We would like to encourage those people doing Bayesian analyses to try out the reference prior so that we can gain experience with it. News from the Statistics Forum / 2 February, 2012

  14. Alex, Eilam, Ohad,... Discussions on ABCD Method Basic idea: Measure x and y (~uncorrelated for bkg) Signal only in A; estimate background in A by: Refinements for correlations, spillover of signal into B, C, D,... Difficulty arises when using error propagation for error of background estimate if some of the boxes have few events. News from the Statistics Forum / 2 February, 2012

  15. Alex, Eilam, Ohad,... “Write down the likelihood” Treat the numbers of events found in A, B, C, D as independent and Poisson distributed; the likelihood is: For details see note by Alex on twiki (Statistics Tools in ATLAS): https://twiki.cern.ch/twiki/bin/viewauth/AtlasProtected/StatisticsTools Use the likelihood to test different values of the signal rate (e.g., profile likelihood ratio); or use in Bayesian analysis. Straightforward generalization to include further refinements (correlations, crossover, systematics, etc.) News from the Statistics Forum / 2 February, 2012

  16. Some other recent and ongoing discussions Unfolding Main effort in SM, exotics, top – B. Malaescu, F. Spano https://indico.cern.ch/conferenceDisplay.py?confId=157136 https://indico.cern.ch/conferenceDisplay.py?confId=156510 Plotting differences between data and expectation D. Casadei and G. Choudalakis, arXiv:1111.2062 Issues related to unexpectedly tight constraints on nuisance parameters from profile-likelihood method. Revisiting the “tevatron” likelihood ratio q = - 2 ln (Ls+b/Lb) (GC). Multiple regression (Samir Ferag), SusyFitter (Dan Short) Asymptotic combination of uncorrelated channels (Ohad Silbert) Your input is welcome! News from the Statistics Forum / 2 February, 2012

  17. Extra slides News from the Statistics Forum / 2 February, 2012

  18. Unfolding twiki page (SM & Top Groups) https://twiki.cern.ch/twiki/bin/viewauth/AtlasProtected/StandardModelUnfolding News from the Statistics Forum / 2 February, 2012

  19. Workshop on how to package our results News from the Statistics Forum / 2 February, 2012

  20. Reminder of Editorial Board Guidelines https://twiki.cern.ch/twiki/bin/viewauth/ AtlasProtected/EditorialBoardGuidelines Ensure the analysis follows the recommendations of the ATLAS Statistics Forum with regard to statistical procedures (e.g. limit setting, unfolding, combinations of data). In cases of doubt, or when complex or unusual procedures are being used, make sure the authors discuss with the Statistics Forum conveners before the analysis reaches the approval stage. News from the Statistics Forum / 2 February, 2012

More Related