BAYES and FREQUENTISM: The Return of an Old Controversy

1. BAYES and FREQUENTISM:The Return of an Old Controversy Louis Lyons Imperial College and Oxford University Gran Sasso Sept 2010

3. Topics • Who cares? • What is probability? • Bayesian approach • Examples • Frequentist approach • Systematics • Summary

4. It is possible to spend a lifetime analysing data withoutrealising that there are two very different fundamental approaches tostatistics: Bayesianism and Frequentism.

5. How can textbooks not even mention Bayes / Frequentism? For simplest case with no constraint on then at some probability, for both Bayes and Frequentist (but different interpretations) See Bob Cousins “Why isn’t every physicist a Bayesian?” Amer Jrnl Phys 63(1995)398

6. We need to make a statement about Parameters, Given Data The basic difference between the two: Bayesian : Probability (parameter, given data) (an anathema to a Frequentist!) Frequentist : Probability (data, given parameter) (a likelihood function)

7. PROBABILITY MATHEMATICAL Formal Based on Axioms FREQUENTIST Ratio of frequencies as n infinity Repeated “identical” trials Not applicable to single event or physical constant BAYESIANDegree of belief Can be applied to single event or physical constant (even though these have unique truth) Varies from person to person *** Quantified by “fair bet”

8. Bayesian versus Classical Bayesian P(A and B) = P(A;B) x P(B) = P(B;A) x P(A) e.g. A = event contains t quark B = event contains W boson or A = I am at Gran Sasso B = I am giving a lecture P(A;B) = P(B;A) x P(A) /P(B) Completely uncontroversial, provided….

9. Bayesian Bayes’ Theorem p(param | data) α p(data | param) * p(param) posterior likelihood prior Problems: p(param) Has particular value “Degree of belief” Prior What functional form? Coverage

10. P(parameter) Has specific value “Degree of Belief” Credible interval Prior: What functional form? Uninformative prior: flat? In which variable? Even more problematic with more params Unimportant if “data overshadows prior” Important for limits Subjective or Objective prior?

12. Prior

13. Prior = zero in unphysical region

14. Bayes: Specific example Particle decays exponentially: dn/dt = (1/τ) exp(-t/τ) Observe 1 decay at time t1: L(τ) = (1/τ) exp(-t1/τ) Choose prior π(τ) for τ e.g. constant up to some large τL Then posterior p(τ) =L(τ) * π(τ) has almost same shape as L(τ) Use p(τ) to choose interval forττ in usual way Contrast frequentist method for same situation later.

15. Bayesian posterior  intervals Upper limit Lower limit Central interval Shortest

16. Ilya Narsky, FNAL CLW 2000

17. P (Data;Theory) P (Theory;Data) HIGGS SEARCH at CERN Is data consistent with Standard Model? or with Standard Model + Higgs? End of Sept 2000: Data not very consistent with S.M. Prob (Data ; S.M.) < 1% valid frequentist statement Turned by the press into: Prob (S.M. ; Data) < 1% and therefore Prob (Higgs ; Data) > 99% i.e. “It is almost certain that the Higgs has been seen”

18. P (Data;Theory) P (Theory;Data) Theory = male or female Data = pregnant or not pregnant P (pregnant ; female) ~ 3%

19. P (Data;Theory) P (Theory;Data) Theory = male or female Data = pregnant or not pregnant P (pregnant ; female) ~ 3% but P (female ; pregnant) >>>3%

20. Example 1 : Is coin fair ? Toss coin: 5 consecutive tails What is P(unbiased; data) ? i.e. p = ½ Depends on Prior(p) If village priest: prior ~ δ(p = 1/2) If stranger in pub: prior ~ 1 for 0 < p <1 (also needs cost function)

21. Example 2 : Particle Identification Try to separate π’s and protons probability (p tag; real p) = 0.95 probability (π tag; real p) = 0.05 probability (p tag; real π) = 0.10 probability (π tag; real π) = 0.90 Particle gives proton tag. What is it? Depends on prior = fraction of protons If proton beam, very likely If general secondary particles, more even If pure π beam, ~ 0

22. Peasant and Dog • Dog d has 50% probability of being 100 m. of Peasant p • Peasant p has 50% probability of being within 100m of Dog d d p x River x =0 River x =1 km

23. Given that: a)Dog d has 50% probability of being 100 m. of Peasant, is it true that:b) Peasant p has 50% probability of being within 100m of Dog d ?

25. Classical Approach Neyman “confidence interval” avoids pdf for Uses only P( x; ) Confidence interval : P( contains ) = True for any Varying intervals from ensemble of experiments fixed Gives range of for which observed value was “likely” ( ) Contrast Bayes : Degree of belief = is in

26. μ≥0 No prior for μ

27. Frequentism: Specific example Particle decays exponentially: dn/dt = (1/τ) exp(-t/τ) Observe 1 decay at time t1: L(τ) = (1/τ) exp(-t1/τ) Construct 68% central interval t = .17τ dn/dt τ t t = 1.8τ t1 t

28. 90% Classical interval for Gaussian σ = 1 μ≥ 0 e.g. m2(νe)

29. at 90% confidence Frequentist Bayesian

30. Coverage Fraction of intervals containing true value Property of method, not of result Can vary with param Frequentist concept. Built in to Neyman construction Some Bayesians reject idea. Coverage not guaranteed Integer data (Poisson)  discontinuities Ideal coverage plot C μ

31. Coverage : L approach (Not frequentist) P(n,μ) = e-μμn/n! (Joel Heinrich CDF note 6438) -2 lnλ< 1 λ = P(n,μ)/P(n,μbest) UNDERCOVERS

32. Frequentist central intervals, NEVER undercovers(Conservative at both ends)

33. Feldman-Cousins Unified intervalsFrequentist, so NEVER undercovers

34. FELDMAN - COUSINS Wants to avoid empty classical intervals  Uses “L-ratio ordering principle” to resolve ambiguity about “which 90% region?”  [Neyman + Pearson say L-ratio is best for hypothesis testing] No ‘Flip-Flop’ problem

35. Xobs = -2 now gives upper limit

36. Flip-flop Black lines Classical 90% central interval Red dashed: Classical 90% upper limit

38. Poisson confidence intervals. Background = 3 Standard FrequentistFeldman - Cousins

41. Standard Frequentist Pros: Coverage Widely applicable Cons: Hard to understand Small or empty intervals Difficult in many variables (e.g. systematics) Needs ensemble

42. Bayesian Pros: Easy to understand Physical interval Cons: Needs prior Coverage not guaranteed Hard to combine

43. SYSTEMATICS For example we need to know these, probably from other measurements (and/or theory) Uncertainties error in Physics parameter Observed for statistical errors Some are arguably statistical errors Shift Central Value Bayesian Frequentist Mixed

44. Bayesian Without systematics prior With systematics Then integrate over LA and b

45. If = constant and = truncated Gaussian TROUBLE! Upper limit on from Significance from likelihood ratio for and

46. Frequentist Full Method Imagine just 2 parameters and LA and 2 measurements N and M Physics Nuisance Do Neyman construction in 4-D Use observed N and M, to give Confidence Region for LA and 68% LA

47. Then project onto axis This results in OVERCOVERAGE Aim to get better shaped region, by suitable choice of ordering rule Example: Profile likelihood ordering

48. Full frequentist method hard to apply in several dimensions Used in 3 parameters For example: Neutrino oscillations (CHOOZ) Normalisation of data Use approximate frequentist methods that reduce dimensions to just physics parameters e.g. Profile pdf i.e. Contrast Bayes marginalisation Distinguish “profile ordering” See Giovanni Punzi, PHYSTAT05 page 88

49. Talks at FNAL CONFIDENCE LIMITS WORKSHOP (March 2000) by: Gary Feldman Wolfgang Rolke hep-ph/0005187 version 2 Acceptance uncertainty worse than Background uncertainty Limit of C. Lim. asσ 0 Lim Need to check Coverage σ

50. Bayesian versus Frequentism BayesianFrequentist