P Values - part 4 The P value and ‘rules’

P Values - part 4The P value and ‘rules’ Robin Beaumont 10/03/2012 With much help from Professor Geoff Cumming

probability Putting it all together P Value sampling Alternatives Rules statistic

Review Summary so far • A P value is a conditional probability which considers a range of outcomes = ‘area’ in a PDF graph . • The SEM formula allows us to: predict the accuracy of your estimated mean across a infinite number of samples! • Taking another random sample will give us a different P value • How different? - Does not follow a normal distribution • Dance of the p values – Geoff Cumming • Depends upon if the null hypothesis is actually true in reality! Remember we have assumed so far that the null hypothesis is true.

9.037 n =15 t density: s = x 96 Original units: 120 Shaded area=0.0188 0 2.656 0 -2.656 t Rules Given that the sample was obtained from a population with a mean (i.e. parameter value) of 120 a sample with a T(n=15) statistic of -2.656 or 2.656 or one more extreme with occur 1.8% of the time. This is less than one in twenty (i.e. P value < 0.05). Mark the one in twenty value = the critical value for T(n=15) = 2.144 = alpha (α) level =0.05) Rule -> If our result has a P value more extreme than our level of acceptability = our critical value Reject the parameter value we based the p value on. = reject the null hypothesis Therefore we dismiss the possibility that our sample came from a population with a mean of 120

Rules Set a level of acceptability = critical value (CV) Say one in twenty 1/20 = Or 1/100 Or 1/1000 or . . . . When p value is in the critical region Reject the parameter value we based the p value on is considered untenable = reject the null hypothesis What is the mean of the population if we have now ruled out one value? What value do we now give the parameter value ?

Why do we want to reject the null distribution? • Allows decision making • Moves thing forward • If the sample did not come from the ‘null distribution’ indicates there is some effect • Population mean of 120 for substance X indicates they have a propensity to a range of nasty diseases • Given them miracle drug Y which is believed to reduce the level has this have any effect in our sample of 15? • If the probability of obtaining our t value (i.e. + associated p value) is below a certain critical threshold we can say that our sample does not come from the null distribution and there is a effect.

Fisher – only know and only consider the model we have i.e. The parameter we have used in our model – when we reject it we accept that any value but that one can replace it. Hnull = μ=120versus alternative Halt = μ≠120 [μ = population mean] Hnull = T= 0versus alternative Halt = T≠ 0 [T = t statistic] Neyman and Pearson + Gossling Hnull = μ=120versus alternative Halt = μ = XXX [μ = population mean] Hnull = T= 0versus alternative Halt = T = xxx [T = t statistic] Bayesians

View 1 - The infinite variety of alternative distributions - Fisher Fisher – infinitely many alternative ‘red’ distributions Alternative distributions: Become flatter further away from null Become more asymmetrical as further away from null When coincides with null= same shape How do we define the alternative hypothesis? Distance measures: Effect size / Non centrality Parameter Null distribution t(df, ncp=0) Alternative distributions t(df, ncp)

Take the distribution around the sample value • Then work our the difference SD’s: View 2 - The single specified alternative – Neyman + Pearson Population mean of 120 for substance X, SD= 35 units/100 ml. 15 random subjects take miracle drug have a mean = 96 Delta = Δ Non centrality parameter = capital = d x √15 = -.6857 x 3.872 = -2.6557

Red line = null distribution Red areas = critical regions = α alpha regions Blue line = specified alternative distribution Blue shaded area = β beta regions (far right– very small) Gpower shows it all! Population mean of 120 for substance X, SD= 35 units/100 ml. 15 random subjects take miracle drug have a mean = 96

= 96 = 120 α = the reject region Two correct and 2 incorrect decisions Correct decisions incorrect decisions

Power = 1 - Beta Power is good Correct decisions

More Power the better up unto a certain point! Insufficient power – unlikely to get a p value in the critical region Too much power always p value in critical region but possibly trivial effect size

Cumming - 2008 Replication and P intervals Red line = null distribution Red areas = critical regions = α alpha regions Blue line = specified alternative distribution Blue shaded area = β beta regions (far right– very small)

Implications Power and Cohens effect size measure d reflect one another. When power high p value distribution does provides a measure of evidence for the specific alternative hypothesis. Computer simulations - reality different Replication is a vitally important research strategy – meta analysis. Power analysis during study design and Confidence intervals along with effect size measures when reporting results – alternative strategy? Always specify specific alternative hypothesis www.robin-beaumont.co.uk/virtualclassroom/contents.html

Students bloomers • The p value did not indicate much statistic significance • Given that the population comes from one population • The p value is 0.003 thus rejecting the null hypothesis and there is a statistical significance • Correlation = 0.25 (p<0.001) indicating that assuming that the data come from a bivariate normal distribution with a correlation of zero you would obtain a correlation of <0.000. There is 95% chance that the relationship among the variables is not due to chance

P Values - part 4 The P value and ‘rules’