CSE 5331/7331 Fall 2011 P-Value and Statistical Significance

CSE 5331/7331Fall 2011P-Value and Statistical Significance Margaret H. Dunham Department of Computer Science and Engineering Southern Methodist University .

Outline Overview P-Value Statistical Significance Test Statistics Examples

Data Mining - Remember When we analyze data we may only be dealing with a sample of the complete data (which may be infinite) We often want to generalize our findings to the entire data population Calculation of a P-value can help us to determine how likely our results are to apply to the entire population How do we do this?

Normal Distribution Will G. Hopkins, “A New View of Statistics; P Values and Statistical Significance,” 2002, http://sportsci.org/resource/stats/pvalues.html

But what are we comparing? Two different results: • Employment rate for MSCS vs MBA • Scores on a standardized test between male and female or USA and another country. • Classification by two different classifiers Sometimes a correlation coefficient or a test statistic whose distribution is known (such as chi squared) is examined.

Correlation Coefficient * Will G. Hopkins, “A New View of Statistics,” 2002, http://www.sportsci.org/resource/stats/pvalues.html When we look at Statistical Significance we will be comparing two values. The figure below shows the probability coefficient values in a sample of size 20 when the correlation is 0 (that is no correlation).*

Statistics Assumptions Sample data is representative of the entire population Sample data is randomly chosen These assumptions often (usually) do not hold for a given real world sample (training data). • Real world data format is unknown. • May wish to extrapolate to a larger population

Hypotheses Alternative Hypothesis (HA): This is the relationship between the variables that you expect (hope) your experiments will demonstrate. Null Hypothesis (H0). This is just the opposite relationship. There is no relationship between the variables. In significance testing we really determine whether we should reject the null hypothesis.

P-Value The probability that a variable has a value greater than the observed value http://en.wikipedia.org/wiki/P-value http://sportsci.org/resource/stats/pvalues.html

P-Value “The probability that a variate would assume a value greater than or equal to the observed value strictly by chance.”* “the probability of obtaining a test statistic at least as extreme as the one that was actually observed, assuming that the null hypothesis is true.”** Smaller the value the better (You want to reject the Null Hypothesis. * Weisstein, Eric W. "P-Value." From MathWorld--A Wolfram Web Resource. http://mathworld.wolfram.com/P-Value.html , 9/21/11. ** Wikipedia, “P-value”, http://en.wikipedia.org/wiki/P-value, 9/19/11.

Finding P-value May be able to calculate P-value directly or you may have to convert the data (by using a test statistic) into a value that can be used. Example: Find correlation between two variables. The correlation coefficient can be used directly if we know its distribution. However, in some cases a new statistic is used to provide that P-value.

Another Way to Look at It P-value is a measure of how much evidence you have against the Null Hypothesis. Null Hypothesis: Hypothesis of no change Critical Regions – values of statistics for which, if they occur you will reject the Null Hypothesis

Confidence Intervals Range of values that contains the true parameter. Size depends on sample size and variance. Larger sample size, smaller interval If the variance is large, larger interval.

P-Value and Confidence Intervals 95% confidence intervals * Will G. Hopkins, “A New View of Statistics,” 2002, http://www.sportsci.org/resource/stats/pvalues.html Confidence Intervals and P-Values are related. However, 95% confidence interval is not the same thing as P=0.05 It is the same as a P-value of 0.05 “that doesn’t overlap zero.” [3] “If the 95% CI includes no difference between groups, then the P values is > 0.05.” [1] “If the 95% CI does not include no difference between groups then the P value is < 0.05.” [1]

So … So, if we obtain a data value from a population distribution that is normal AND we know that it occurs in one of the 0.2% wings of the distribution, than we should be somewhat convinced that that value is not as likely (although still possible) to have seen as the more commonly occurring data values. How sure are you that the results you’ve found in the experiments are actually true???

Statistical Significance Way to assign a confidence value to a finding. Probability that a finding is true in the general population and not a fluke (not due to chance or random). A significance level is a measure as to how likely the result is due to chance. P-value is a test or measure of statistical significance. What is probability that relationship between two variables exists? What is the probability that this relationship (our experiments seem to indicate exists) is due only to random chance?

Using Significance Test State Alternative Hypothesis State Null Hypothesis. Perform research Identify statistic and its distribution Decide on alpha threshold Calculate statistic Calculate P-Value Compare P-Value to threshold • If lower, probability is small that the result was by chance therefore the finding is significant • If higher, then finding is not significant

Probability of Error Alpha Level (Threshold) Threshold value to make decision Rule of thumb: 0.05 Since you hope to reject the Null Hypothesis, then you hope you’ll find p < 0.05. This means you are beyond (outside in the distribution) the Alpha level and your chance of a Type I error is acceptable.

Significance Levels Rule of thumb: 0.95 – The result has a 95% chance of being true. However, results are being stated in terms of not being true. (i.e. P-value = 1 – Level )So then a 0.05 value is good. So p<0.005 indicates a significant result. The smaller the p value the better. P=0.001 is better than p=0.05. If we know what the distribution of the statistic is, then we can estimate the extreme areas outside (furthest from the mean). This can tell us the significance

Sample P-Value Levels Suppose your threshold is P=0.05 Table from GraphPad, “Interpreting statistical significance,” 1999, http://www.graphpad.com/articles/interpret/principles/stat_sig.htm .

DM &Significance Tests In MOST data mining experiments we want to determine if the results we found in the sample can be generalized to the general population. So the two variables being compared are the one found in the sample and the one that should be found in the population (if we could even get the entire population set).

Two-Tailed vs One-Tailed Will G. Hopkins, “A New View of Statistics; P Values and Statistical Significance,” 2002, http://sportsci.org/resource/stats/pvalues.html Whether the test assumes both sides of the statistics distribution If direction of difference or relationship is important than a one-tailed probability is include – otherwise two-tailed.

Types of Errors Type I: Conclude relationship exists when in fact it doesn’t and evidence shows that it doesn’t. • Null hypothesis should be accepted. • Alpha error Type II: Conclude relationship does not exist when in fact it does and the evidence shows it does. • Null hypothesis should be rejected. • Beta error To be on safe side, try to minimize Type I errors. • Use high Alpha probability • Alpha level is the probability of making an error that you are willing to accept. • Often Alpha=0.05 or even 0.01.

Test Statistics So how do we calculate P-Value? We need to convert the raw data generated by experiments into a single value which we know follows some known frequency distribution. We can then calculate the P-Value based on that These are called Test Statistics

Standard Test Statistics Common Test Statistics http://en.wikipedia.org/wiki/Statistical_hypothesis_testing T – Compares distributions of means of two groups http://www.socialresearchmethods.net/kb/stat_t.php http://mathworld.wolfram.com/Studentst-Distribution.html http://www.stattools.net/tTest_Tab.php Conversion between http://www.graphpad.com/quickcalcs/pvalue1.cfm

Chi Square Test Experiments may actually general nominal or ordinal data. How do you convert these values into numbers that can be evaluated? Often a Chi Square Test is used. Here the collected data is summarized and then converted into a statistic that can be used to measure how the values compare to what would be expected randomly.

Chi Square Distribution Not normal but approaches normal as degrees of freedom (number of indepdent variables) increases. http://stattrek.com/lesson3/chisquare.aspx

Chi Square Test Process Complete contingency table based on two subset types and possible values. Even though actual values are not numeric, when we complete the table we then have counts (frequencies) which are numeric and can be analyzed. The Chi Squared Statistic is calculated by comparing the observed frequency values to the expected frequency values. This statistic is what is examined for significance. It follows a chi squared distribution. Note: http://epm.sagepub.com/content/52/1/57.short

Chi Squared Example Suppose we compare the hiring rate for recent MS CS graduates to MBA graduates. We find that 150 out of 200 MS students have a job and 100 out of 200 MBA students have a job. Hypothesis: MSCS students are hired at a higher rate than MBA students. Null Hypothesis: MBA and MSCS students are hired at the same rate. Suppose we use p=0.05 as significance level.

Chi Squared Example (cont’d) 250 students have jobs so 250/400=0.625 150 do not have jobs, so 150/400=0.375 We would expect these percentages to hold for each population if the Null Hypothesis is true. 150/200=0.75 MSCS have jobs 100/200=0.5 MBA have jobs We would expect 0.625 * 200 = 125 in each group to have jobs

Chi Squared Example (cont’d) http://course1.winona.edu/sberg/Equation/chi-squa.gif Calculate Chi Square http://www.graphpad.com/quickcalcs/contingency1.cfm Chi Square value is 26.67 How do we convert this to a P-Value? http://www.statsoft.com/textbook/distribution-tables/#chi P-Value is <0.05. Reject Null Hypothesis Yes it is significant (Actually p<0.0001 so it is extremely significant)

Example 1 – Iris Data ===Summary ===Correctly Classified Instances 144 96 %Incorrectly Classified Instances 6 4 %Kappa statistic 0.94 Mean absolute error 0.035 Root mean squared error 0.1586Relative absolute error 7.8705 %Root relative squared error 33.6353 %Total Number of Instances 150 J48 Default Cross Validation We have the Kappa statistic but can’t use it to calculate P-value without variance. http://twiki.org/p/pub/Main/SigurdurRunarSaemundsson/Interrater_agreement.Kappa_statistic.pdf

Example 1 – Iris Data === Detailed Accuracy By Class === TP Rate FP Rate Precision Recall F-Measure ROC Area Class 0.98 0 1 0.98 0.99 0.99 Setosa 0.94 0.03 0.94 0.94 0.94 0.952 Versicolor 0.96 0.03 0.941 0.96 0.95 0.961 VerginicaWeighted Avg. 0.96 0.02 0.96 0.96 0.96 0.968=== Confusion Matrix === a b c <-- classified as 49 1 0 | a = Setosa 0 47 3 | b = Versicolor 0 2 48 | c = Verginica View confusion matrix as contingency Table and Calculate Chi Squared

Example 1 (cont’d) http://faculty.vassar.edu/lowry/newcs.html Chi Square = 266 P < 0.0001 Extremely statistically significant

Example 2 NektariosLeontiadis, Tyler Moore and Nicolas Christin. "Measuring and Analyzing Search-Redirection Attacks in the Illict Online Prescription Drug Trade". 20th USENIX Security Symposium. August 10-12, 2011: San Francisco, Ca. http://cs.wellesley.edu/~tmoore/usenix11.pdf

References American College of Physicians- American Society of Internal Medicine, “Primer on 95% Confidence Intervals,” Effective Clinical Practice, September/October 2001, Vol 4, No 6, pp 229-231. Will G. Hopkins, “A New View of Statistics,” 2002, http://www.sportsci.org/resource/stats/pvalues.html M. A. Saint-Germain, “PPA 696 Research Methods,” 1/1/01, http://www.csulb.edu/~msaintg/ppa696/696menu.htm

CSE 5331/7331 Fall 2011 P-Value and Statistical Significance