1 / 45

Scientific Methods 1

Scientific Methods 1. ‘Scientific evaluation, experimental design & statistical methods’ COMP80131 Lecture 4: Statistical Methods-Probability. Barry & Goran. www.cs.man.ac.uk/~barry/mydocs/myCOMP80131. Probability. There are two useful definitions of probability: Bayesian probability:

shellyl
Download Presentation

Scientific Methods 1

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Scientific Methods 1 ‘Scientific evaluation, experimental design & statistical methods’ COMP80131 Lecture 4: Statistical Methods-Probability Barry & Goran www.cs.man.ac.uk/~barry/mydocs/myCOMP80131 COMP80131-SEEDSM12-4

  2. Probability There are two useful definitions of probability: • Bayesian probability: A person’s belief in the truth of a statement S, quantified on a scale from 0 (definitely not true) to 1 (definitely true). 2. Experimental (or frequentist) probability: Limit of M / N as N tends to infinity, where M = number of times that a statement S is found to be true if it is tested N times . COMP80131-SEEDSM12-4

  3. Different language • By either defn, probability P(S) is a number in range 0 to 1. • Multiply by 100 to express as a percentage. • Or express as odds: e.g. ‘4 to 1 against’ means 1/5 = 0.2 = 20%. • What do odds of ‘4 to 1 on’ mean? • What does ‘50-50’ mean ? COMP80131-SEEDSM12-4

  4. Calculating probability • The 2 defns of probability are related but subtly different. • By examining a coin, we could give ourselves good reason for believing that tossing it just once will give an even chance of getting heads, i.e. that the Bayesian defn of P(S) = 0.5 where S = ‘get heads’. • If coin is then tossed N = 100 times we would expect about M = 50 occurrences of heads meaning that M/N  0.5. • Increasing N to 1000 & then to 1000000 would be expected to produce closer & closer approximations to P(S) = 0.5. • If this does not happen, our ‘a-priori’ belief may be wrong. • The coin may be ‘weighted’. COMP80131-SEEDSM12-4

  5. Random process • Tossing a coin is a random process. • It generates a ‘random variable’ Heads or Tails. • Random because the outcome cannot be predicted exactly. • If 1= heads & 0 = tails we have a random binary number. • Throwing a dice generates a random integer in range 1-6. • Roulette wheel generates random integers in range 0-36. • Exams produces random numbers in range 0-100 • These are random processes producing discrete variables. • Some random processes produce continuous variables. e.g. measuring people’s heights. COMP80131-SEEDSM12-4

  6. Simulating random process • MATLAB has functions that generate pseudo-random numbers. • ‘rand’ produces a pseudo-random number ‘uniformly distributed’ in the range 0 to 1. • May be considered ‘continuous’ since floating pt is very accurate. • Calling ‘rand’ repeatedly produces numbers evenly distributed across the range 0 to 1. • ‘Pseudo-random’ because if we know the algorithm used, we can predict the numbers. • So we pretend we do not know the algorithm. • ‘rand’ may be considered to simulate some random process that generates truly random numbers, uniformly distributed. COMP80131-SEEDSM12-4

  7. Simulating coin tossing in MATLAB for n=1:20 R = rand; if R > 0.5, Heads(n)=1 else Heads(n) = 0; end; end; % of n loop Heads 10110001110101011101 - 12 heads & 8 tails • I changed 20 to 10,000 & got 5066 heads: P(Heads)  0.5066 • I ran it again & got 4918 heads : P(Heads)  0.4918 COMP80131-SEEDSM12-4

  8. Using an unfair coin for n=1:20 R = rand; if R > 0.4, Heads(n)=1 else Heads(n) = 0; end; end; % of n loop Heads 00101001110101010101 - 10 heads & 10 tails • I changed 20 to 10,000 & got 6012 heads: P(Heads)  0.6012 • When I ran it again, I got 5979 heads : P(Heads)  0.5979 COMP80131-SEEDSM12-4

  9. Estimating probability experimentally • Cannot measure probability with 100% accuracy. • All measurements are estimates • They may be slightly or totally wrong. • According to experimental defn, we must perform an expt an infinite number of times to measure a probability. • This is clearly impossible. • In practice, we have to perform the experiment a finite number of times • (Cannot spend all our lives tossing coins) • Accept resulting measurement as estimate of true probability. COMP80131-SEEDSM12-4

  10. Bayesian Definition • According to Bayesian defn of probability, a person’s belief in the truth of a statement may be affected by one or more assumption (hypotheses). • “I assume it is a fair coin” • Different people may have different beliefs. • Can only estimate probability using information we have available, • Can modify this estimate later if we get new information. COMP80131-SEEDSM12-4

  11. Conditional probability • P(S  S1) = probability of ‘statement S’ being true given that we know that another statement, S1, is definitely true. • If S  ‘get heads’ we may at first believe that P(S) = 0.5. • But what if someone tells us that the statement S1: ‘coin is weighted with heavier metal on one side’, is true? • Change our measurement of probability to P(S  S1). • P(S) is then referred to as the ‘prior’ probability • P(S  S1) is the ‘conditional’ or ‘posterior’ probability. COMP80131-SEEDSM12-4

  12. Bayes’s Theorem • Expresses probability of some fact ‘A’ being true when we know that some other fact ‘B’ is true: • E.g. let A = ‘coin is fair’ & B = ‘get 12 heads out of 20’ • P(A) is ‘prior’ as it does not take into account any information about B. • Similarly P(B) is ‘prior’. • P(A|B) and P(B|A) are ‘conditional’ or ‘posterior’ prob. • Can write P(B) = P(B | A)P(A) + P(B | not A)P(not A) COMP80131-SEEDSM12-4

  13. What is prob of getting 12 heads out of 20? clear all; % WITH FAIR COIN H=zeros(21,1); for rep=1:1000 for n=1:20 R = rand; % Unif random number between 0 & 1 if R > 0.5, Heads(n)=0; else Heads(n)=1; end; end; % of n loop Count = sum(Heads); H(1+Count) = H(1+Count)+1; end; % of rep loop figure(1); stem(0:20,H); COMP80131-SEEDSM12-4

  14. Histogram for 1000 trials FAIR COIN 200 180 160 140 Frequency out of 1000 trials 120 100 80 60 40 20 0 0 2 4 6 8 10 12 14 16 18 20 Number of Heads obtainable with 20 coin-tosses COMP80131-SEEDSM12-4

  15. Estimate of probability distribution FAIR COIN 0.2 0.18 0.16 0.14 Estimate of prob distribution based on 1000 trials 0.12 0.1 0.08 0.06 0.04 0.02 0 0 2 4 6 8 10 12 14 16 18 20 Number of Heads obtainable with 20 coin-tosses COMP80131-SEEDSM12-4

  16. Probability 0.2 0.1 Outcome 2 4 1 3 Probability distribution (discrete) • When there is a finite number of possible outcomes, • For each possible outcome, gives its probability of occurring. COMP80131-SEEDSM12-4

  17. pdf Possible outcome x P Q Probability density function(pdf) • For an infinite number of possible outcomes, • Real numbers, continuous over range -∞ to  • Prob outcome lies betw P & Q: • Area under the curve. • NB prob (P) = prob(Q) = 0 COMP80131-SEEDSM12-4

  18. pdf(x) 1 x a b 1 pdf(x) m a b x m- m+ Continuous random processes • Characterised by probability density functions (pdf) Uniform pdf: Prob of the random variable x lying between a and b is: Gaussian (Normal) pdf with mean m & std dev . 95.5% for m  299.7% for m  3 68% COMP80131-SEEDSM12-4

  19. Continuous distributions • More about these later. • Back to discrete now. COMP80131-SEEDSM12-4

  20. Probability estimate (fair coin) • Estimated discrete probabilities: for 0:9 heads 0 0 0 0 0.008 0.011 0.024 0.087 0.119 0.160 for 10:20 heads 0.194 0.157 0.115 0.076 0.03 0.012 0.003 0.003 0.001 0 0 • Estimated probability of getting 12 heads out of 20 with a fair coin is 0.115. COMP80131-SEEDSM12-4

  21. What is prob of getting 12 heads out of 20? clear all; %WITH 60-40 WEIGHTED COIN H=zeros(21,1); for rep=1:1000 for n=1:20 R = rand; % Unif random number betw 0 & 1 if R > 0.4, Heads(n)=1; else Heads(n)=0; end; end; % of n loop Count = sum(Heads); H(1+Count) = H(1+Count)+1; end; % of rep loop figure(1); stem(0:20,H); COMP80131-SEEDSM12-4

  22. 200 180 160 140 120 Frequency out of 1000 trials 100 80 60 40 20 0 0 2 4 6 8 10 12 14 16 18 20 Number of Heads obtainable with 20 coin-tosses HISTOGRAM for ‘60-40’ weighted coin COMP80131-SEEDSM12-4

  23. 0.2 0.18 0.16 0.14 Estimate of prob distribution based on 1000 trials 0.12 0.1 0.08 0.06 0.04 0.02 0 0 2 4 6 8 10 12 14 16 18 20 Number of Heads obtainable with 20 coin-tosses Prob distribution estimate for ‘60-40’ weighted coin COMP80131-SEEDSM12-4

  24. Cumulative prob distrib (discrete CDF) • Prob of ordered output being less than or equal to a given outcome x. CDF 1 0.1 2 Outcome x 4 1 3 COMP80131-SEEDSM12-4

  25. Estimate Cumulative Prob Distrib • Prob of getting between 0 and n Heads • Easily derived from a Histogram or Prob Distribution. CDF(1)= H(1)/1000; for n=2:21, CDF(n)=CDF(n-1) + H(n)/1000; end; figure(3); stem(0:20,CDF); COMP80131-SEEDSM12-4

  26. 1 0.9 0.8 0.7 0.6 Estimate of cumulative prob dist based on 1000 trials 0.5 0.4 0.3 0.2 0.1 0 0 2 4 6 8 10 12 14 16 18 20 Number of Heads obtainable with 20 coin-tosses Estimate of Cumulative Prob Dist FAIR COIN Usually an S shaped function COMP80131-SEEDSM12-4

  27. 4 coin-tosses: how many possible outcomes? How many with 0 heads? 1 How many with 1 heads? 4 = 4C1 How many with 2 heads? 6 = 4C2 = 43/ (2!) How many with 3 heads? 4 = 4C3 How many with 4 heads? 1 Combinations: nCr = no of ways of choosing r from n = n(n-1) …(n-r+1) / (r!) 0000 0001 0010 0011 0100 0101 0110 0111 1111 1001 1010 1011 1100 1101 1110 1111 COMP80131-SEEDSM12-4

  28. Binomial Prob Distribution • Distributions have up to now been estimated. • For random processes with just 2 outputs, we can derive a true distribution: • If p=prob(Heads), prob of getting Heads exactly r times in n independent coin-tosses is: nCr pr (1-p)(n-r) • For a fair coin. p=0.5,  this becomes nCr /2n • For a fair dice, prob of throwing 3 sixes in five throws is: [54/(3 2 1)] (1/6)3  (5/6)2 COMP80131-SEEDSM12-4

  29. Implementing formula (fair coin) • p = 0.5; % for fair coin tossing • n=20; • for r=0:n • nCr = prod(n:-1:(n-r+1))/prod(1:r); • P(1+r) = nCr * (p^r) * (1-p)^(n-r); • end; • figure(4); stem(0:20,P); • axis([0 20 0 0.2]); grid on; COMP80131-SEEDSM12-4

  30. 0.2 0.18 0.16 0.14 0.12 True probability of getting that no of heads 0.1 0.08 0.06 0.04 0.02 0 0 2 4 6 8 10 12 14 16 18 20 No of heads obtainable with n coin-tosses True prob distribution (n=20) Fair coin COMP80131-SEEDSM12-4

  31. True probability from formula • For 0-9 heads: 0 0 0.0002 0.0011 0.0046 0.015 0.037 0.074 0.12 0.16 • For 10-20 heads: 0.176 0.16 0.12 0.074 0.037 0.015 0.0046 0.0011 0.0002 0 0 • True prob of getting 12 heads with a fair coin is 0.12. • True prob of getting ≥12 heads with fair coin is 0.252. • Changing p to 0.4, we find that true probability of getting 12 heads out or 20 with a ‘60-40’ weighted coin is: 0.18 • True prob of getting ≥12 heads with weightd coin is 0.61. COMP80131-SEEDSM12-4

  32. Back to Bayes’s Theorem • There are 2 coins a fair one & a ‘60-40’ weighted one. • Choose a coin at random & toss it 20 times. • What is the probability of having a weighted coin when I get 12 heads out of 20? • A  ‘coin is weighted 60-40’ & B  ‘get 12 heads from 20’ • We know that P(B Fair coin) is 0.12 & P(B A) is 0.18. • P(B) = P(B | not A)  P(not A) + P(B | A) P(A) = 0.12  0.5 + 0.18 0.5 = 0.15 • P(A B) = P(B A)  P(A) / P(B) = 0.18  0.5 /0.15 = 0.6 • If B  ‘get ≥12 heads from 20’, P(A/B)=0.60.5/(0.6 0.5+0.252 0.5) = .3/.42 = 0.71 COMP80131-SEEDSM12-4

  33. Further illustration of Bayes Theorem • At a college there are: 10 students from France 5 girls & 5 boys 15 from UK 5 girls & 10 boys 20 from Canada 5 girls & 15 boys COMP80131-SEEDSM12-4

  34. Calculation • If we choose a student at random, the a-priori probability that this student is French is P(French) = 10/45 = 2/9  0.22 • If we notice that this student is a boy, how does this change the probability that the student is French? • Use Bayes’ Theorem as follows: • = 0.5  (10/45) / (30/45) = 1/6  0.167 • The fact that we notice that the chosen student is a boy gives us additional information that changes the probability that the student chosen at random will be French. COMP80131-SEEDSM12-4

  35. Check the calculation • We can check the previous result by common sense, • Notice that out of 30 boys, 5 are from France. • Therefore, P(FB) = 5/30 = 1/6. COMP80131-SEEDSM12-4

  36. Usefulness of Bayes Theorem • It allows us to take additional information into account when calculating probabilities. • Without the additional information, we have a ‘prior’ probability • With it we have a ‘conditional’ or ‘posterior’ probability. COMP80131-SEEDSM12-4

  37. Bayes Theorem in medicine • A patent goes to a doctor with a bad cough & a fever. The doctor needs to decide whether he has ‘swine flu’. • Let statement S = ‘has bad cough and fever’ and statement F = ‘has swine flu’. • The doctor consults his medical books and finds that about 40% of patients with swine-flu have these same symptoms. • Assuming that, currently, about 1% of the population is suffering from swine-flu and that currently about 5% have bad cough and fever (due to many possible causes including swine-flu), we can apply Bayes theorem to estimate the probability of this particular patient having swine-flu. COMP80131-SEEDSM12-4

  38. Another problem to solve • A doctor in another country knows form his text-books that for 40% of patients with swine-flu, • The statement S, ‘has bad cough and fever’ is true. • He sees many patients and comes to believe that the probability that a patient with ‘bad cough and fever’ actually has swine-flu is about 0.1 or 10%. • If there were reason to believe that, currently, about 1% of the population have a bad cough and fever, what percentage of the population is likely to be suffering from swine-flu? COMP80131-SEEDSM12-4

  39. Concept of a ‘null-hypothesis’ • A null-hypothesis is an assumption that is made and then tested by a set of experiments designed to reveal that it is likely to be false, if it is false. • Testing is done by considering how probable the results are, assuming the null hypothesis is true. • If the results appear very improbable the researcher may conclude that the null-hypothesis is likely to be false. • This is usually the outcome the researcher hopes for when he or she is trying to prove that a new technique is likely to have some value. COMP80131-SEEDSM12-4

  40. An example • Assume we wish to find out if a proposed technique designed to benefit users of a system is likely to have any value. • Divide the users into two groups and offer the proposed technique to one group and something different to the other group. • The null-hypothesis would be that the proposed technique offers no measurable advantage over the other techniques. COMP80131-SEEDSM12-4

  41. The testing • Carried out by looking for differences between the sets of results obtained for each of the two groups. • Careful experimental design to eliminate differences not caused by the techniques being compared. • Must take a large number of users in each group & randomize the way the users are assigned to groups. • Once other differences have been eliminated as far as possible, remaining difference will hopefully be indicative of effectiveness of techniques being investigated. • Vital question is whether they are likely to be due to the advantages of the new technique, or the inevitable random variations that arise from the other factors. • Are the differences statistically significant? • Can employ statistical significance test to find out. COMP80131-SEEDSM12-4

  42. Failure of the experiment • If results are not found to look improbable under the null-hypothesis, i.e. if differences between the two groups are not statistically significant, then no conclusion can be made. • Null-hypothesis could be true, or it could still be false. • It would be a mistake to conclude that the ‘null-hypothesis’ has been proved likely to be true in this circumstance. • It is quite possible that the results of the experiment give insufficient evidence to make any conclusions at all. COMP80131-SEEDSM12-4

  43. P-Value • Probability of obtaining a test result at least as extreme as the one that was actually observed, assuming that the null hypothesis is true. • Reject null-hypothesis if p-value is less than some value α (significance level) which is often 0.05 or 0.01. • When null-hypothesis is rejected, result is statisticallysignificant. COMP80131-SEEDSM12-4

  44. Checking whether a coin is fair • Suppose we obtain heads 14 times out of 20 flips. • The p-value for this test result would be the probability of a fair coin landing on heads at least 14 times out of 20 flips. • This is: (20C14 + 20C15+20C16+20C17+20C18+20C19+20C20) / 220 = 0.058 • This is probability that a fair coin would give a result as extreme or more extreme than 14 heads out of 20. COMP80131-SEEDSM12-4

  45. Significance test • Reject null-hypothesis if p-value  α . • If α= 0.05, rejection of null-hypothesis is at the 5% (significance) level. • Probability of wrongly rejecting null-hypothesis (Type 1 error) will be equal to α. • This is considered ‘sufficiently low’. • In this ‘coin testing’ case, p-value > 0.05, • Therefore observation is consistent with null-hypothesis and we cannot reject it. • Cannot conclude that coin is likely to be unfair. • But we have NOT proved that coin is likely to be fair. • 14 heads out of 20 flips can be ascribed to chance alone. • It falls within the range of what could happen 95% of the time with a fair coin. COMP80131-SEEDSM12-4

More Related