1 / 79

Biostat 200 Lecture 3

Biostat 200 Lecture 3. Announcements. Assignments Put your name on the assignment (inside the file) Submit one file only – no attachments Bring your textbook to lab this Thursday 10/7 Bring your laptop to lecture next Tuesday 10/12 Discussion of last 2 labs Review of some probability facts

lev
Download Presentation

Biostat 200 Lecture 3

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Biostat 200 Lecture 3

  2. Announcements • Assignments • Put your name on the assignment (inside the file) • Submit one file only – no attachments • Bring your textbook to lab this Thursday 10/7 • Bring your laptop to lecture next Tuesday 10/12 • Discussion of last 2 labs • Review of some probability facts • Check in on what you should have learned so far • Probability distributions

  3. Today’s topics • Discussion of last 2 labs • Review of some probability facts • Check in on what you should have learned so far • Probability distributions

  4. Thoughts from lab • Data cleaning is always necessary with a new data set • The first step is to use tables and summary statistics and graphs to identify outliers and anomalies • Outliers are defined as extreme values • We do NOT automatically remove outliers !!!

  5. Outliers – what do we do? • First consider if the value is physically possible • Example of person 3’4” tall . Yes, that is physically possible but fairly unusual. • Look at the other variables for clues. We found age=3. • For this one, we remove the entire observation from the analysis data set because of ineligibility • We document this, and save a copy of the original data set

  6. Outliers – what do we do? • If age had been =20, we might have asked the interviewer about this value. • Another example – there were a few other strange heights: 5’12”, 5’20”, 5’41” ... • Probably typos? Check original source document. • You can program your data entry programs not to accept out of range values.

  7. Outliers – what do we do? • We also have 2 observations with weight=25, 30 pounds... • If we can’t explain but we are pretty sure that these values are not reasonable, we might exclude these values (but not the whole observation unless we suspect poor data throughout!)

  8. Outliers – what do we do? • What about these high values?

  9. Outliers – what do we do? • What about outliers that seem reasonable? • May have large influence on some analyses • Be aware of them, do not exclude them. • Think about more robust analyses. E.g. which measures of central tendency might you use?

  10. Stata coding strategies • Keep a .do file for all your recodes • At the beginning of the file read in the original raw data • At the end of the file save the data to another filename • Use comments to remind yourself why you are making these recodes (e.g., dropping the 3 year old, assuming 5’12” really is 5’1”) • Make separate .do files for your analyses • Make a .do file to create value labels that you might use across data sets • label define 0 “Male” 1 “Female” • label define 0 “Negative” 1 “Positive” 2 “Indeterminate” • Use the command include *.do to include the value label .do file in your recode .do file

  11. use "H:\Work files\Teaching\Biostat 200­­_2010\biostat200_colddata_2010.dta", clear • include "H:\Work files\Teaching\Biostat 200­­_2010\label defines.do" • label values educeducl • label values sex sexl • tab1 educ sex • save "H:\Work files\Teaching\Biostat 200­­_2010\biostat200_colddata_2010_v3.dta"

  12. From last lecture: Independence vs. mutual exclusivity • Mutual exclusivity: P(B ∩ A) = 0 • A and B cannot occur together • Independence A and B are independent: P(B | A)=P(B | Ā) = P(B) P(A | B) = P(A) P(A ∩ B) = P(A)P(B) • A and B can still co-occur (they actually cannot be mutually exclusive), but A has no bearing on B • Example: The chance of malaria is not affected by wearing a blue shirt, but the events are not mutually exclusive (if you wear a blue shirt you can still get malaria).

  13. What you should have learned from the past 2 weeks • Types of variables • The ability to perform in Stata and understand: • Basic manipulation of data, opening and saving data sets and .do files, basic data cleaning • Basic summaries relevant to different types of variables • Basic graphical analyses of different types of variables • Basic probability concepts, especially conditional probability, mutual exclusivity, and independence

  14. Probability distributions • Variables whose outcome can occur by chance, i.e. are not fixed, are called random variables • Probability distributions describe the possible values of the random variable

  15. For discrete variables the probability distribution describes the probability of each possible value • For example , consider the experiment where you flip a coin 2 times and count the number of heads. • The possible outcomes of the experiment are: HH, TH, HT, TT. • You want to focus on the number of heads, which could be 0,1, or 2. The probability of each outcome is:

  16. The table looks similar to a frequency table of the data, but it is actually the theoretical distribution • If you perform an infinite number of experiments, your data will look like this table

  17. The graphical representation of this probability distribution is:

  18. Note that the probabilities add to 1. This is true of all probability distributions. • This is a theoretical probability distribution based on our understanding of coin tossing • The probability of a head on each toss is .5 • The probability of heads on the first toss is independent of the second toss • It’s actually the binomial distribution • We can write down a formula for P(X=x)

  19. We can use this theoretical distribution to make predictions about future experiments • E.g. The probability that there will be at least 1 head in a trial of 2 coin tosses P(X≥1) = P(X=1) + P(X=2) (by what probability rule?) = .5 +.25 = .75

  20. If you performed the experiment once, you’d get 0,1, or 2 heads • Performing the experiment 10 times: 2, 1, 1, 1, 1, 0, 0, 0, 1, 1 • What if we did the experiment 100 times? 1000 times? What would the frequency distribution for the outcomes look like?

  21. Empirical Probability distributions • Empirical probability distributions are based on real data • They are usually based on a large sample or complete enumeration of a population • The probabilities are calculated from the relative frequencies of the data

  22. Probability distributions • For discrete variables the probability distribution describes the probability of each possible value • For continuous variables, the distribution describes the probability of a range of values

  23. Bernoulli distribution • If you have a variable that can take on one of two values with a constant probability p, then it is a Bernoulli random variable • If the proportion of people in the population with a disease (the prevalence) is 15%, then when you randomly select one person, the probability that he/she has the disease is P(Y=1)=p= 0.15 And the probability that a randomly selected person does not have the disease is P(Y=0)=1-p =0.85 • p is the parameter that characterizes the distribution • The Bernoulli distribution is a discrete distribution

  24. Binomial distribution • Example: The proportion of people in the population with the disease (the prevalence) is 15%, then P(Y=1)=0.15 and P(Y=0)=0.85. • If we take a random sample of 5 people from this population, there will be 0,1,2,3,4, or 5 people with the disease. • If the probability of disease in each person is independent, then we can write down the probability of each of these outcomes.

  25. For example, the probability that ALL of them will have the disease is: =P(X1=1)* P(X2=1)* P(X3=1)* P(X4=1)* P(X5=1) = 0.15 x 0.15 x 0.15 x 0.15 x 0.15 = 0.00008 by the multiplication rule for independent outcomes P(A ∩ B)=P(A)P(B)

  26. For example, the probability that NONE of them will have the disease is: =P(X1=0)* P(X2=0)* P(X3=0)* P(X4=0)* P(X5=0) =0.85 x 0.85 x 0.85 x 0.85 x 0.85 = 0.444

  27. The probability that exactly one person P(X=1) has the disease = P(X1=1)* P(the other 4=0) + P(X2=1)* P(the other 4=0) + P(X3=1)* P(the other 4=0) + P(X4=1)* P(the other 4=0) + P(X5=1)* P(the other 4=0) = 0.15 x 0.85 x 0.85 x 0.85 x 0.85 + 0.85 x 0.15 x 0.85 x 0.85 x 0.85 +0.85 x 0.85 x 0.15 x 0.85 x 0.85 + 0.85 x 0.85 x 0.85 x 0.15 x 0.85 + 0.85 x 0.85 x 0.85 x 0.85 x 0.15 = 0.392

  28. The probability that exactly two people P(X=2) of 5 have the disease = 0.15 x 0.15 x 0.85 x 0.85 x 0.85 + 0.15 x 0.85 x 0.15 x 0.85 x 0.85 + 0.15 x 0.85 x 0.85 x 0.15 x 0.85 + 0.15 x 0.85 x 0.85 x 0.85 x 0.15 +0.85 x 0.15 x 0.15 x 0.85 x 0.85 +0.85 x 0.15 x 0.85 x 0.15 x 0.85 +0.85 x 0.15 x 0.85 x 0.85 x 0.15 +0.85 x 0.85 x 0.15 x 0.15 x 0.85 +0.85 x 0.85 x 0.15 x 0.85 x 0.15 +0.85 x 0.85 x 0.85 x 0.15 x 0.15 =10 * .152* .853 = 0.138

  29. The probability that no people P(X=0) of 5 have the disease = .444 The probability that exactly one person P(X=1) of 5 has the disease = .392 The probability that exactly two people P(X=2) of 5 have the disease = .138 The probability that exactly three people P(X=3) of 5 have the disease = .024 The probability that exactly four people P(X=4) of 5 have the disease = .002 The probability that exactly five people P(X=5) of 5 have the disease = .00008

  30. What do these probabilities sum to?

  31. The probability that exactly one person P(X=1) has the disease P(X=1, n=5, p=0.15) = = 0.15 x 0.85 x 0.85 x 0.85 x 0.85 + 0.85 x 0.15 x 0.85 x 0.85 x 0.85 +0.85 x 0.85 x 0.15 x 0.85 x 0.85 + 0.85 x 0.85 x 0.85 x 0.15 x 0.85 + 0.85 x 0.85 x 0.85 x 0.85 x 0.15 = 0.392 = 5 * .151 *.854 = 5 * p1 * (1-p)4 5 is the number of different ways you could get one success in the 5 “trials”

  32. Binomial distribution This generalizes to: Which is the formula for the binomial distribution • p is probability of success • n is the number of “trials” (e.g., coin flips, persons assessed for disease status, etc.) • n and p are the parameters of the binomial distribution, i.e. the values that summarize the distribution • x is the number of “successes” (e.g. heads, numbers with the disease, etc.) • Note that Stata and table A.1 use the symbol k for x

  33. Binomial distribution • Assumptions: • There are a fixed number of trials n, each of which results in one of two mutually exclusive outcomes • The outcomes of the n trials are independent • The probability of success p is constant for each trial

  34. is called “n choose x” and is the number of different ways to get x successes in n trials There are 5 ways that there could be 1 success in 5 trials There are 10 ways there could be 2 successes in 5 trials

  35. There formula for n choose x is 5 choose 1 = 5! / (1! * 4!) = (5*4*3*2*1) / (1*4*3*2*1) = 5 5 choose 2 = 5! / (2! * 3! ) = (5*4*3*2*1) / (2*1*3*2*1) = 5*4/2 = 10 5 choose 3 = 5! / (3! * 2!) = 10 In Stata: display comb(n,k) . display comb(5,3) 10

  36. Ways to find out binomial probabilities without using the previous equations • Table A.1 in the textbook • Stata

  37. Table A.1 • What is the probability of exactly 2 cases of disease in a sample of n=5 where p=0.15? • Table A.1 gives you P(X=k) • Look up p=.15, n=5, k=2, answer=.1382

  38. Stata • display binomialtail(n,k,p) returns P(X≥k) • To find P(X=k) you need display binomialtail(n,k,p) – binomialtail(n,k+1,p) = P(X ≥k) – P(X ≥k+1) = P(X=k) • display binomialtail(5,2,.15) -binomialtail(5,3,.15) .13817813

  39. What is the probability of 1 or more cases of disease in a sample of n=5 where p=0.15? • Remember Table A.1 gives you P(X=k). • We want P(X≥k) • One way would be to look up all the probabilities: P(X=1)+P(X=2)+ ... +P(X=5) • But remember P(X≥k) = 1-P(X<k) = 1-P(X=0) • Looking this up we get 1- 0.4437 = 0.5563

  40. What is the probability of 1 or more cases of disease in a sample of n=5 where p=0.15? • In Stata, binomialtail(n,k,p) gives us P(X≥k) so we can use it without manipulation • display binomialtail(5,1,.15) .55629469

  41. The binomial distribution can be used to calculate the probability of observing at least X successes, or cases of disease, etc, in a population of size n in which the true probability of disease is p. • Example. The Cambodia prevalence of TB infection is 495 per 100,000 (0.00495), yet there have been 7 cases in a school of 1000 children (0.007). You wonder how this compares to the national prevalence. • Prob would see 7 or more cases in 1000 students if p=.00495?

  42. Prob would see 7 or more cases in a school of 1000 if p=.00495? display binomialtail(1000,7,.00495) .23016477 What if there had been 20 cases? Prob would see 20 or more cases in a school of 1000 if p=0.00495? binomialtail(1000,20,.00495) 2.654e-07 What might you conclude?

  43. Binomial distribution • The mean of a binomially distributed random variable X is np • This means that over an large number of samples of size n with probability p of success, the average number of successes x over the samples will be approximately np

  44. Binomial distribution • The variance of a binomially distributed random variable X is np(1-p) • This means that over a large number of samples of size n, the variance of the x’s will be approximately np*(1-p) • Shorthand way to say this: • The mean of the binomial distribution is np • The variance of the binomial distribution is np*(1-p)

  45. So for our example with n=5 and p=.15, the mean is: • The variance is: • The standard deviation is:

  46. Binomial distribution • Binomial mean = np • Binomial variance= np(1-p) • Variance is largest when p=0.5, smaller when p closer to 0 or 1 • The distribution is symmetric when p=0.5 • The distribution is a mirror image for 1-p (i.e. the distribution for p=0.05 is the mirror image of the one for p=0.95)

  47. P(X=2) ? P(X≥2) ?

More Related