Error and Sample Sizes

1. Error and Sample Sizes PHC 6716 June 1, 2011 Chris McCarty

2. Types of error • Non-sampling error – Error associated with collecting and analyzing the data • Sampling error – Error associated with failing to interview the entire population

3. Non-Sampling Error • Coverage error • Wrong population definition • Flawed sampling frame • Interviewer or management error in following sampling frame • Response error • Badly worded question results in invalid or incorrect response • Interviewer bias changes response • Non-response error • Respondent refuses to take survey or is away • Respondent refuses to answer certain questions • Processing errors • Error in data entry or recording of responses • Analysis errors • Inappropriate analytical techniques, weighting or imputation are applied

4. Sampling Error • Sampling error is known after the data are collected by calculating the Margin of Error and confidence intervals • Surveys don’t have a Margin of Error, questions do • Power analyses use estimates of the parameters involved in calculating the margin of error • It is common to see sample sizes of 400 and 1000 for surveys (these are associated with 5% and 3% margins of error) • In most cases the size of the population being sampled from is irrelevant • The margin of error should be calculated using the size of the subgroups sampled

5. Margin of Error Formula • H = Half interval expressed in units of standard deviation • z = z score associated with level of confidence (typically 95%) • s = standard deviation • n = sample size

6. The z score • The z value is the z score associated with a level of confidence • Typically (almost exclusively) surveys use 95% • This means that if the survey were replicated 100 times, 95 times out of 100 the estimate would be within the margin of error • The z score associated with 95% is 1.96

7. The standard deviation (s) • For a continuous variable the standard deviation is typically not known • Previous research may suggest some reasonable range for the margin of error • After you have collected the data the standard deviation is known

8. Example: Age of Floridians • Sample of 406 Floridians • Age range 18 to 92 • Mean age of sample = 52.3 • Standard deviation = 17.6 • 95 times out of 100 sample estimate would be between 50.58 and 54.01 (Frequentist interpretation)

9. Margin of Error for a Proportion • p = proportion

10. Example: Floridians employed • Sample of 415 Floridians • 55.29 percent employed • 44.47 percent not employed • 95 times out of 100 the estimate of the percent employed would be between 50.59 and 59.99

11. Margin of Error with Finite Population Adjustment

12. Example: Floridians employed with finite population adjustment • With the finite population adjustment the margin of error is .01 percent lower

13. No real value to adjustment until you reach 10 percent of population • H adjusted falls to zero as you approach a census • H unadjusted never does

14. Formula to determine sample size given a desired margin of error

15. Calculator sites • http://www.americanresearchgroup.com/moe.html • http://www.surveysystem.com/sscalc.htm

16. Power Analysis

17. Dillman formula