You want to survey a school

2. You want to survey a school • You draw your sample from the first day of school student enrollment list • This list would be your ____???____ • Which students are not on this list? • A phenomenon known as? • Potentially problematic because? • (Hint: Dillman, p. 196)

3. Some reminders… • Population: The group about whom we want to draw our inference • Sample Frame: Members of the population who could potentially be in our sample • Coverage Error: The extent to which members of population are excluded from sample frame (not good)

4. Welcome… • …to a hopefully productive lesson on SAMPLING METHODOLOGY! • What’s ideal? • Nifty tricks?? • Common misconceptions??? • Limitations of our methods????????? • P.S. We are going to do (some) math and it is going to be FUN!!!

5. Simple Random Sampling(what’s ideal) • Members of a sample frame, which hopefully includes our entire population, are selected one at a time • independently & without replacement • (Drawing names out of a hat) • Sample is equal in expectation to population on all outcomes, but no guarantees

6. Stratified Random Sampling(possibly even more ideal) • Use criterion to divide sample frame by group membership (e.g. racial category) • Randomly sample within each group • What is the advantage of this procedure?

7. Scenario… • We want to know what percentage of Americans support Obama for president • We need 1100 members from each racial group to be confident about group means (more on this later) • American Indians / Alaskan Natives comprise 1% of our population. • Through simple random sampling, how large of a sample would we theoretically need to reach n = 1100 for this subgroup?

8. Scenario cont’d… • OR, we could use stratified random sampling and draw 1100 from each subgroup without all this trouble. • BUT, now we have oversampled from American Indians--they are over-represented in our sample! • Implications? • Solutions?

9. (This data is very fake) • Proportion supporting B.O. African American: .50 Asian American: .50 Latino: .50 White: .50 American Indian: 0 Unweighted avg: ??

10. Weighting (nifty trick) • Now, let’s do a weighted average instead… What’s going on here? 99% (.50) + 1% (0) = 49.50% • Big difference, eh?

11. So, why was 1100 an ideal subgroup number? • Because no matter how large your population, a sample of 1100 will get you very close to the true population value if your outcome is binary (e.g. Obama: Yes or No) • How come?

12. Because this man said so • William Sealy Gossett (1876-1937) • Chemist, “math person”, Guinness Brewery worker • A patient man

13. Yes, a patient man • Using barley (somehow), spent two years empirically studying relationship between sample means and population means. • “The Probable (Standard) Error of a Mean” (1908) • Standard errors are what we use to estimate sampling error

14. Sampling error • Describes how closely our sample mean allows us to estimate our population mean • Conceptually similar to a confidence interval (Dillman, p. 207; http://www.researchsolutions.co.nz/sample_sizes.htm • Depends on: Population variance (“spread”) (estimated by sample variance) Sample size Population size (to a point)

15. Sampling error: big picture • Larger variances and (to a point) larger population sizes require larger samples to estimate the population mean at a given level of precision • Increasing sample size reduces sampling error, BUT there are diminishing returns to increasing our sample size

16. Sampling error: big picture • Diminishing Returns? For large populations… Increasing “n” from 100 to 200 is helpful Increasing from 500-600 is less helpful Increasing from 1200-1300 helps very little (no matter how large the population)

17. Why Diminishing Returns? • Because there is an upper bound (“ceiling”) on the variance of any sample. • For binary (Yes/no, “1” or “0”) outcomes, max variance is .25 • Thus, it’s only a matter of time till more “n” in the denominator makes our standard error very low

18. Why Diminishing Returns? • Even for continuous outcomes, there is still an upper bound on variance unless scale is infinite • Thus, there are still diminishing returns on increasing “n” • For more on this topic… -take S-012 -look up Confidence Intervals in stats books “You don't need a large sample of users to obtain meaningful data:
Continuous Data (e.g. Task Time)” http://www.measuringusability.com/sample_continuous.htm

19. Limitations of Sampling error calculations • Does not take coverage error into account! • Assumes you have drawn an simple random sample (e.g. does not take “clustering” into account)

20. Clustering??? • There are 20,000 students in a city with 40 schools. We want a sample of 1100 • Ideally, we would draw students at random from every school. • But, it would be cheaper and easier if we drew a few schools at random and obtained information from every student • Implications?

21. Clustering??? • If there is a lot of school-level variation in our outcome, our sample will not be representative and our sample estimate will be biased. • Sampling error formula does not account for this possibility

22. One more limitation of sampling error formula • Non-response bias • Even if you have drawn a beautifully random sample, your sample estimate will be biased if those who do not return your survey are different on your outcome of interest. • That’s why Dillman’s advice on getting high response rates is so important!