Maximizing Data Quality: The Importance of Experiments and Observational Studies

1. Chapter 4Gathering data • Learn …. How to gather “good” data About Experiments and Observational Studies

2. Section 4.1 Should We Experiment or Should we Merely Observe?

3. Population, Sample and Variables • Population: all the subjects of interest • Sample: subset of the population - data is collected on the sample • Response variable: measures the outcome of interest • Explanatory variable: the variable that explains the response variable

4. Types of Studies • Experiments • Observational Studies

5. Experiment • A researcher conducts an experiment by assigning subjects to certain experimental conditions and then observing outcomes on the response variable • The experimental conditions, which correspond to assigned values of the explanatory variable, are called treatments

6. Observational Study • In an observational study, the researcher observes values of the response variable and explanatory variables for the sampled subjects, without anything being done to the subjects (such as imposing a treatment)

7. Example: Does Drug Testing Reduce Students’ Drug Use? • Headline: “Student Drug Testing Not Effective in Reducing Drug Use” • Facts about the study: • 76,000 students nationwide • Schools selected for the study included schools that tested for drugs and schools that did not test for drugs • Each student filled out a questionnaire asking about his/her drug use

8. Example: Does Drug Testing Reduce Students’ Drug Use?

9. Example: Does Drug Testing Reduce Students’ Drug Use? • Conclusion: Drug use was similar in schools that tested for drugs and schools that did not test for drugs

10. Example: Does Drug Testing Reduce Students’ Drug Use? • What were the response and explanatory variables?

11. Example: Does Drug Testing Reduce Students’ Drug Use? • Was this an observational study or an experiment?

12. Advantages of Experiments over Observational Studies • We can study the effect of an explanatory variable on a response variable more accurately with an experiment than with an observational study • An experiment reduces the potential for lurking variables to affect the result

13. Experiments vs Observational Studies • When the goal of a study is to establish cause and effect, an experiment is needed • There are many situations (time constraints, ethical issues,..) in which an experiment is not practical

14. Good Practices for Using Data • Beware of anecdotal data • Rely on data collected in reputable research studies

15. Example of a Dataset • General Social Survey (GSS): • Observational Data Base • Tracks opinions and behaviors of the American public • A good example of a sample survey • Gathers information by interviewing a sample of subjects from the U.S. adult population • Provides a snapshot of the population

16. Section 4.2 What Are Good Ways and Poor Ways to Sample?

17. Setting Up a Sample Survey • Step 1: Identify the Population • Step 2: Compile a list of subjects in the population from which the sample will be taken. This is called the sampling frame. • Step 3: Specify a method for selecting subjects from the sampling frame. This is called the sampling design.

18. Random Sampling • Best way of obtaining a representative sample • The sampling frame should give each subject an equal chance of being selected to be in the sample

19. Simple Random Sampling • A simple random sample of ‘n’ subjects from a population is one in which each possible sample of that size has the same chance of being selected

20. Example: Sampling Club Officers for a New Orleans Trip • The five offices: President, Vice-President, Secretary, Treasurer and Activity Coordinator • The possible samples are: (P,V) (P,S) (P,T) (P,A) (V,S) (V,T) (V,A) (S,T) (S,A) (T,A)

21. The possible samples are: (P,V) (P,S) (P,T) (P,A) (V,S) (V,T) (V,A) (S,T) (S,A) (T,A) What are the chances the President and Activity Coordinator are selected? • 1 in 5 • 1 in 10 • 1 in 2

22. Selecting a Simple Random Sample • Use a Random Number Table • Use a Random Number Generator

23. Methods of Collecting Data in Sample Surveys • Personal Interview • Telephone Interview • Self-administered Questionnaire

24. How Accurate Are Results from Surveys with Random Sampling? • Sample surveys are commonly used to estimate population percentages • These estimates include a margin of error

25. Example: Margin of Error • A survey result states: “The margin of error is plus or minus 3 percentage points” • This means: “It is very likely that the reported sample percentage is no more than 3% lower or 3% higher than the population percentage” • Margin of error is approximately:

26. Be Wary of Sources of Potential Bias in Sample Surveys • A variety of problems can cause responses from a sample to tend to favor some parts of the population over others

27. Types of Bias in Sample Surveys • Sampling Bias: occurs from using nonrandom samples or having undercoverage • Nonresponse bias: occurs when some sampled subjects cannot be reached or refuse to participate or fail to answer some questions • Response bias: occurs when the subject gives an incorrect response or the question is misleading

28. Poor Ways to Sample • Convenience Sample: a sample that is easy to obtain • Unlikely to be representative of the population • Severe biases my result due to time and location of the interview and judgment of the interviewer about whom to interview

29. Poor Ways to Sample • Volunteer Sample: most common form of convenience sample • Subjects volunteer for the sample • Volunteers are not representative of the entire population

30. A Large Sample Does Not Guarantee An Unbiased Sample Warning: