Collecting Data: Validity and Methods

1. Chapter 1 Collecting Data in Reasonable Ways Created by Kathy Fritz

2. Should you believe what you read? Data and conclusions based on data are everywhere • Newspapers • Magazines • On-line reports • Professional publications Will doing tai chi one hour per week increase the effectiveness of your flu shot? These type of questions are answered with data gathered from samples or from experiments. Will eating cheese before going to bed help you sleep better? Should you eat garlic to prevent a cold?

3. Population – the entire collection of individuals or objects that you want to learn about Sample – the part of the population that is selected for study

4. In the article, “The ‘CSI Effect’ – Does It Really Exist?”(National Institute of Justice [2008]: 1-7), the author speculates that watching crime scene investigation TV shows may be associated with the kind of high-tech evidence that jurors expect to see in criminal trials. Do people who watch such shows on a regular basis have higher expectations than those who do not watch them? Observational study– a study in which the person conducting the study observes characteristics of a sample selected from one or more existing populations. How would one go about answering this question? To answer this question, one would need to select a sample of people who watch these shows and a sample of people who do not. Interview these people to determine their level of expectation of high-tech evidence in criminal cases. This is called an observational study. The goal of an observational study is to use data from the sample to learn about the corresponding population.

5. Suppose a chemistry teacher wants to see the effects on students’ test scores if the lab time were increased from 3 hours to 6 hours. Experiment-a study in which the person conducting the study looks at how a response variable behavesunder different experimental conditions. Would you use an observational study to answer this question? Why or why not? Questions in the form “What would happen when . . .?” or “What is the effect of . . .?” CANNOT be answered with data from an observational study. An experimentMUST be used.

6. A big difference between an experiment and an observational study is . . . • in an experiment, the person carrying out the study determines who will be in what experimental groups and what the experimental conditions will be • in an observational study, the person carrying out the study does NOT determine who will be in what groups In the example about the “CSI effect”, the researcher did NOT determine whether or not the people watched investigative TV shows. In the example about increasing lab time in a chemistry class to see the effect on test scores, the teacher would determine which students are in the 3-hour lab and which are in the 6-hour lab.

7. Observational Studies • Purpose is to collect data that will allow you to learn about a single population or about how two or more populations will differ • Allows you to answer questions like “What is the proportion of . . .?” “What is the average of . . .?” “Is there an association between . . .?” The “ideal” study would be to carry out a census. Obtaining information about the entire population is called a census.

8. Why might we prefer to take select a sample rather than perform a census? Measurements that require destroying the item Measuring how long batteries last Safety ratings of cars Difficult to find entire population Length of fish in a lake Limited resources Time and money Census versus Sample Most common reason to use a sample

9. When you answer questions like “What is the proportion of . . .?” “What is the average of . . .?” You are interested in the population characteristic. A population characteristic is a number that describes the entire population. A statistic is a number that describes a sample. How can we be sure that the sample is representative of the population? One way is to take a simple random sample. It is important that a sample be representative of the population.

10. A sample of size n is selected from the population in a way that ensures that every different possible sampleof the desired size has the same chance of being selected. Suppose you want to select 10 employees from allemployees of a large design firm. Number each employee with a unique number. Use a random digit table random, a random number generator, or numbers selected from a hat to select the 10 employees for the sample. Simple Random Sample The letter n is used to denote sample size; the number of individuals or objects in the sample. In order to be a simple random sample – EVERY sample of size 10 MUSThave an equal chance to occur. Thus, it is possible that 10 full-time, 10 part-time, or any combination of full-time and part-time employees are selected. What is the value of n ? To select a simple random sample, create a list (called a sampling frame) of all the employees in the firm.

11. The following is part of the random digit table found in the back of your textbook: How to use a Random digit table Suppose our design firm has 250 employees. Number employees from 1 to 250. Select 3-digit numbers from the table. If the number is not within 1-250, ignore it. The sample would be the employees that correspond to the selected numbers.

12. Sampling in which an individual or object, once selected, is put back into the population before the next selection. This allows an object or individual to be selected more than once for a sample. Sampling with replacement In practice, sampling with replacement is rarely used.

13. Sampling in which an individual or object, once selected, is NOTput back into the population before the next selection. That is once an individual or object is selected, they are not replaced and cannot be selected again. Sampling without replacement In practice, sampling without replacement is more common. Although sampling with and without replacement are different, they can be treated as the same when the sample size n is relatively small compared to the population size (no more than 10% of the population).

14. Can a smallersample size be representative of the population? population Although it is possible to obtain a simple random sample that is not representative of the population, this is likely ONLY when the sample size is very small. Samples of varying sizes Notice that the sample of size 50 is still representative of the population.

15. The population is first divided into non-overlapping subgroups(called strata). Then separatesimple random samples are selected from each subgroup (stratum). Suppose we wanted to estimate the average cost of malpractice insurance for doctors in a particular city. We could view the population of all doctors in this city as falling into one of four subgroups: (1) surgeons, (2) internists and family practitioners, (3) obstetricians, and (4) all other doctors. Now let's take a look at other sampling methods. Stratified Random Sample Strata are subgroups that are similar (homogeneous) based upon some characteristic of the group members. The real advantage of stratified sampling is that it often allows you to make more accurate inferences about a population than does simple random sampling. In general, it is much easier to produce relatively accurate estimates of characteristics of a homogeneous group than of a heterogeneous group. Stratified random sampling is often easier to implement and is more cost effective than simple random sampling.

16. Cluster Sampling Sometimes it is easier to select groups of individuals from a population than it is to select individuals themselves. Cluster sampling divides the population of interest into nonoverlapping subgroups, called clusters. • Which would be easier to do? • Find 75 randomly selected individuals • Find 3 randomly selected homerooms Clusters are then selected at random, and ALL individuals in the selected clusters are included in the sample. Suppose that a large urban high school has 600 senior students, all of whom are enrolled in a first period homeroom. There are 24 senior homerooms, each with approximately 25 students. The school administrators want to select a sample of roughly 75 seniors to participate in a survey. Randomly selecting 3 senior homerooms and then include all the students in the selected homerooms in the sample. The ideal situation for cluster sampling is when each cluster mirrors the characteristics of the population.

17. Put into strata (homogenous groups) Then random select individuals from each group Stratification Clustering Put into clusters Then random select entire clusters Be careful not to confuse clustering and stratification! Sample

18. Selects an ordered arrangement from a population by first choosing a starting point at random from the first k individuals then every k th individual after that Suppose you wish to select a sample of faculty members from the faculty phone directory. You would first randomly select a faculty from the first 20 (k = 20) faculty listed in the directory. Then select every 20th faculty after that on the list. 1 in k Systematic Sampling k is often selected so that a certain sample size is produced. Let N = population size and n = sample size. k = N ÷ n This method works reasonably well as long as there are no repeating patterns in the population list.

19. Selecting individuals or objects that are easy or convenient to sample. Suppose your statistics professor asked you to gather a sample of 20 students from your college. You survey 20 students in your next class which is music theory. Will this sample be representative of the population of all students at your college? Why or why not? Convenience Sampling Convenience sampling is rarely representative of the population, so DON’T USE IT!

20. Voluntary response is a type of convenience sampling which relies solely on individuals volunteering to be part of the study. People who are motivated to volunteer responses often hold strong opinions. It is extremely unlikely that they are representative of the population!

21. Identify the sampling method 1)The Educational Testing Service (ETS) needed a sample of colleges. ETS first divided all colleges into 6 subgroups of similar types (small public, small private, medium public, medium private, large public, and large private). Then they randomly selected 3 colleges from each group. Stratified random sample

22. Identify the sampling design 2) A county commissioner wants to survey people in her district to determine their opinions on a particular law up for adoption. She decides to randomly select blocks in her district and then survey all who live on those blocks. Cluster sample

23. Identify the sampling design 3) A local restaurant manager wants to survey customers about the service they receive. Each night the manager randomly chooses a number between 1 & 10. He then gives a survey to that customer, and to every 10th customer after them, to fill it out before they leave. Systematic sampling

24. Consider the following example: In 1936, Franklin Delano Roosevelt had been President for one term.  The magazine, The Literary Digest, predicted that Alf Landon would beat FDR in that year's election by 57 to 43 percent.  The Digest mailed over 10 million questionnaires to names drawn from lists of automobile and telephone owners, and over 2.3 million people responded - a huge sample. At the same time, a young man named George Gallup sampled only 50,000 people and predicted that Roosevelt would win.  Gallup's prediction was ridiculed as naive.  After all, the Digest had predicted the winner in every election since 1916, and had based its predictions on the largest response to any poll in history.  But Roosevelt won with 62% of the vote.  The size of the Digest's error is staggering.  Biasis the tendency for samples to differ from the corresponding population in some systematic way. This is a classic example of how bias affects the results of a sample!

25. Selection bias Occurs when the way the sample is selected systematically excludes some part of the population of interest May also occur if only volunteers or self-selected individuals are used in a study People with unlisted phone numbers – usually high-income families People without phone numbers –usually low-income families People with ONLY cell phones – usually young adults Sources of bias Suppose you take a sample by randomly selecting names from the phone book – some groups will not have the opportunity of being selected!

26. Measurement or Response bias Occurs when the method of observation tends to produce values that systematically differ from the true value in some way Improperly calibrated scale is used to weigh items Tendency of people not to be completely honest when asked about illegal behavior or unpopular beliefs Appearance or behavior of the person asking the questions Questions on a survey areworded in a way that tends to influence the response People are asked if they can trust men in mustaches – the interviewer is a man with a mustache. Sources of bias Suppose we wanted to survey high school students on drug abuse and we used a uniformed police officer to interview each student in our sample – would we get honest answers? A Gallup survey sponsored by the American Paper Institute (Wall Street Journal, May 17, 1994) included the following question: “It is estimated that disposable diapers accounts for less than 2% of the trash in today’s landfills. In contrast, beverage containers, third-class mail and yard waste are estimated to account for about 21% of trash in landfills. Given this, in your opinion, would it be fair to tax or ban disposable diapers?”

27. Nonresponse occurs when responses are not obtained from all individuals selected for inclusion in the sample To minimize nonresonse bias, it is critical that a serious effort be made tofollow up with individuals who did not respond to the initial request for information Sources of bias The phone rings – you answer. “Hello,” the person says, “do you have time for a survey about radio stations?” You hang up! How might this follow-up be done?

28. Sources of bias Will increasing the sample size reduce the effects of bias in the study? No, it does nothing to reduce bias if • The method of selection is flawed • If non-response is high

29. Identify one potential source of bias. Suppose that you want to estimate the total amount of money spent by students on textbooks each semester at a local college. You collect register receipts for students as they leave the bookstore during lunch one day. Convenience sampling – easy way to collect data or Selection bias – students who buy books from on-line bookstores are excluded.

30. Identify one potential source of bias. To find the average value of a home in Plano, one averages the price of homes that are listed for sale with a realtor. Selection bias – leaves out homes that are not for sale or homes that are listed with different realtors. (other answers are possible)

31. The article “What People Buy from Fast-Food Restaurants: Caloric Content and Menu Item Selection” (Obesity [2009]: 1369-1374) reported that the average number of calories consumed at lunch in New Your City fast-food restaurants was 827. The researchers selected 267 fast-food locations at random. The paper states that at each of these locations “adult” customers were approached as they entered the restaurant and asked to provide their food receipt when exiting and to complete a brief survey. Will this study result in data that is representative of the population?

32. The people who eat at fast-food restaurants in NYC Yes – because the researchers randomly selected the fast-food restaurants, this is a reasonable way to select the sample Yes – because the researchers randomly selected the fast-food restaurants, it is reasonable to regard the people eating at these locations as representative • Two potential sources of bias • Response bias since customers were approached before they ordered • Nonresponse – some people refused to participate

33. Experimental Design

34. Experiments Suppose we are interested in determining the effect of room temperature on the performance on a first-semester calculus exam. So we decide to perform an experiment. What variable will we “measure”? the performance on a calculus exam What variable will “explain” the results on the calculus exam? the room temperature This is called the explanatory variable. Explanatory variables– those variables that have values that are controlled by the experimenter (also called factors) This is called the response variable. Response variable– a variable that is not controlled by the experimenter and that is measured as part of the experiment

35. Room temperature experiment continued . . . We decide to use two temperature settings, 65° and 75°. How many treatments would our experiment have? the 2 treatments are the 2 temperature settings Experimental condition– any particular combination of the explanatory variables (also called treatments)

36. Room temperature experiment continued . . . This is an example of a confounding variable. Confounding variable – two variables are confounded when their effects on the response can NOT be distinguished. Suppose we have 10 sections of first-semester calculus that have agree to participate in our study. On who or what will we impose the treatments? the 10 sections of calculus Should the instructors of these sections be allowed to select to which room temperature that their sections are assigned? No, since the instructor would probably select the same temperature for all their sections, then it would be difficult to tell if the scores are due to the temperature or to the instructor’s teaching style These are our subjectsorexperimental units. Experimental units– the smallest unit to which a treatment is applied.

37. Designing Strategies for Single Comparative Experiments The goal of a single comparative experiment is to determine the effects of the treatment on the response variable. To do this: You must consider other potential sources of variability in the response • Eliminate them OR • Ensure they produce chance-like variability

38. Room temperature experiment continued . . . In an experiment, these other variables need to be “controlled”. Direct controlis holding the other variables constant so that their effects are not confounded with those of the experimental conditions (treatments). Remember – the explanatory variable is the room temperature setting, 65° and 75°. The response variable is the grade on the calculus exam. Are there other variables that could affect the response? What about the variables that the experimenter can’t directly control? What can be done to avoid confounding results? Can the experimenter control these other variables? If so, how? Instructor? Textbook? Time of day? Ability level of students?

39. Room temperature experiment continued . . . Remember – the explanatory variable is the room temperature setting, 65° and 75°. The response variable is the grade on the calculus exam. The experimenter cannot control who the instructors are. Therefore, the instructors may be potentially confounding. Another way to control a variable is to block by that variable. We use each instructor as his/her own block. Then sections of each instructor will be randomly assigned to the two treatments. Instructor? Textbook? Time of day? Ability level of students?

40. Room temperature experiment continued . . . What about other variables that we cannot control directly or that we don’t even think about? Random assignment should evenly spread all other variables, that are not controlled directly, into all treatment groups. We expect these variables to affect all the experimental groups in the same way; therefore, their effects are not confounding.

41. Room temperature experiment continued . . . To randomly assign the 10 sections of first-semester calculus to the 2 treatment groups, we would first number the classes 1-10. Place the numbers 1-10 on identical slips of paper and put them in a hat. Mix well. There are 10 sections. This is called replication. Replication ensures that there is an adequate number of observations for each experimental condition. 8 5 7 3 9 3 9 7 5 8 1 2 4 6 10 Randomly select 5 numbers from the hat. Those will be the sections that have the room temperature set at 65°. The remaining sections will have the room temperature set at 75°.

42. Key Concepts of Experimental Design Direct controlholds potential confounding variables constant so their effects are not confounded with the treatments. Blocking uses potentially confounding variables to create groups (blocks) that are similar. All experimental conditions (treatments) are then tried in each block. Random assignmentremoves the potential for confounding variables by creating equivalent experimental groups Random assignment is a criticalcomponent of a good experiment. Replication ensures that there is an adequate number of observations for each treatment.

43. What to do with Potentially Confounding Variables • Potentially confounding variables that are known and incorporated into the experimental design: • Use: • Direct control – hold potentially confounding variables fixed • Blocking – allow for valid comparisons because each treatment is tried in each block • Potentially confounding variables that are NOT known or NOT incorporated into the experimental design: • Use: • Random assignment

44. Measure response for A Experimental Units Treatment B Treatment A Compare treatments Measure response for B Types of Experimental Design Completely Randomized Design An experiment in which experimental units are randomly assigned to treatments is called a completely randomized experiment. Random Assignment

45. Experimental Units Compare the results from the 2 blocks Measure response for A Measure response for A Treatment B Treatment A Treatment A Treatment B Block 2 Compare treatments for block 2 Compare treatments for block 1 Block 1 Measure response for B Measure response for B Randomized Block Design An experiment that incorporates blocking by dividing the experimental units into homogeneous blocks and then randomly assigns the individuals within each block to treatments is called a randomized block experiment. Random Assignment Create blocks Random Assignment

46. Can moving their hands help children learn math? An experiment was conducted to compare two different methods for teaching children how to solve math problems of the form 3 + 2 + 8 = ___ + 8. One method involved having students point to the 3 + 2 on the left side of the equal sign with one hand and then point to the blank on the right side of the equal sign before filling in the blank to complete the equation. The other method did not involve using these had gestures. To compare the two methods, 128 children, ages 9 and 10, were randomly assigned to the two experimental conditions. This is an example of what type of experimental design? Completely Randomized Design

47. Measure number correct on math test 128 children Math without hand gestures Math with hand gestures Compare number correct for those who used hand gestures and those who did not Measure number correct on math test A Diagram of the math experiment: This is a completely randomized experiment. The 128 children are randomly assigned into the two treatment groups. Random Assignment

48. Can moving their hands help children learn math? Suppose that you were worried that gender might also be related to performance on the math test. One possibility would be to use direct control of gender – use only boys or only girls. But, any conclusions can ONLY be generalized to the group that was used. Another strategy is to incorporate blocking into the design. The researchers could create two blocks, one consisting of girls and one consisting of boys. Then, once the blocks are formed, randomly assign the girls to the two treatments and randomly assign the boys to the two treatments. This is an example of a Randomized Block Design.

49. Compare number correct with hand gestures and without hand gestures 128 Children Measure number correct Measure number correct Math with no hand gestures Math with hand gestrues Math with hand gestires Math with no hand gestures 47 boys Compare number correct with hand gestures and without hand gestures for boys 81 girls Compare number correct with hand gestures and without hand gestures for girls Measure number correct Measure number correct A Diagram of the math experiment: The 128 students are blocked by gender. Then the students are randomly assigned to the two treatments. Random Assignment Create blocks Random Assignment

50. Revisit the room temperature experiment . . . In the room temperature experiment, we have only 2 treatment groups, 65° and 75°. We do NOT have a control group. A control group • allows the experimenter to assess how the response variable behaves when the treatment is not used. • provides a baselineagainst which the treatment groups can be compared to determine whether the treatment had an effect. Control group- is an experimental group that does NOT receive any treatment. In experiments that use human subjects, the use of a placebo may be necessary. A placebo is something that is IDENTICAL to the treatment received, except it contains NO active ingredients.