Methods of Comparison: Controlled Experiments and Observational Studies

Methods of Comparison: Controlled Experiments and Observational Studies Math 1680

Overview • Introduction • Controlled Experiments • Minimizing Bias • Observational Studies • Association, not Causation • Comparing Rates • Salk Vaccine Trial • Breast Cancer Screening • Summary

Introduction • Suppose a company wanted to market a weight-loss pill • After design is complete, the company still needs to answer one basic question • Does the pill actually work? • How can the company determine this?

Introduction • One method would be to give the pill to everyone who wants to lose weight and see if they actually do • What’s wrong with this approach? • First of all, the cost of giving everyone the pill is prohibitive, and there may be dangerous side effects for the untested pill. • Secondly, people who want to lose weight may be exercising or eating more healthily, which could overshadow the effects of the pill (This is an example of a confounding factor.)

Controlled Experiments • To test the pill, two groups are created from a sample of people • Control group is given a placebo • Treatment group is given the pill

Controlled Experiments • Why is it important that the people don’t know which group they are in? • It has been shown repeatedly that a patient's symptoms can be alleviated by an otherwise ineffective treatment if the individual expects or believes that it will work. Conversely, members of a control group knowingly receiving an inert substance tend to report a worsening of symptoms. This phenomenon is called the placebo effect.

Controlled Experiments • The most ideal form of experiment is one where neither the subjects nor the evaluators know who received the treatment • This is a double-blind experiment • This design provides the least opportunity for biased results

Controlled Experiments • After a period of time, the company checks to see how much weight people in each group have lost • If there is a significant difference between the groups, assume the pill was the reason • If there is no significant difference between the groups, assume the pill had no effect

Minimizing Bias • What if the control group and the treatment group differ in some way other than the treatment? • Then we have confounding factors in the study, and the results will be biased and/or invalid • Confounding factors can include… • Age • Race • Gender • Socio-economic status (SES) • Location • Political/religious orientation

Minimizing Bias • In the pill example, suppose all of the men were placed into the treatment group and all of the women were placed into the control group • What is wrong with this design? The pill may interact differently with people based on gender. For example, suppose that high levels of testosterone diminish the effect of the pill. Then the results would be biased against the pill’s effectiveness.

Minimizing Bias • In order to minimize the risk of bias, researchers assign people to each group randomly from the pool of subjects • If enough people are in the pool, the groups will look similar in terms of age, race, gender, or any other demographic which could be a confounding element

Observational Studies • How would you design a controlled experiment to test if second-hand smoke causes lung cancer? Answer: You can’t, really. The time scale is too large and it would be virtually impossible to ensure that your control group never got second-hand smoke exposure.

Observational Studies • In many studies, designing a controlled experiment is simply not possible • In such cases, researchers must settle for observational studies • Observational studies let the subjects select which group to join • Observational studies should be placed under great scrutiny because the control and treatment groups may be quite different in many ways • These differences may introduce confounding influences on the study results

Observational Studies • A TV commercial observes that volunteers who switched to having to Cheerios for breakfast reduced their cholesterol level by 4%. Does this show that Cheerios cereal is effective in reducing your cholesterol level? Not necessarily. People who volunteer for such a study may already be interested in improving their health and may be taking steps such as exercising daily. This could lower their cholesterol level.

Observational Studies • One form of observational study involves using historical controls to compare a modern treatment with • The difference in conditions between past and present can itself be a confounding factor • For example, in testing a polio vaccine, one would not want to compare infection rates in the 1950’s with infection rates in the 1900’s because of the large differences in standard of living and health care quality in the two periods

Association, not Causation • An important limitation of all studies is that they can only show the strength of association between two variables • Association may point to causation • For example, if exposure to a virus causes a disease, then people who are exposed should be sicker than similar people who are not exposed • However, a study alone cannot show causation • Human inference is needed to connect causal links • Generally, a casual link is accepted if • A scientific theory is derived which explains the association • Enough studies controlling all potential confounding factors are performed, and all of them show the same association

Association, not Causation • Cervical cancer is more common among women who have been exposed to the herpes virus, according to many observational studies. Is it fair to conclude that the virus causes cervical cancer? No. Both herpes and cervical cancer have been shown to be sexually transmitted diseases. Women who have many partners are at higher risk for both diseases than women with few partners, so the number and type of partners is a confounding influence which explains the association.

Association, not Causation • Some studies find an associationbetween liver cancer and smoking. Does this mean smoking causes liver cancer, or is there a confounding influence at play? People who smoke also tend to drink, so drinking is associated with smoking. It has been shown repeatedly that over consumption of alcohol can cause liver cancer.

Comparing Rates • Sometimes the groups in comparison are of different sizes • Rather than comparing the actual numbers, use rates or proportions such as percents • Adjusting to rates allows for a more direct comparison

Comparing Rates • In the U.S. in 1990, there were 2.1 million deaths from all causes compared to 1.7 million in 1960 • According to Census data, there were about 177 million Americans in 1960 and about 250 million in 1990 • True or false: The public’s health got worse over the period 1960-1990. False. Convert the death counts to percentage rates. The death rate in 1960 was 0.96%, while the death rate in 1990 was 0.84%. If anything, public health improved.

Salk Vaccine Trial • In 1954, researchers tested the effectiveness of a vaccine for polio • Approximately 400,000 children were selected • All children in the study had parental consent to be vaccinated (Why?) • Children randomly divided into equal sized treatment and control groups • Control group was given a placebo • Is this a controlled experiment or an observational study?

Salk Vaccine Trial • Results • Was the vaccine effective?

Breast Cancer Screening • In 1963, an insurance company used 62,000 women (all insured by the plan) as subjects • Women ranged in age from 40 to 64 • 31,000 of the women were used as controls • The others were encouraged to undergo annual screening • Roughly 20,200 women from the treatment group came for annual screenings, the other 10,800 refused • After 5 years, the death rates were checked

Breast Cancer Screening • Is this a controlled experiment or an observational study? It depends on what questions you are trying to answer. If you are only interested in the difference between the treatment and control groups, then you are using the results of a controlled experiment. However, when you start drawing inferences between the screened and refused groups, you are looking at an after-the-fact observational study.

Breast Cancer Screening Researchers noted three things: 1.Screening had little impact on diseases other than breast cancer. 2.Poorer women were less likely to accept screening than richer ones. 3.Most diseases fall more heavily on the poor than on the rich.

Breast Cancer Screening • Does screening save lives? Which numbers prove your point?

Breast Cancer Screening • Why is the death rate from all other causes in the whole treatment group about the same as the rate for the control group?

Breast Cancer Screening • Why is the death rate from all other causes higher for the “refused group” than the examined group?

Breast Cancer Screening Unlike most diseases, breast cancer afflicts the rich more than the poor. Which numbers in the table confirm this association?

Case Study: Child care and behavior • Question: If parents choose to use child care, are they more likely to see undesirable behaviors in their children? • Question: How should one find out? By use of controlled experiments, or by observational study?

Case Study: Child care and behavior • Answer: In 1991, a study commenced on 1364 infants (“subjects”) and followed them through their sixth year in school. 12 years later, an article was published. “the more time children spent in child care from birth to age four-and-a-half, the more adults tended to rate them, both at age four-and-a-half and at kindergarten, as less likely to get along with others, as more assertive, as disobedient, and as aggressive.”

Case Study: Child care and behavior • A summary of the study noted, “The study authors noted that their study was not designed to prove a cause and effect relationship. …” (that more child care time causes more aggressive behavior) • Possible confounding factors? Perhaps… perhaps … perhaps…

Case Study: Child care and behavior • Question: How about a controlled experiment instead? (i.e. the researchers select children to place in child care) • What are the advantages? • What problems might arise?

Summary: Basic Principles of Statistical Experimental Design • Raise question to answer: Explanatory variable(s) & Response variable(s). • Leading principle: Comparison. • Other principles: randomize, repeat (or involve many subjects to reduce chance variation) • Why:

Summary: Principles of Statistical Experimental Design • Limitations: Statistical analysis of an experiment cannot tell us how far the results will generalize to other settings. (Experiment run in Massachusetts; how about in the entire USA?) • However, the randomized comparative experiment, because of its ability to give convincing evidence for causation, is one of the most important ideas in statistics.

More Elaborate Statistical Designs • Matched Pairs Design: Pair subjects with the same age, sex, income, etc. and compare their responses • Example: Compare two advertisements for the same product • Common variation of MPD: impose both treatments on the same subjects, so that each subject serves as his or her own control. (See HW2, Q9)

More Elaborate Statistical Designs • Question: How does randomization work here? • Answer: Which one of a matched pair sees the first ad is decided at random. Or, which treatment a person gets first is decided at random if he or she also serves as his or her own control. • Additional Example: Matched Pairs for the cell phone experiment. (Details: trained in using simulator, order in which a subject drives with and without the phone be random, two drives be on separate days)

Summary • Statisticians use the method of comparison • To determine the effectiveness of a “treatment”, they usually compare the responses of a treatment group with a control group • If the control group is comparable to the treatment group (apart from the treatment), then a significant difference in the responses of the two groups is likely to be due to the effect of the treatment

Summary • If the treatment group is different from the control group with respect to other factors, the effects of these other factors are likely to be confounded with the effect of the treatment • To make sure the treatment group is like the control group, investigators put subjects into either group at random

Summary • Whenever possible, the control group is given a placebo • The response should be to the treatment itself rather than to the idea of the treatment • In a double-blind experiment, the subjects do not know whether they are in treatment or in control, and neither do those who evaluate the responses • Guards against bias, both in the responses and in the evaluations

Summary • In an observational study, the investigators do not assign the subjects into treatment or control groups • Subjects which have the condition are the treatment group and the ones which do not are the control • Observational studies can establish association, which is not necessarily causation

Summary • Observational studies are particularly susceptible to confounding factors • This can make them very misleading about cause and effect relationships • With observational studies, try to find out how the subjects came to be in treatment or control • Are the groups comparable? • What factors are confounded with the treatment? • What adjustments (if any) did the investigators make to control for these factors?

Methods of Comparison: Controlled Experiments and Observational Studies