CAUSE Teaching and Learning Webinar - January 11, 2011

1. Introducing Informal Inference Using Data-Centric Lab ExercisesRakhee Patel, Rob Gould and Gretchen DavisUCLA Department of Statistics CAUSE Teaching and Learning Webinar - January 11, 2011

2. Outline • Why teach informal inference early? • Informal inference using randomization • Introductory computer lab assignments • Implementation • Goals • Lab example 1: tuberculosis data • Lab example 2: births and smoking data • Discussion CAUSE Teaching and Learning Webinar - January 11, 2011

3. Informal Inference • Students often encounter difficulties with formal inference methods later in introductory courses • Particularly, abstract concepts are difficult • What is a null distribution/hypothesis? • How do we know when to reject the null? • What is a p-value? • Introducing informal inference early in a course may help with these struggles • Can do so without using complex vocabulary and mathematical machinery CAUSE Teaching and Learning Webinar - January 11, 2011

4. Randomization Tests • Randomization tests are a good way to illustrate these concepts in a way that students can see with their own eyes • Also a good way to illustrate power of computers • Can use many measures • In some cases, they more closely match the design of experiment • See Cobb (2007), The Introductory Statistics Course: A Ptolemaic Curriculum? CAUSE Teaching and Learning Webinar - January 11, 2011

5. Computer Lab Assignments • Students complete eight weekly lab assignments • Each lab uses real data to answer a particular investigative question • Lab also includes intermediate questions that help guide students to answer the investigative question • Students ultimately turn in set of summary questions regarding: • ultimate goals of lab • methods used • conclusions about research question • applications of concepts to real world CAUSE Teaching and Learning Webinar - January 11, 2011

6. Implementation Goals • To understand how concepts in lecture are applied to understanding real data • Design labs so that learning the software does not get in the way of doing statistical analysis • First lab assignment introduces Fathom software • Subsequent labs provide additional instructions for any new methods used • TAs will talk less, assist more. CAUSE Teaching and Learning Webinar - January 11, 2011

7. Informal Inference Goals • To understand why permuting observations simulates a null distribution • To understand that null sampling distribution is used to make decisions about rejecting null hypothesis • Develop intuition behind chance models • To understand how to use the null distribution to estimate the p-value & make decisions • Use chance model to make inference about actual data CAUSE Teaching and Learning Webinar - January 11, 2011

8. Example Lab 1: TB or Not TB? • Investigative Question • Is Streptomycin an effective treatment for tuberculosis? • Objectives • Using two-way tables, we can see how to make conclusions about cause-and-effect by comparing actual results to a chance model. • Data • Outcomes from Austin Bradford Hill’s first randomized study in 1948 examining the effects of the antibiotic Streptomycin vs. a control on 107 tuberculosis patients CAUSE Teaching and Learning Webinar - January 11, 2011

9. Example Lab 1: Intermediate Question 1 • Based on Hill’s study, does it seem that treatment and outcome are independent or dependent? Treatment and outcome appear to be dependent. 26% died in control group while only 7% died in Streptomycin group. CAUSE Teaching and Learning Webinar - January 11, 2011

10. Example Lab 1: Intermediate Question 2 • If treatment and outcome were independent, how many patients in the Streptomycin group, according to Hill’s study, would we expect to die if we replicated the study again? We would expect a proportion of 18/107 = 0.168 to die overall. So of the 55 patients in the Streptomycin group, we expect 0.168 of them, or 9.25 patients (roughly 9), to die. CAUSE Teaching and Learning Webinar - January 11, 2011

11. Example Lab 1: Intermediate Question 3 • Design and execute an appropriate simulation in Fathom using a chance model to replicate Hill’s study under the assumption that treatment and outcome are independent. Repeat the simulation 100 times and then use the results from the chance model to determine whether (a) or (b) is the most reasonable explanation for the actual data in Hill’s study. • (a) Streptomycin is a significantly more effective treatment for tuberculosis than bed rest. Thus, treatment and outcome are dependent. • (b) The actual difference between treatments is due to chance variation and Streptomycin may have no effect on tuberculosis. Thus, it is possible that treatment and outcome are independent. CAUSE Teaching and Learning Webinar - January 11, 2011

12. Example Lab 1: Intermediate Question 3 • Student discussions are crucial in determining how to set up the simulations, as there are many ways • Question 2 might guide them to look at the number of deaths in the Streptomycin group for each simulation As expected, the chance model shows an average of about 9 to 10 deaths in the Streptomycin group. Only 4 died in the actual data which never occurs in these simulations of the model, so the chance model does not fit well and (a) is more reasonable. CAUSE Teaching and Learning Webinar - January 11, 2011

13. Example Lab 1: Common Misconceptions • Students confuse randomized data and expectations with actual data: • Students look at the distance between the actual and the chance model rather than how often the actual occurs under the chance model If outcome is independent of treatment, we expect 9.25 deaths in the Streptomycin group. The chance model also shows an average of between 9 and 10 deaths, so we cannot reject chance. Since the actual value of 4 is far from the expected value of 9.25 then we reject the chance model. CAUSE Teaching and Learning Webinar - January 11, 2011

14. Example Lab 1: Intermediate Question 4 • Is Hill’s study an example of an observational study or an experiment? If we had seen a real difference between the Streptomycin group and the control group and concluded that Streptomycin was effective (and maybe you did in Question 3), can we conclude that the cause is the antibiotic? Why? Hill’s study is an experiment because he imposed the Streptomycin and control treatments on the patients, while keeping all other factors constant. If we found Streptomycin to be effective (which we did), we can conclude that the antibiotic is the cause of the lower number of deaths because Hill could completely control the study and treatments. CAUSE Teaching and Learning Webinar - January 11, 2011

15. Example Lab 1: Observations • Without using the terms null hypothesis or p-value, students should be able to find a significant difference between the groups and a causal relationship • Students may have trouble with the concepts and software used for this lab, but subsequent labs repeat the same techniques to illustrate the same general inference steps for various other types of data • It is important to show students several realizations from the chance model to illustrate what happens under the “no difference” hypothesis • Any differences that occur are strictly by chance CAUSE Teaching and Learning Webinar - January 11, 2011

16. Example Lab 2: Compared to What? • Investigative Question • How does birth weight differ between smoking and non-smoking mothers? • Objectives • Using statistical summaries and the chance model, we can explore whether mothers who smoke have babies with quantifiably different weights than mothers who do not. • Data • Characteristics of mothers and their babies for a sample of 1,000 births in North Carolina in 2004 CAUSE Teaching and Learning Webinar - January 11, 2011

17. Example Lab 2: Intermediate Question 1 • If a physician told you that a mother's smoking affected the birth of her baby, how much, typically, would you say the mother's smoking changes a baby's weight at birth, based on the actual data? I would guess (based on the sample we have) that smoking during pregnancy typically lowers the baby’s weight by around 0.25 lbs, the difference in median birth weights. CAUSE Teaching and Learning Webinar - January 11, 2011

18. Example Lab 2: Intermediate Question 2 • We will work with a chance model that assumes that smoking has NO effect on the babies' weights. • Using this chance model, simulate a set of data and calculate the difference in median birth weights between the smoking and non-smoking groups. Repeat this 100 times. • According to the chance model (your 100 simulations), what would you estimate is the typical difference in medians between the two groups? What would you consider to be an unusually large difference, according to the chance model? CAUSE Teaching and Learning Webinar - January 11, 2011

19. Example Lab 2: Intermediate Question 2 • Again, it is important for students to work together to determine how to set up the simulation I would expect the typical difference in median birth weights to be close to zero, which is reiterated by the average difference of -0.011 shown in the summary table. The plot shows that an unusually large difference seems to be around 0.2 lbs. CAUSE Teaching and Learning Webinar - January 11, 2011

20. Example Lab 2: Intermediate Question 3 • In the actual data, the true difference in median birth weights between the smoking mothers and non-smoking mothers is 0.25 pounds. How often did the chance model produce a difference this large or larger? In the 100 chance model simulations, I rarely saw a difference in medians as large as 0.25 lbs (the actual difference) as indicated by the dot plot. In fact, only 2 out of the 100 simulations resulted in a difference as large as the actual result. CAUSE Teaching and Learning Webinar - January 11, 2011

21. Example Lab 2: Intermediate Question 4 • We have mentioned two possible reasons for differences in the weights of babies from smoking mothers versus non-smoking mothers in the actual data: • Smoking mothers really do have lighter babies. • The difference between birth weights from smokers and non-smokers is due to chance. • Use your answer to Question 3 to justify which of these explanations seems most reasonable Since the actual difference in birth weights between the smoker and non-smoker groups rarely occurs using the 100 chance model simulations, the chance model does not seem to fit the data very well and therefore it seems that smoking mothers actually do have lighter babies. CAUSE Teaching and Learning Webinar - January 11, 2011

22. Example Lab 2: Intermediate Question 5 • Is this study an example of an observational study or an experiment? Justify your answer. If we had seen a real difference between birth weights from smokers and non-smokers and concluded that smoking and birth weight are associated (and maybe you did in Question 4), can we conclude that smoking causes low birth weight? Why? This is an observational study because the smoking was not imposed on the mothers. If we had found a real difference in birth weights between smokers and non-smokers (which we did), we still cannot conclude that smoking is the cause of the lighter babies. This due to the fact that with an observational study, we cannot control all factors of the study (such as other bad habits, for example) and so these other outside factors could affect the weight of the babies as well. CAUSE Teaching and Learning Webinar - January 11, 2011

23. Example Lab 2: Observations • Randomizing allows students to examine any measure on a data set (such as the median) • Again, students should see several iterations from the chance model to illustrate the “no difference” hypothesis • Once students practice using randomization to make inference, the intuition should become more and more clear • Does chance model fit the data well? • If not, what does that mean? CAUSE Teaching and Learning Webinar - January 11, 2011

24. Discussion • Repetition is key • Seeing is believing • Some future labs follow same structure • Fit model to data • Make inference about validity of model • Normal models can be thought of as nice shortcut to use when assumptions are met • Students generally show less frustration once the formal methods are taught • Performance on inference related problems has improved CAUSE Teaching and Learning Webinar - January 11, 2011

25. Final Impressions and Future Work • Based on graded lab examples and evaluations: • Students seem to understand why permuting simulates a null distribution, but are not sure what the null distribution itself represents • Students don't know how to use the null • Need more work to integrate the lessons learned in lab into the lecture • Perhaps more support materials are necessary • Integrating randomization-based testing with parametric testing is more difficult than anticipated CAUSE Teaching and Learning Webinar - January 11, 2011

26. Thank You • This work has been supported by grant NSF DUE‐0737126 • Contact information: • Rakhee Patel: rakhee@stat.ucla.edu • Rob Gould: rgould@stat.ucla.edu CAUSE Teaching and Learning Webinar - January 11, 2011