Applied Quantitative Methods

1. Applied Quantitative Methods Lecture 1. Causal Inferences and Experimental Design September 22nd, 2010

2. Course Information Title: Applied Quantitative Methods Language: English Lectures Lecturer: Renata Ivanova Time: Wednesday, 9:15- 10:45, room RB 212 Office hours (Renata): Wednesday, 11:30 – 13:00, NB 339 Exercise sessions Teaching Assistant (TA): Jan Zouhar Time: Thursday, 11:00-12:30 and 14:30-16:00 Office hours (Jan): Tuesday, 11:00 – 12:30, Wednesday, 10:30- 12:00, NB 431 N!B! Course page: http://home.cerge-ei.cz/Ivanova/teaching.html

3. Course Information (Cont.) • Reading materials • Book: Wooldridge, J. (2003). Introductory Econometrics: A Modern Approach. Mason Thomson. 2nd edition • Articles: Compulsory and recommended (on the course page) • Course calendar • Lectures 1-6 • Midterm exam • Lectures 7-13 • Final exam • Term paper

4. Course Policies • Attendance • Recommended though not compulsory • No laptops allowed during the lecture • Taking notes is not compulsory • Late arrivals will not be tolerated • Class participation is highly encouraged! • Academic dishonesty (cheating) will never be tolerated • Grading • Your grade (100 %) = Midterm (20 %) + Final (40 %) + Term paper (40 %) • Final exam • December 20th, 2010 – February 2nd, 2011 • Make ups

5. 1.Causal Inferences

6. Ice-cream consumption and crime rates

7. Correlation between X and Y: • Causal relationship: changes in X drive changes in Y, or visa versa • Ice-cream causes violent behavior or increasing crime rates make people nervous and they demand more sweets to feel happier • Third variable Z affects X and Y • Outside temperature affects ice-cream consumption and crime rates • 3. Coincidence – meaningless relationship • Bottom line: Correlation does not imply causality!

8. Causal relationship between X and Y: • Variation in X (independent variable) leads to the change in Y (dependent variable) • X is a cause; Y is an effect • “If –Then” statement: • If X, then Y • If not X, then not Y • Criteria for causality (Cook & Campbell, 1979): • Association (correlations) between potential cause and effect • Temporal precedence: Cause must precede the effect in time • No plausible alternative explanation

9. Public press headlines • Facebook users get bad grades in college • Church attendance boosts immunity • Breast implants boost suicide • Credit cards make you fat • Eating fish prevents crime • Misinterpretationof correlation with causation • Be skeptical about causal claims!!!

10. Research Design: Experiments • Research question (Krueger, 1999; QJE): What is the effect of class size on the students’ academic achievement (GPA)? • Hypothesis: Students who attend small classes (< 20) study better compared to students who attend regular (large) classes • Data: GPA of students in large and small classes • Descriptive statistics: average GPA in small classes is roughly the same as in big classes.

11. Research Design: Experiments (Cont.) Our conclusion? Class size does not affect the performance. Policy implication – schools will save money by cutting the number of teachers and gathering students in bigger classes! Descriptive statistic (average GPA in each group) doesn’t reflect the true effect of class size on achievement. General practice to assign weak students to smaller classes – selection bias

12. Counterfactual worlds For any pupil i exist two potential outcomes: Y0i and Y1i Y0 i – GPA of pupil i if in regular class Y1i - GPA of pupil i if in small class Treatment (D): attendance of small class Di = 1, if pupil attends small class Di = 0, if pupil attends regular class Effect of treatment: Y1i - Y0i (individual specific and random) Problem: we observe either Y1i or Y0i , but never both => missing data Yi = Y1iDi + Y0i(1 - Di) = Y0i + (Y1i - Y0i)Di Causal effect

13. Average Treatment Effect (ATE) • Expected effect for randomly drawn person from the population • ATE = E(Y1- Y0) • Averages across entire population • ATE = E(Yi | Di=1) - E(Yi | Di=0) = • = E(Y1i | Di=1) - E(Y0i | Di=1) + E(Y0i | Di=1) - E(Y0i | Di=0) • Counterfactual: E(Y0i | Di=1) • What GPA would have pupil i if he got into small class Average causal effect of small class for pupils in small class Selection bias The goal of empirical research is to overcome selection bias

14. Solution # 1:Random Assignment Pupils are assigned to small and regular classes by random process (tossing a coin, tables of random numbers) Selection bias vanishes: Di and Yi are independent => E(Y0i | Di=1)= E(Y0i | Di=0) The effect of random assignment to small classes on pupils from small classes is the same as the effect of small class attendance on the average pupil from the population ATE = E(Y1i | Di=1) - E(Y0i | Di=1) = = E(Y1i -Y0i | Di=1) = E(Y1i -Y0i)

15. Experiments with random assignment • Experiment – a study in which an intervention is deliberately introduced to observe the effect • “Golden standard” – 3 criteria for causality • Elements of experimental design • Two groups: treatment (experimental) & control • - Random assignment • Manipulation of the potential cause by the researcher • Areas of application • Medicine: testing the effect of new drugs (blind and double-blind experiments) • Labor economics: effect of training programs on employment and earnings • Education research: effect of classroom environment on students’ performance

16. The Concept of Validity Validity - the quality of research conclusions, “approximation of truth” Threats to validity: why conclusions or inferences may be wrong Internal validity: Assuming that there is a relationship, is this relationship a causal one? Threats: Selection bias, small sample size or a non-random sample, historical events influencing the outcome, unreliable measures, etc. External validity: How generalizable the established effect to other contexts? To which degree the conclusions in the study hold for other people, settings and at other times.

17. Experiments with Random Assignment (Cont.) • Strengths • - Strong internal validity: satisfy 3 criteria of causality • Identification of causal mechanism • The highest degree of control over the context • Shortcomings • Limited possibilities for generalization (low external validity) • High costs • Not always feasible (non-manipulable age, gender, race) or socially acceptable • Subjects behave differently if they learn that they are under experiment

18. A. Krueger (1999), QJE22 • Education production: effect of class size on pupils’ achievement • STAR: Student/Teacher Achievement Ratio (Tennessee, USA) • Longitudinal study: starting date 1985 with 4 years of follow up • Sample: 11,600 participants (over 4 years) in 80 schools • Assignment of pupils to three types of classes: • Small (13-17 pupils) • Regular (22-25) • Regular with full-time teacher aide (22-25) • Randomization algorithm: within school • Realization: tracking test scores for 4 years in each class type within the school • Standardized tests on reading ability, word recognition and mathematics

19. A. Krueger (1999), QJE (Cont.) Key issue: whether randomization successfully balanced pupils’ characteristics across different treatment groups -pupils assigned to different classes are otherwise comparable - STAR experiment did not collect pretreatment test scores Descriptive statistics: starting point of any empirical research Pupils’ characteristics: age, race, being from disadvantageous family F-test H0: Sample means are equal across three class types Conclusion: Within schools there is no apparent evidence that initial assignment to class type was correlated with student characteristics

22. A. Krueger (1999), QJE (Cont.) Kernel density • An average pupilin small classes performed better than the average pupil in regular classes; • Average score of pupils who entered experiment in higher grades was lower

23. A. Krueger (1999), QJE (Cont.) Econometric model Yics- average score of student i in class c at school s 1, if student assigned to small class SMALLcs = 0, otherwise 1, if student assigned to regular class with an aide REG/Acs = 0, otherwise Xics - a vector of observed pupils and teacher covariates αs – school dummy

24. A. Krueger (1999), QJE (Cont.) Pupil char. Teacher char.

25. A. Krueger (1999), QJE (Cont.)

26. A. Krueger (1999), QJE (Cont.) • Major findings • Pupils in small classes perform better than those in regular classes • Average performance gap • Kindergarten: 6 percentile points • 1st grade: 8.6 percentile points • 2nd and 3rd: 5-6 percentile points • Pupils in regular class with aide perform slightly better than in regular class • The main benefit of small class arises after 1st year • After the 1st year, additional effect on performance is smaller • Girls perform 3-4 percentile points better than boys • “Free lunch” children perform worse • Teacher characteristics explain little achievement on tests

27. Research Design: Natural Experiments • Natural experiment (economics) = Quasi-experiment (psychology) • Observational (non-experimental) studies • Assignment to treatment and control is “as if” random • Transparent exogenous source of variation in the explanatory variable that determines the treatment assignment • Sources of exogenous variation: Differences in government policies and laws • Validating the randomization claim • “Pre-treatment equivalence” – equivalence of units w/r to factors other than treatment • External validity: Results might not be generalizable across other groups

28. Meyer, Viscusi & Durbin (1995), AER • Research question: Does higher compensation induce workers to stay out of work longer? • Focus: work-related injury compensation • Compensation = Payment for medical care + Weekly cash payment (benefit) • Temporary total disability (TTD) : a person is not able to work, but expected to recover soon and return • Duration of out-of-work stay is determined by the employee and his/her physician • Higher compensation may lead to: • Decreasing incentives to avoid injury • Increase in benefit claims for non-work injuries • Expanding (overstaying) the time out of work • TTD claims have no fixed duration

29. Meyer, Viscusi & Durbin (1995), AER Increase in maximum weekly benefits for high-earning groups in Michigan and Kentucky

30. Meyer, Viscusi & Durbin (1995), AER Changes in weekly maximum benefits for high-earnings group Kentucky (1980): from $ 131 to $ 217 per week (66 % increase) Michigan (1982): from $ 181 to $ 307 per week (70 % increase) Assignment to treatment and control: Injury date Objective: comparison of claims duration between people injured a year before the increase in benefits and those injured a year after the increase Treatment group: high-earnings group injured before and after the change Control group: low-earnings group injured before and after the change Identification issues Correlation of previous earnings and disability compensation

31. Meyer, Viscusi & Durbin (1995), AER • Data: Detailed Claim Information dataset • Random sample of cash claims from the insurance companies • Available variables • Date of injury • Duration of TTD • Weekly benefit amount • Previous earnings • Type of injury • Socio-economic characteristics (age, gender, and marital status) • Earnings’ thresholds (average wages) • Kentucky: E1= $ 66; E2= $ 196; E3= $ 298 • Michigan: E1= $ 216; E2= $ 271; E3= $ 556

33. Claims’ duration elasticity with respect to weekly benefits Kentucky: 0.36 Michigan: 0.62

35. Lecture 1: Summary • Correlation DOESN’T necessarily imply causality • Three criteria for causality: association, time precession, and no alternatives • Identification strategy = Source of variation + Econometric technique • Random assignment eliminates selection bias • Experimental research design: • - researcher generates variation • - high internal validity at the expense of external validity • Natural (Quasi-) experiments: • - observational studies • - assignment “as if” random • - transparent exogenous source of variation • - low internal validity • Validity – quality of research conclusions: • Internal: Is the obtained relationship a causal one? • External: How generalizable is the relationship?

36. Next Lecture Topic: Causal Inferences and Observational Data • !Reading assignment for the next week • Waldman, M., Nicholson, S. & Adilov, N. (2006). Does Television Cause Autism? NBER Working Paper 12632. • 2. Bradford, D. (2003). Pregnancy and the demand for cigarettes, AER.vol. 93, No. 5, pp. 1752-1763.