David Figlio , Northwestern U Mark Rush, U Florida Lu Yin, American Institutes for Research

1. Is it Live or is it Internet? Experimental Estimates of the Effects of Online Instruction on Student Learning David Figlio, Northwestern U Mark Rush, U Florida Lu Yin, American Institutes for Research

2. Background • Two major trends affecting higher education • Declining state and local appropriations  more fiscal challenges, increasing demands for “efficiency” (new NRC panel) • Rapid improvements in technology • As a consequence, millions of students are now taking classes online

3. What do online classes look like? • Two major types: • Innovative, highly interactive classes aimed at exploiting special nature of the internet and modern technology • Traditional lectures presented in an online format • While door #1 is the type of class advocated by learning scientists (NB: I’m working with colleagues to develop this type of class in high school science), door #2 is what most universities are doing.

4. What is the causal question of interest? • Do students learn more in a traditional lecture when it is broadcast on the internet than when it is presented in a standard lecture hall? • In principle, the results could be positive or negative: • Pro: increased flexibility; no need to rely on others’ notes; ability to annotate/rewatch lectures • Con: increased barriers to student-faculty interaction; reduced ability to ask just in time questions; incentives to defer work

5. What is the causal question of interest? (part 2) • Are there heterogeneous effects of internet-based traditional lectures on students? • Why interesting? • Some groups may face additional communication barriers (e.g., language minority students) • Some groups may have lower self-regulation skills (e.g., college-aged men; relatively lower-achievers)

6. What is the evidence to date? • Almost nothing • Mainly small-scale case studies; unsurprisingly, almost all studies are based on type 1 of internet-based class rather than type 2 of internet-based class • Somewhat larger studies have poor treatment-control contrast • Considerable need for experimental evidence on both types of internet class, but especially the broadcast-traditional-lecture type  this study

7. What is the ideal experiment? • Treatment and control students should be taught in tandem • Same instructor, same exams, same lectures, same supplementary material, same readings • Pure randomization to live vs. internet treatment • No opt-out from experiment • No opportunities for contamination of treatment and control • Live students cannot view internet lectures; internet students cannot attend live lectures • Ample opportunities for detecting heterogeneous effects and improving external validity • Dozens of experiments in different courses at different institutions, with within-course randomization • Cluster-randomized design; clustering on subgroups to ensure sufficient sample size to detect subgroup-specific effects

8. Fidelity to ideal experiment • Treatment and control students should be taught in tandem • Same instructor, same exams, same lectures, same supplementary material, same readings • Pure randomization to live vs. internet treatment • No opt-out from experiment (instead: randomization of volunteers) • No opportunities for contamination of treatment and control • Live students cannot view internet lectures; internet students cannot attend live lectures (live students may potentially watch with friends) • Ample opportunities for detecting heterogeneous effects and improving external validity (nope: just one class, no group cluster) • Dozens of experiments in different courses at different institutions, with within-course randomization • Cluster-randomized design; clustering on subgroups to ensure sufficient sample size to detect subgroup-specific effects

9. Threats to internal and external validity • (1) Volunteers, rather than pure randomization • How representative are volunteers of the potential study population at the institution? [external validity]  Table 1 • Knowledge that this is an experiment might lead to differential attrition of live vs. internet [internal validity]  Table 2 contrasts; bounding exercise in Table 3

10. Threats tointernal and external validity • (2) Fidelity of randomization and lack of treatment contamination • Do people drop from experiment post-randomization? [internal validity]  as 15 students assigned to “live” dropped from the experiment, Table 3 compares results treating defectors as “live” versus dropping from study • Do “live” students view internet version and do “internet” students attend live lecture? [internal validity] • Door guards checked IDs so we know that no “internet” students attended live lectures • Live students did not have online access, but could have accessed via friends’ log-ins  Figure 1 shows that “live” students attended substantially more lectures than “live+internet” non-volunteers

11. Threats tointernal and external validity • Intro economics might be special • The university in question might be unusual • The particular instructor may translate well/poorly to the internet platform • Lots of hand-wringing • And also a call for more experiments in other subjects and settings and with a design aimed at detecting heterogeneous treatment effects

12. Threats to internal and external validity • (1) Volunteers, rather than pure randomization • How representative are volunteers of the potential study population at the institution? [external validity]  some differences; volunteers had higher GPAs and lower SAT scores and mom was less likely to be a college grad. • Knowledge that this is an experiment might lead to differential attrition of live vs. internet [internal validity]  no evidence of differential attrition: 6 live, 10 online attriters, and no differences between them. Bounding exercise (giving attriters scores of 0 or 100 on missed exam) shows tight bounds when considering attrition.

13. Threats tointernal and external validity • (2) Fidelity of randomization and lack of treatment contamination • Do people drop from experiment post-randomization? [internal validity]  no differences in results when we treat defecting volunteers as “live” versus when we drop them from the study • Do “live” students view internet version and do “internet” students attend live lecture? [internal validity] can’t know for certain about live students viewing lectures online, but it looks like live students definitely attend more live lectures than those with the choice. Note that there is a “professional” private note-taking and tutoring service that is very popular and available to all students regardless of live/online. Other studies at the institution on cramming indicate that many internet course-takers don’t view all (or most, or sometimes any) of the lectures either

14. Threats tointernal and external validity • Intro economics might be special • The university in question might be unusual • The particular instructor may translate well/poorly to the internet platform • This is important, and there’s nothing we can do about this except to call for more experiments

15. Results • Small insignificant positive estimated effects of live-only vs. internet-only instruction; statistically significant (mostly due to larger coefficients rather than smaller standard errors) when conditioning on covariates • Positive estimated effects are largest for Hispanic students (sig) and Asian students (not sig); male students (not sig) and students with relatively low SAT scores (not sig)  larger sample sizes and cluster randomization might have helped detect differences here

16. Conclusions • There might be efficiencies to exploit with internet-based traditional lectures, but there is no free lunch • The results of this experiment should be interpreted as a first piece of evidence; responsible universities should be slow to implement this policy change despite the momentum and push for this. We need many more – including larger -- experiments!