Technology In and Out of the Classroom: The Effect of Online Learning on Student Performance

1. Technology In and Out of the Classroom: The Effect of Online Learning on Student Performance Jodi N. Beggs Economists Do It With Models 7th Annual Economics Teaching Conference October 27, 2011 www.economistsdoitwithmodels.com

2. Motivation • Personal bias  • Increasing fiscal constraints at both the university and K-12 levels • Increased technology availability and cost effectiveness • Adoption rate of online learning is very high • 2.6 million students took at least one online course in fall 2005 • 80 percent of doctoral/research institutions in the US offer online courses • Virtually every institution with over 15,000 students offers online courses • Online courses also increasing prevalent at the K-12 level

3. Motivation • Personal bias  • Increasing fiscal constraints at both the university and K-12 levels • Increased technology availability and cost effectiveness • Adoption rate of online learning is very high • 2.6 million students took at least one online course in fall 2005 • 80 percent of doctoral/research institutions in the US offer online courses • Virtually every institution with over 15,000 students offers online courses • Online courses also increasing prevalent at the K-12 level Online instruction represents a potential opportunity for both cost savings and educational enhancement, but the effect on student understanding needs to be understood before new modes of instruction are widely adopted

4. Brown and Liedholm 2002 • Data from Principles of Microeconomics at Michigan State University • 3 modes of instruction: live, hybrid, virtual • Same textbook, multiple-choice examinations, course web sites and email • Live: 3 class hours per week • Hybrid: 2 class hours per week plus online materials • Required “Problems in Microeconomics” • Virtual: Videos of live lectures plus hybrid materials

5. The Data (Means)

6. Score Regressions By Course Type

7. Score Predictions By Instruction Type

8. Discussion • Differences in performance across modes of instruction increase as questions get higher level • Differences could be at least partially attributable to differences in student effort • Statistics on time spent on course work indicate that students in live sections put in more effort • Gender differences largely go away in hybrid and virtual environments • Could reflect differences in effort across genders • Race differences also go away to some degree • Selection bias is present since students are selecting into different forms of the course

9. Discussion • Differences in performance across modes of instruction increase as questions get higher level • Differences could be at least partially attributable to differences in student effort • Statistics on time spent on course work indicate that students in live sections put in more effort • Gender differences largely go away in hybrid and virtual environments • Could reflect differences in effort across genders • Race differences also go away to some degree • Selection bias is present since students are selecting into different forms of the course Selection bias can only be eliminated via a randomized experiment…

10. Figlio, Rush and Yin 2010 • Large introductory course at a major research university • Randomly assigned to live or internet lectures • Students were given an incentive to participate in the experiment • Volunteers not markedly different from non-volunteers • All other aspects of the course held constant • Multiple-choice exams • Live lecture condition most similar to the hybrid condition introduced earlier

11. Scores By Method of Instruction

12. Scores By Method of Instruction None of the differences are statistically significant!

13. Differences Matter for Some Demographics

14. Differences Matter for Some Demographics Could again reflect differences in effort

15. Caveats • Live/online contamination • Attendance data suggests that some live students were watching lectures online • However, live participants had better attendance than live non-participants • Potential peer effects • Representativeness of study volunteers • Familiarity of students with online platforms may affect the live/online differential • Results may not generalize to non-economics material

16. Caveats • Live/online contamination • Attendance data suggests that some live students were watching lectures online • However, live participants had better attendance than live non-participants • Potential peer effects • Representativeness of study volunteers • Familiarity of students with online platforms may affect the live/online differential • Results may not generalize to non-economics material Do other studies support these findings?

17. Dept. of Education Meta Analysis • Literature search from 1996 through July 2008 • Studies screened for several attributes • Contrasted an online to a face-to-face condition • Measured student learning outcomes • Used a rigorous research design • Provided adequate information to calculate an effect size • 51 independent effects identified • 7 at the K-12 level, the rest with older students • Various subjects examined, the most popular being medicine or health care

18. Findings • On average, students in online learning conditions performed better than those receiving face-to-face instruction • 11 individual effects significantly favored an online or hybrid condition • 2 individual effects significantly favored face-to-face condition • Differences were largest when pure face-to-face instruction contrasted with a hybrid of online and face to face • Elements such as video or online quizzes do not appear to influence the amount that students learn in online courses

19. Discussion • Because online conditions usually included additional learning time and instructional elements, it’s not clear that the online medium is driving results • This is supported by the observation that studies in which online learners spent more time on the course than their face-to-face counterparts saw greater benefits from online instruction

20. Key Success Factors for Technology Integration • Leverage top instructors • Winner-take-all market • Use technology to supplement and economize, not purely to replace • Enable efficient division of labor • Provide incentives/commitment devices/etc. to dissuade procrastination and lack of engagement

21. For Further Reading… David N. Figlio, Mark Rush, and Lu Yin, “Is It Live or Is It Internet? Experimental Estimates of the Effects of Online Instruction on Student Learning.” National Bureau of Economic Research Working Paper 16089. Byron W. Brown and Carl E. Liedholm, “Can Web Courses Replace the Classroom in Principles of Microeconomics?” American Economic Review, May 2002 (Papers and Proceedings), 92(2), pp. 444-448. U.S. Department of Education, Office of Planning, Evaluation, an Policy Development, Evaluation of Evidence-Based Practices in Online Learning: A Meta-Analysis and Review of Online Learning Studies, Washington, D.C., 2009.

22. For more information, go to http://www.economistsdoitwithmodels.com/Carlson