Enhancing Undergrad Computer Science with Automation: Challenges, Solutions, and Benefits

1. The Role of Automation in Undergraduate Computer Science Chris Wilcox Colorado State University 3/5/2015

2. Problem: Expanding Enrollments • “Bachelor’s degree production increased by a double digit percentage for the third straight year. The number of new undergraduate computing majors among U.S. computer science departments rose an astonishing 29.2 percent, 22.8 percent among those departments reporting both this year and last year. This is the fifth straight year of increased enrollment in computing majors by new students.”*1 1) Taulbee Survey 2013 Computer Research Association 2) CSU Enrollment Numbers for Introductory Course 2010-2014 The Role of Automation in Undergraduate Computer Science

3. Problem: Decreasing Funding • “Colorado has reduced its support for higher education by nearly 69.4 percent, from $10.52 in fiscal 1980 (and a peak of $13.85 in fiscal 1971) to $3.22 by fiscal 2011. At this rate of decline Colorado appropriations will reach zero in 2022, 11 years from now.”*1 1) American Council on Education “State Funding: A Race to the Bottom” 2) Center on Budget and Policy Priorities “A Formula for Decline…” The Role of Automation in Undergraduate Computer Science

4. Solution: Increased Automation • Automated Grading • Proprietary web-based framework for the automated submission and assessment of student programs. • Peer Instruction • Partial inverted classroom with peer instruction quizzes automated using iClicker technology. • Online Tutorials – MyProgrammingLab, Zyante • Bulletin Boards – Blackboard, Piazza • Grading Frameworks – Blackboard, Canvas • Web Portal – Proprietary The Role of Automation in Undergraduate Computer Science

5. Key Questions • Question: How do automated tools negatively or positively affect student learning? • Question: Do the benefits of automation, such as resource savings, outweigh the disadvantages? • Answer: Based on my research outcome, we can conserve resources (grade less, teach more) while maintaining or improving academic performance. The Role of Automation in Undergraduate Computer Science

6. Automated Grading • 250 students * (12 assignments + 12 labs + 6 quizzes) = around 7500 programs to grade during one semester! • Let’s make it very easy, maybe 6 minutes per program, so 6 * 7500 = 45000 minutes = 750 hours. • Who has the time for that? That’s more than two teaching assistants working 20 hours per week, which is all I have. • From an economic viewpoint, that’s $10-15K based on average graduate student compensation. • From a pedagogic viewpoint, shouldn’t teaching assistants and instructors maximize time spent in front of students? The Role of Automation in Undergraduate Computer Science

7. Software: Automated Grading The Role of Automation in Undergraduate Computer Science

8. Software: Test Results The Role of Automation in Undergraduate Computer Science

9. Student Feedback Cycle Workflow with Manual Grading Preliminary Feedback Final Feedback WorkflowwithAutomatedGrading 0 1 2 3 4 5 6 7 8 Post Date Due Date The Role of Automation in Undergraduate Computer Science

10. Software: Course Calendar The Role of Automation in Undergraduate Computer Science

11. Evaluation – Exam Scores The Role of Automation in Undergraduate Computer Science

12. Evaluation – Withdrawal Rates The Role of Automation in Undergraduate Computer Science

13. Evaluation – Program Submissions The Role of Automation in Undergraduate Computer Science

14. Evaluation – Online Tutorial The Role of Automation in Undergraduate Computer Science

15. Resource Savings • Each time we grade a student program with automation we save ~16 hours, and we give many assignments and programming quizzes. • Each time we grade a peer instruction quiz with automation, it saves us around ~6 hours of teaching resource, and we give one per week. • Equivalent to at least one TA each semester, or 300 hours at approximately $20 per hour, which is around $6000, not counting tuition, and we use automated grading in 5-6 courses, twice a year. The Role of Automation in Undergraduate Computer Science

16. Automation Caveats However, there are some negative aspects of automated program grading that we have observed: • Automated testing can be perceived as unforgiving and overly strict by students. • Clearly not as beneficial as one on one code reviews for design and coding style issues. • Can create an unhealthy dependence on preliminary testing, which limits the acquisition of testing and debugging skills by the student. The Role of Automation in Undergraduate Computer Science

17. Threats to Validity • Inherent in empirical studies performed on students in a university setting. • External Validity: Automation does not degrade academic performance, and can actually improve it. • Internal Validity: Are there extraneous factors that could account for observed differences? • Semester to semester variation in students. • Staff turnover among teaching assistants. • Changes to course format and content. • However, we were able to tightly control the critical factors: curriculum, assignments, and exams. The Role of Automation in Undergraduate Computer Science

18. Conclusions • The automation of key processes can save a significant amount of scarce resources without negatively impacting academic performance. • The benefits of automation are both tangible, such as higher exam scores, and intangible, such as increased student engagement and interest. • We view this as a fortunate outcome, since expanding enrollments will inevitably provide incentive for even more automation. The Role of Automation in Undergraduate Computer Science