1 / 12

Autolab : An Online Autograding Service for the World

Autolab : An Online Autograding Service for the World. David O’Hallaron Prof of ECE and CS, Carnegie Mellon University http:// autolab.cs.cmu.edu. Cool Labs -> Great Course. 15-213: Intro to Computer Systems (Bryant and O’Hallaron). Autolab.

amiel
Download Presentation

Autolab : An Online Autograding Service for the World

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Autolab: An Online Autograding Service for the World David O’Hallaron Prof of ECE and CS, Carnegie Mellon University http://autolab.cs.cmu.edu

  2. Cool Labs->Great Course 15-213: Intro to Computer Systems (Bryant and O’Hallaron)

  3. Autolab • An online autograding service that allows instructors to offer programming assignments over the Internet. • Two key ideas: autograding and scoreboards. • Autograding • Programs evaluating the quality of other programs. • Student handins automatically autograded on secure VMs. • Scoreboard • Scores are posted in real-time on sorted class scoreboard. • Students anonymize themselves with nicknames. • “kill -9 15213”, “213 makes me ANSI”! • Other services • Gradebook, handin/handout, late penalties, grace days, rosters, manual grading, code annotation, gradebook, cheat checking, bulk emails.

  4. Autolab system VM domain (CMU) VM Pools Browser … rhel handin scoreboard … rpc/ssh rhel122 … rpc rhel411 cmu Web server Tango ec2 http VM domain (EC2) VM Pools DB rpc/ssh … rhelPKU … ubuntu

  5. Autograders • No language restrictions • Autolab interface is last output line (autoresult string) Quality assessment (any form) Autograder (any language) Student’s work (any form) stdout Autoresult string (JSON)

  6. Observation: Autograding coupled with a scoreboard has a profound impact: • Fosters a sense of fun and community. • Gives students multiple chances to learn and improve. • Creates healthy competition when coupled with absolute thresholds.

  7. Fall, 2012 • 2,000 students in 10 courses worldwide • 15-213/18-213: Intro to Comp Systems [451 students] • 15-213: Intro to Comp Systems (Peking Univ) [219] • 15-110: Principles of Computing [368] • 15-112: Fundamentals of Programming [310] • 15-122: Principles of Imperative Computing [349] • 15-237: Mobile Web Apps [120] • 15-440: Distributed Systems [107] • 15-440w: Distributed Systems (Qatar) [10] • 15-740/18-740: Comp Arch [55]

  8. Past and future • Since Fall, 2010: • 29 offerings of 12 different courses in three countries • 6,300 total users • Over 5M page views • 160K autograded jobs • Future: • 112 and 213 self-study (with Kosbie) • Other schools using Bryant/O’Hallaron CS:APP text • New autograding service for OLI

  9. Proposal: Provide Autolab as a global autograding service to the world’s universities • Instructors: • Receive free ec2 instance(s) for testing/autograding. • Build a course Web page by selecting from a library of labs. • Evaluate and manage student’s work. • Students: • Pay a small lab fee (covers royalties and ec2 instances) • Download, develop, test, evaluate, and upload labs • Lab developers: • Submit their cool new labs to the library for vetting. • Receive credit when lab is adopted by an instructor.

  10. Potential Impact on Education • Improve the quality of education • Teachers helping other teachers instead of struggling alone. • Create a new publishing industry for students and academics • Developing a new lab is much less effort and risk than writing a book (Trust me!) • Foster a reputation-based community for teachers: • Kosbie: “My sorting lab was used by 100 schools last year.” • “I’m using Ganger’s awesome file system lab.” • “My lab builds onEckhardt’s groundbreaking OS lab.” • “Platzer has developed one of the most widely used compiler labs in the world.”

  11. Credits • Autolab was developed by Dave O’Hallaron and a team of talented CMU undergraduates: • Hunter Pitelka (ex-lead, now Palantir) • Kelly Rivers (ex-lead, now CMU HCI PhD) • Tom Abraham (lead), AbhayBuch, Steven Fackler, Dylan Swen, Peter Xiao, Ashley Kumar • Prof David Kosbie was a key early adopter. • Funding provided in part by Vanguard, CMU OLI, and Intel • For more info: http://autolab.cs.cmu.edu

More Related