1 / 46

Labrador: A Tool for Automated Grading Support in Multi-Section Courses

Labrador: A Tool for Automated Grading Support in Multi-Section Courses. Drexel University Programming Learning Experience (DUPLEX) http://duplex.mcs.drexel.edu Christopher D. Cera, Robert N. Lass, Bruce Char, Jeffrey L. Popyack, Nira Herrmann, Paul Zoski Drexel University

Download Presentation

Labrador: A Tool for Automated Grading Support in Multi-Section Courses

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Labrador: A Tool for Automated Grading Support in Multi-Section Courses Drexel University Programming Learning Experience (DUPLEX) http://duplex.mcs.drexel.edu Christopher D. Cera, Robert N. Lass, Bruce Char, Jeffrey L. Popyack, Nira Herrmann, Paul Zoski Drexel University Mathematics and Computer Science

  2. Roadmap • Introduction • Problems and Solution Goals • Labrador • Discussion

  3. The Group • Bruce Char • Professor, Computer Science • Nira Herrmann • Professor and Head, Mathematics • Jeffrey L. Popyack • Associate Professor, Computer Science • Paul Zoski • Instructor, Math and Computer Science • Christopher D. Cera • Computer Science Graduate Student • Robert N. Lass • Computer Science Undergraduate Student

  4. Who Am I • First TA in MCS to experiment with WebCT in December 2000 • Courses I have TA’ed using WebCT • 3 x Introductory Programming I, II • 1 x Object Oriented Programming • Migrated course content to WebCT from previous course website • HTML assignments and labs into question database • Gradebook maintainer • Wrote demo’s and documentation to train other TAs • Developer of software supplements to WebCT to support course administration

  5. The Course • Intro Programming I, II • Variety of Majors • Computer Science • Computer Engineering • Information Systems • Mathematics • Digital Media • Class • 1 x 1 hour lecture • 2 x 1 hour labs • People • 250-300 Students • 2-3 Professors • 10-12 Teaching Assistants

  6. The Duplex Project: An Overview • Take advantage of advances in Information Technology to improve instruction and reduce costs for computer programming courses • Modular Structure • Multiple Entry Points • Multiple Audiences • Multiple levels of knowledge (Bloom’s Taxonomy) • Computer Supported Cooperative Work (CSCW) in student labs • Online Services – Today’s Topic

  7. Roadmap • Introduction • Problems and Solution Goals • Labrador • Discussion

  8. Before WebCT • Hundreds of people involved in paper exchanges of handwritten assignments and quizzes • Testing programs required floppy exchanges • No Chat • No newsgroup-style threads • Feedback and grades are not online

  9. Before WebCT • Large Classes • Managing several hundred students is difficult due to the logistics involved with organizing paperwork and people.

  10. WebCT • Service from Information Resources & Technology (IRT) department (v3.1) • MCS started Dec. 2000 (v3.5) • Upgraded to v3.7 recently • Transition to v3.8 planned in Fall

  11. Noteworthy Features • General Course Website • Labs: Online Quizzes with Automated Grading • Homework: Online Assignments • Joint Staff--Student Chat • Discussion Threads

  12. Software Design Goals • Bulk Assignment Downloading (prior to 3.7) • Bulk Quiz Downloading • Section Sorting for bulk downloads, or only one section

  13. Software Design Goals • Post-Processing student files • Archive Extraction (tar, zip) • Decompress (gz, zip) • Decode (uue) • Minimal staff intervention when transferring submissions between systems, eg. Plagiarism Detection Systems (PDS) • Automatically collate source code • Generate electronic documents to facilitate grading and archiving • Remote Execution • downloading to computer x (on campus) while operating at computer y (off campus)

  14. The Bigger Picture • Course-Specific Tasks • Not all processing should be done on the server • Select files have to be transferred to a different system for further processing and analysis by staff • Client-side support is needed to perform this, preferably automated and not necessarily using a web browser.

  15. Roadmap • Introduction • Problems and Solution Goals • Labrador • Discussion

  16. Labrador: Our Solution • Client-side WebCT Supplement • Implemented in Perl • Works for users with TA and Designer access to WebCT

  17. Perl • Practical Extraction and Reporting Language • Created in 1987 to replicate Unix shell functionality • Powerful text manipulation • High-level: rapid prototyping and development • Large library of modules and active development community • Perl is used by many applications, including WebCT

  18. A Taste of Labrador Perl Programming

  19. Labrador works on Windows, Mac, Linux, Other Unix Variants, etc. • Perl has been ported to all major operating systems • Perl programs can be compiled: • No need for the end user to install Perl • Perl2exe for Windows support (www.perl2exe.com)

  20. User Interface • Different users / Different UI preferences • Command-line • Interactive • Configuration File • GUI (in development)

  21. Required Info http://webct.drexel.edu/SCRIPT/CS164_Fall2002/scripts/designer/dropbox_edit.pl?DROPBOX_ASSN_VIEW+_side_nav++1006285909 • Username/Password • Server URL webct.drexel.edu • Course ID CS164_Fall2001 • Submission Name or ID 1006285909 • Optional Username List

  22. Components JPlag Submission Downloader Section Sorting Post Processing PDF Generator Moss

  23. Bulk Downloader • Web-Crawler: simulates clicks of actual staff member • Downloads actual web content (HTML) • Parses HTML to find desired text or URLs to crawl to next • Uses HTTP operations • Works on assignments and quizzes • Only component in Labrador which interacts with WebCT

  24. Organizing Submissions by Section • Not supported in original Labrador prototype, added later on. • Organizes files into directories • 3 possible input methods: • Creating a Section column in gradebook • “Username, Section” CSV File • Username file

  25. PDF Generation for Electronic Mark-up • Adobe Portable Document Format can be viewed on all major platforms • With Adobe Acrobat, PDFs can be annotated by graders • Sony VAIO Slimtop PC (PCV-LX920) • Wacom Pen Tablet.

  26. VAIO / Wacom

  27. PDF Markup Example

  28. Redistribution • How can we return annotated PDF’s back to students using WebCT? • WebCT Mail • WebCT Groups • Version 3.8 now addresses this issue

  29. Heterogeneous Systems • Different systems want different formats • Plagiarism Detection Systems • Moss has been used most extensively • Began experimenting with JPlag more recently • Future work: Automatic program compiling and testing

  30. Automated Plagiarism Detection • Digital formats make "borrowing" easy • Browsing similar works needs a simple and quick user interface. • Careful review by faculty to assess results and present to students

  31. Moss • Processes C, C++, Java, ML, Lisp, Scheme, Pascal, and Ada programs. • Common code feature reduces false positives

  32. Moss Interface

  33. JPlag • Processes C, C++, Scheme, and Java programs • For plain text files, it matches a user specified number of words appearing in succession • Could be used for any course grading written (text) documents

  34. Command Line Invocation

  35. Interactive

  36. Quiz Download with Config File

  37. Roadmap • Introduction • Problems and Solution Goals • Labrador • Discussion

  38. Recent WebCT Enhancements • Relevant to this talk: • 3.7 • Addressed bulk download issue for assignments • 3.8 • Attaching documents to an assignment

  39. Future WebCT Enhancements • Power users will need functionality not yet supported • Every domain will also require additional functionality • Not feasible for all domain-specific functionality to run on the WebCT server

  40. Wanted • API for non-administrators • Stateful protocol so clients can be built by 3rd parties

  41. HTTP: Insufficient Protocol • Inconvenient content interchange • Heavy client interaction • A HTTP based approach will be sensitive to the exact location of web pages, and format of text within them.

  42. Labrador Availability • Contact Us • http://duplex.mcs.drexel.edu

  43. Project Support • The Pew Learning and Technology Program at the Center for Academic Transformation • National Science Foundation, Division of Undergraduate Education, DUE-#0089009 • The Ramsey-McCluskey Family Foundation, Margaret Ramsey, '84 • Drexel University

  44. References • [1] Alex Aiken. Moss: A system for detecting software plagiarism (unpublished), http://www.cs.berkeley.edu/~aiken/moss.html. • [2] Jeffrey L Popyack, Bruce Char, Paul Zoski, Nira Herrmann, and Christopher D. Cera. Managing course management systems. In Proceedings of the thirty-third SIGCSE technical symposium on Computer Science Education, Birds-of-a-Feather Sessions, page 423, 2002. • [3] L. Prechelt, G. Malpohl, and M. Philippsen. Jplag: Finding plagiarisms among a set of programs. Technical Report 2000-1, Fakultat fur Informatik, Universitat Karlsruhe, Germany, March 2000. • [4] Larry Wall, Tom Christiansen, and Jon Orwant. Programming with Perl. O’Reilly and Associates, 3rd edition, 2000. • [5] Michael J. Wise. Yap3: Improved detection of similarities in computer program and other texts. In Proceedings of the twenty-seventh SIGCSE technical symposium on Computer Science Education, pages 130–134. ACM Press, 1996.

  45. Roadmap • Introduction • Problems and Solution Goals • Labrador • Discussion • Bonus Slides

  46. Blooms Taxonomy • Knowledge • remembering of previously learned material; recall (facts or whole theories); bringing to mind. • Comprehension • grasping the meaning of material; interpreting (explaining or summarizing); predicting outcome and effects (estimating future trends). • Application • ability to use learned material in a new situation; apply rules, laws, methods, theories.   • Analysis • breaking down into parts; understanding organization, clarifying, concluding. • Synthesis • ability to put parts together to form a new whole; unique communication; set of abstract relations.  • Evaluation • Ability to judge value for purpose; base on criteria; support judgment with reason. (No guessing).

More Related