My Reviewers Report Joe Moxley, Terry Beavers & Mike Shuman USF/CGS 455 | 1/23/12

1. My Reviewers ReportJoe Moxley, Terry Beavers & Mike ShumanUSF/CGS 455 | 1/23/12

2. Mid-Year 2: My Reviewers Development • Compatibility with USF infrastructure: http://my.usf.edu (SSO/USF NET ID) • Prepared Reports for SACS as requested by the Office of Institutional Effectiveness; Visited General Ed Council; Met with WRIN instructors • Reported on My Reviewers to WPA Directors in FL (UNF expressed interest in using) • Successful 2011 “Pilot” with FYC • Spring 2011: 124 Composition Sections; ~10,000 essays uploaded • Fall 2011: 124 Composition Sections; 15,999 essays uploaded • Spring 2012 • Pilot: WRIN 12 classes (World Lit; Expos Wrtg; Prof Wrtg, etc.) • 75 Composition Sections

3. Example of Usage: 15,999(not counting peer reviews)

4. National Recognition NCTE: “The University of South Florida’s “First-Year Writing Program” has an innovative My Reviewers tool that allows for faculty and students to see feedback across time.”

5. New PublicationsResearch byGraduate Students! Completed Research • Vieregge, Quentin, Kyle Stedman, Taylor Mitchell, and Joe Moxley. Agency in the Age of Peer Production. Urbana, IL: National Council of Teachers of English, in press. SWR (Studies in Writing and Rhetoric) Series • Moxley, Joseph M. “Objectivity 2.0: Teacher Response and Assessment of Writing Programs.” [Submitted] • McIntyre, Megan, Karen Langbehn and Joseph Moxley. “Using Real-Time Formative Assessments to Close the Assessment Loop.” McKee, Heidi and Danielle Nicole DeVoss, eds. Digital Writing Assessment. [Edited book under development] Research in development • Branham, Cassandra. Peer Review Models and My Reviewers (MA Thesis) • A study that compares 3 curriculum approaches (traditional, online, and collaborative) • Moxley, Joseph M and Terry Beavers. “Writing Development, a Longitudinal Study.” Implications • Attract competitive new graduate students who are interested in assessment research, social pedagogy, interface design

6. New Features! • Instructors/Administrators can change semesters to view past results • Instructors can • Emulate students • View learning analytics • Administrators can • Emulate teachers/students

7. Admin View: Peer Review Grade Center

8. Teacher View: Making Students Accountable for Peer Reviews

9. Learning Analytics • Improve retention • Make evidence-based curriculum changes • Enhance student success

10. Learning Analytics Instructors can • Track overall class progress/learning from project to project • Track rubric criteria from project to project (raw scores/weighted scores) • Compare their classes with the mean of the other 500 sections Administrators can • Emulate teachers/students • Track class progress/learning from project to project • Track criteria from project to project • Compare instructors’ grading patterns/timeliness

11. My Reviewers Enables Objectivity 2.0 • *Statistically significant at 0.05 level • **Total N excludes a student with missing data .

12. (Admin View) Crowd-Sourcing Assessment: Instructors/Administrators Can See Where They Stand in the Crowd

13. Demonstrate Undergraduate Learning

14. Evidence-Based Curriculum Changes (Project 2: Spring 2010 vs Fall 2011

15. Compare Alternative CurriculumsFall 2010: 1101 vs 1102

16. Compare Alternative Curriculums1101: Spring 2010 vs Fall 2010

17. Develop New Models for Teaching/Learning

18. Develop New Models for Teaching/Learning

19. What’s Next: Current Funding Cycle • Add Library of Comments that can be embedded into students’ texts and that hyperlink out to OER resources (http://writingcommons.org) • Create batch upload/download to Safe Assignment • Add Writing Center Template

20. What’s Next? (for FYC)(Dependent on Additional Funding) • Allow users to add/customize rubrics (to handle other USF assessments, such as historical processes, for other courses) • Add audio comments • Allow multiple formats for text upload conversion to pdf • Allow for upload of multiple formats (podcasts/video, etc.) for grading and peer review

21. What’s Next? (for FYC)(Dependent on Additional Funding) • Improve Library of Comments (enable students and instructors to archive their own comments) • Enable tool to learn from student error patterns and connect related lessons/quizzes • Improve OER resources (videos/comments/texts • Improve Help Center • Flesch Reading reports

22. What’s Next? (for FYC) (Dependent on Additional Funding) • Learning Analytics • Track progress from course to course (e.g., look at style across multiple courses) • Enable downloadable Excel reports • Allow students to track progress from project to project/course to course (game theory/students learn more when in control of their process)

23. What’s Next? (for FYC) (Dependent on Additional Funding) • Continue our effort to revolutionize peer review • Add workflow to enable greater flexibility; e.g., a teacher filtering peer-review inside the tool/distributed assessment • Develop social helpfulness algorithm • Add Badges • Develop “My Reviewers in a Box” so it can be sold by USF to other universities & so USF can recoup its investment