Expertiza: Improving Course Materials and Learning Outcomes through Peer Review of Student Work

Expertiza: Improving Course Materials and Learning Outcomes through Peer Review of Student Work Edward F. Gehringer Dept. of Computer Science North Carolina State University Supported by NSF DUE under a CCLI grant NCSU Center for Teaching and LearningNCSU LITRE (Learning in a Technology-Rich Env.) Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu

Introduction Expertiza Rationale Demo Experiment 1: Improving a Textbook Experiment 2: Review Wiki Submissions Outline Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu

Electronic peer review is students reviewing other students’ work over the Web. Building resources through electronic peer review gets students working together to improve others’ learning experiences, helps them learn, by performing tasks that are similar to real-world responsibilities, gives them experience in writing their ideas up for an audience of their peers, allows each cohort to “stand on the shoulders” of students in earlier classes. Introduction Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu

For students to demonstrate mastery of the subject. Every student does the same thing—redundant effort. Work is graded and thrown away, never benefiting anyone but the student who did it. Now the best work can be published, to help others learn. Homework, traditionally … Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu

Introduction Expertiza Rationale Demo Experiment 1: Improving a Textbook Experiment 2: Review Wiki Submissions Conclusion Outline Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu

Improve student learning Improve teaching Better utilize resources The Rationale Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu

The Expertiza platform improves student learning in these ways: Integrates active and cooperative learning Active learning allows students to take responsibility for their own learning. Extends active learning to out-of-classroom activities and distance education DE has been a roadblock to the use of active learning … students viewing lectures remotely can work only by themselves. Discourages plagiarism Multiple deadlines and milestones make it impossible to submit a finished product obtained from an external source. Improving student learning Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu

The Expertiza platform improves teaching in these ways: Increases the supply of examples/homework problems/test questions Students are assigned to make up such examples/questions, and these are peer reviewed. Focuses students on explaining/understanding the concepts that are hardest to master “Write an example to clarify the hardest concept in lecture k.” Improving teaching Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu

Some work is peer-graded, so teaching assistants can spend more time working with students and less time grading. Having inadequate TA support no longer limits the amount and kinds (e.g., design problems) of homework that can be assigned. Students rely more on their peers for help, less on the course staff. Makes teaching large classes an advantage! Large classes can produce more and better resources. Improving resource utilization Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu

Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu

Our class Master’s level course CS & ECE Substantial DE enrollment In Fall 2005, we used a new object-oriented design text for the first time, Dale Skrien’s, An Intrduction to Object-Oriented Design and Design Patterns Using Java Improving an OOD Text Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu

Improve an explanation of a topic covered in the text. Create a new example of a concept covered in the text. Write a new exercise for a chapter in the text. All students did not do these exercises in the same order. Three Homework Assignments Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu

Classroom discussion of the most difficult topics Students select a topic from a list. Several students are allowed to select the same topic, But the number of slots is limited. The Review Process: How It Begins Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu

Signing up for a First-Round Topic Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu

Student Choices Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu

Students review a submission based on the following questions (for examples). Does the example fully illustrate the concept being explained? Is the example easy to understand, i.e., as clear as it could be and still illustrate the concept? Does the example model the real world, i.e., could it be implemented in practice? Is the example code elegant? Does the example use up-to-date Java code or UML? Does the text that accompanies the example explain it well? Is the example more useful than the examples in this section of the book? Does the example seem to be original? Other: How would you rate the submission on factors not reflected in the scores on other questions? The Review Rubric Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu

After the Initial Review • Resubmission phase. 2–7 days to revise work in response to reviewer comments. • Grading phase. 3–7 days to make final comments and assign scores. • Review of review phase. Students review each other’s reviews. Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu

Results • 17 student submissions selected for text! Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu

A wiki is, essentially, a Web site that can be edited by any user. Homework done on wikis promotes collaboration between students. Problem: How to assess so much writing. Solution: Peer review. Teaching with Wikis Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu

Students select from a set of topics for a weekly wiki assignment. The students are given class time to work on the topic in groups. Then they take a couple of days to finish up their submission. Their submission is peer-reviewed by other students. They have a chance to revise their submission. Students rate the contributions of their partners. Instructor considers the reviews, revisions, and partner evals in assigning grades. Our Approach Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu

Wiki seems to be preferred as a submission mechanism. “The wiki was really fantastic for creating pages because it guarantees a uniform style for everything, putting the major focus on the content created, rather than formatting issues. It also helped format content so that it was more understandable.” Contributions are more extensive than we have seen when files are submitted and resubmitted. Peer review is the only scalable solution. Expertiza gives a framework for review. Students Seem to Benefit Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu

Lecture annotations Code reviews: Contributions to OSS projects Class “proceedings” in a research course Survey papers or research papers FAQ in a service-learning course Other Uses Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu

“Peer-assisted” review Total Quality Management Automated reviews of reviews Future Work Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu

Students can do good work— work that will help their classmates learn the material work that will help the instructor improve the class work that allows each class to “stand on the shoulders” of earlier classes. The Expertiza approach enables faculty to introduce these exercises into their courses. Conclusion Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu

http://research.csc.ncsu.edu/efg/expertiza Ed Gehringer, efg@ncsu.edu For more info … Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu

Expertiza: Improving Course Materials and Learning Outcomes through Peer Review of Student Work