Calibrated Peer Review TM

1. Calibrated Peer ReviewTM Orville L. Chapman, Arlene A. Russell, and Michael A. Fiore University of California, Los Angeles

2. CPR Design • CPR design is based on peer review of scientific proposals and manuscripts. • CPR permits only competent, calibrated peer review of student documents • CPR ensures careful reading for content. • CPR makes students conscious of style issues. • CPR enables learning by writing.

3. Motivation for Creating CPR • Science Literacy • Constructivist Learning • Critical Thinking

4. Science Literacy

5. Literacy • Discourse based on comprehension, understanding, organization, and logic.

6. Construction of Knowledge • Integration of new knowledge • Organization of knowledge • Critical evaluation of new information

7. Critical Thinking • Rigorous thinking • Testing ideas and data against existing intellectual models • Rejecting intellectual models • Creating new intellectual models

8. Bloom’s Taxonomy of the Cognitive Domain • Evaluation--highest level • Synthesis • Analysis • Application • Comprehension • Knowledge--lowest level CPR * * * * *

9. What Is CPR? • CPR is a networklearningtool. • CPR is accessed through an Internet browser. • CPR is platform independent. • CPR enables writing assignments in any class, regardless of size or subject. CPR works even with the largest classes. • CPR enables shared writing assignments among institutions over the Internet.

10. Two factors Make CPR Special: • CPR is discipline independent. Art, anthropology, English, social science, history, economics, business, law, chemistry, physics, life science, and mathematics are using it. CPR can serve any curriculum. • CPR is level independent.It serves grades 6-12, colleges and universities, graduate and professional schools.

11. What Does CPR Comprise? • Student interface • Instructor interface • Assignment authoring tool • Reporting tool • Assignment management system • Assignment libraries

12. How Does CPR Work? • The instructor gives the student a writing assignment. • The student submits his or her document to CPR electronically. • CPR then gives the student three calibration documents for review. • When the student has completed the calibration, CPR provides a detailed report on the student’s performance.

13. How Does CPR Work? • After the studenthassuccessfully completedthe calibration, CPR provides three peer documents for review. • CPR then gives the student his or her document for self-review. • Finally, CPR provides a detailed report of the peer review and the self-review.

14. The Review Process • The review process is the same for calibration, peer review, and self-review. • The student answers content questions for each document. • The student answers style questions for each document. • The student assigns a score to each document on a 1 to 10 basis.

15. Student Reports • summary of the calibration results

16. Instructor Reports • CPR provides a detailed report on each individual student’s work. • CPR delivers a detailed summary report for the class. • CPR reports are broken down among calibration, peer reviews, and self-review. • Students are scored on their writing and their reviewing.

17. Assignment Libraries • Molecular Science Library • CSUN Business School Library • Libraries enable frequent writing assignments with minimal instructor time.

18. Creating CPR Assignments • Identify important course topics. • Locate appropriate sources for student learning. • Compose appropriate guiding questions. • Write the calibration documents--one should be an exemplar. • Create carefully crafted content and style questions. • Have peers review your assignment.

19. Creating CPR Assignments: • Start by reading “Workshop Manual and Writing CPR Assignments,” by Arlene Russell. • To obtain a copy of the manual or to get advice on creating CPR assignments, contact Arlene Russell: russell@chem. ucla.edu.

20. Collaborative Library Building • Among institutions (NM CC’s; Other CC’s) • Within an institution (CSUN Business School) • Within a project (MSC consortium) • Within a school cluster (Narbonne-San Pedro School Cluster, ~24,000 students). • Publishing Companies • Consortia and individual partnerships

21. If You Create CPR Assignments, • You will change the way you teach. • You will sharpen the focus on what you want your students to learn. • You will become more effective. • You will be forced to ponder issues that you are currently ignoring. • You will be forced to define your assessment strategy first.

22. How We Use CPR • Create a scientific database. • Design an exploration of that database. • Give the students guiding questions that lead them to ask the questions that really matter. • Assign a writing task. • Create carefully thought out content and style questions. • Have colleagues criticize the content and edit the assignment.

23. A Useful Guide • John C. Bean, “Engaging Ideas: The Professor’s Guide to Integrating Writing, Critical Thinking, and Active Learning in the Classroom,” Jossey-Bass, San Francisco, 1996

24. What Does CPR Achieve? • Students learn science by writing. They construct meaning. • Students develop their writing skills. • Students learn to evaluate peer writing. • Students learn abstracting, reading for content, and self-evaluation. • Students think more deeply about important aspects of what they are studying.

25. Using CPR, Students Learn • Observation through description. • Comprehension and understanding through analysis and clear, precise expression. • Evaluation through critical assessment of peer work. • Synthesis and argument through organization and logical expression.

26. CPR Enables • Development of an instructor’s own writing assignments, at a cost of about 6-8 hours per assignment using the authoring tool. • Collaborative assignment development over the net. • Significant student writing with little instructor effort (about 5 minutes per assignment), using assignment libraries. • Cooperative distance-learning courses

27. An AP Chemistry Course, Arlene Russell • A one-year AP course, which bears credit, for rural high school students is being distributed as distance learning. • This project is funded by the Office of the President of the University of California. • This course is designed around use of learning objects from the Molecular Science Curriculum project and CPR.

28. Does CPR Work? • A study from CSUN Business School: • Two economics classes, each about 40 students, were selected for the study. • Both classes took the same examinations. • On each examination, two questions were taken from material not covered by CPR assignments.

29. Does CPR Work? • On each examination, two questions covered material that the test class had studied as CPR assignments and the control class had learned in lecture and discussion sections. • On the essay questions in which neither class had access to CPR assignments, the two classes were indistinguishable.

30. Does CPR Work? • On essay questions in which one class had used traditional methods and the other class used CPR, the CPR class scored 60% better than the traditional class. • These results were confirmed in a second test of the same design.

31. What Makes CPR Special? • CPR is discipline independent. • CPR is level independent. • CPR has been used by more than 10,000 students.

32. CPR Funding • The National Science Foundation through the initiative for reform in chemical education • The Howard Hughes Medical Institute, for applications in life science

33. If you have questions about CPR, Contact Orville L. Chapman, at chapman@chem.ucla.edu, or Arlene A. Russell, at russell@chem.ucla.edu. You may access the slides and text for this presentation at http://www.molsci.ucla.edu/chapman/cpr.htm.