Scoring Options for Essays and Other Assignments

1. Scoring Options for Essays and Other Assignments

2. Scoring Options for Essays and Other Assignments Instructors typically evaluate assignments in terms of the number of points achieved. The number of points in turn is based on teacher expectation that certain key points or criteria be included in the assignment. In theory, if a student includes all criteria in an assignment, he/she achieves the maximum point total.

3. What is a Rubric? • Rubrics are essentially guides for scoring student assignments. • If instructors are uncomfortable with the word rubric, terms such as grading criteria or scoring guide may be preferred. • Rubrics exist in many different formats, from the simplest (checklist) to more complex (descriptive or holistic rubrics). • Why should instructors consider developing and using rubrics?

4. Making a Case for Scoring Rubrics • Rubric development can be time consuming. Teachers who are not familiar with using scoring rubrics may need to be convinced that there is a return on their investment of time and energy.

5. Benefits of Rubrics • The following are some of the benefits of rubrics as described in Walvoord & Anderson (1998). • Make grading more consistent and fair. • Save time in the grading process; once the initial work is completed, instructors can move through student work relatively quickly. • Diagnose student strengths and weaknesses in detail in order to teach more effectively. • Track changes in student performance over several semesters and see how teaching changes influence student performance. • Establish consistency among colleagues for common/standard examinations and sequenced courses.

6. And More Benefits of Rubrics… • Suskie (2004) described several benefits of using rubrics that focus more on the interaction between teachers and their students. • Help students understand instructor expectations. • Inspire better student performance. • Improve communication between instructors and students. • Reduce arguments between instructors and students.

7. The Checklist • Scoring options for assignments can include a simple checklist that serves as a scoring guide. • For example, there may be six identified criteria or key points that must be in included and an instructor decides that each criterion is worth 1 point. • The assignment is therefore worth a total of 6 points. • As an instructor reads through the individual assignment, he/she checks off each criterion as it is covered and then totals the points achieved at the end.

8. Benefits of Checklists • Checklists can increaseobjectivity in grading and thus reduce subjectivity. • They can also increase the efficiency of grading for an instructor and make the grading process more consistent and fair to all students. • Figure 1 provides an example of a simple checklist.

9. Figure 1 – A Checklist for an Automotive Technology Curriculum • Adjust valves on engines with mechanical or hydraulic lifters. • Remove and replace timing belt. • Verify correct camshaft timing. • Remove and replace thermostat. • Inspect and test mechanical/electrical fans, fan clutch, fan shroud/ducting, air dams, and fan control devices; perform necessary action.

10. A problem with simple checklists… …is that they don’t allow for objective evaluation of the degree to which students describe or achieve the criteria. Should a student who merely mentions a criterion receive the same credit as a student who elaborates on it? A scoring rubric can be a solution for this issue.

11. Scoring Rubrics • Like scoring checklists, scoring rubrics identify the criteria that are to be included in an assignment. • A rubric establishes the “rules” for a student assignment (MSCHE, 2003) and goes further than a check list in that it also includes a rating scale to define levels of mastery related to each criterion. • To do this, an instructor would describe what full credit for a particular criterion would look like. The next step would be to describe what partial credit would look like and finally, what no credit would look like. This would yield a three-point rating scale. • Rubrics can include more than three levels of mastery.

12. Where to Start Creating a rubric begins with stating the criteria to be included in the assignment. Lloyd-Jones (1977) described a method called Primary Trait Analysis(PTA) as a careful and thorough method for stating criteria and establishing standards for rubric criteria.

13. What’s good about PTA? PTA moves an instructor from grading based on unstated criteria (gut instinct) to highly explicit criteria. The steps to be taken in PTA are as follows: • Identify factors or traits that will count for the scoring (for example: thesis statement, use of color, eye contact, etc.; Walvoord & Anderson (1998) recommend that these traits be described as nouns or noun statements); • Build a scale for scoring the student’s performance on each trait; and • Evaluate student performance against those criteria. More detailed recommendations for these three steps follow.

14. Step One – Identify Factors or Traits One of the best places to begin the process of identifying traits to be included in the rubric is in the review of past student work. • Look at student assignments across a range of quality and determine what made a paper a good paper and what made a paper a bad one. • Specifically determine the characteristics of an A paper, a B paper, a C paper, and so on. • You are likely to see traits begin to develop during this process. If you don’t have past assignments available, perhaps you could envision what you would consider an ideal assignment.

15. Step One – Identify Factors or Traits (continued) Walvoord and Anderson (1998) also recommend collaborating with an instructor in a different discipline from yours. • Get the other instructor to review a few pieces of student work and evaluate them. • You can then describe why you do or do not agree with him/her, but must explain in detail. • Your traits should begin to emerge from the detail explained in this process.

16. An example of two student responses to a short answer question in a biology class: Describe a feedback loop and list its components. Student A: A feedback loop is a cycle of events in which the status of a body condition is monitored, evaluated, adjusted and then re-monitored, re-evaluated, and re-adjusted on an ongoing basis. The primary components are the receptors, a control center, and the effectors. Student B: A feedback loop is equilibrium in the body. The components are glucose, blood pressure, and temperature. An instructor can review and compare these two answers, the first one being an A answer and the second being an F answer.

17. The traits that emerge from the review of the sample answers are: • The feedback loop as a cycle of events • The events of the cycle include monitoring, evaluating and adjusting, over and over. • The primary components are receptors, control center and effectors. With three traits identified, next comes the task of constructing a scale to evaluate each of the traits.

18. Step Two – Construct a Scale to Evaluate Each Trait The process of identifying traits is likely to also lead to some of the performance levels that will be used in your PTA. Each instructor decides how many levels will exist on the scale. • Walvoord & Anderson (1998) describe two, three, four and five level scales. • MSCHE (2003) states that with numerical scales a neutral midpoint should be used, which in turn would require an odd number of levels. However, MSCHE further describes that an even number would force a “non-neutral response” (2003). • One can see that the number of level scales can be chosen to suit one’s desired outcomes.

19. Step Two – Construct a Scale to Evaluate Each Trait (continued) To come up with the language to use for each point in the scale, again, past assignments can provide a wealth of information. What elements did A papers have for a given trait? Write them down – you may have the word contains preceding those elements. What elements were missing from the lower-grade papers? Write them down – you may have the word lacks preceding each of those elements.

20. Step Two – Construct a Scale to Evaluate Each Trait (continued) • Using the previous example, Student A correctly identified feedback loops as a cycle of events. That becomes the scale point for full-credit related to that particular trait. • Student B incorrectly identified the trait and that answer forms the basis for the zero-credit box on the rubric. • If using a 3-point scale, the instructor will identify a middle area between the two: “feedback loops are described with partial accuracy.” See Table 1 for an example.

21. Table 1 – Rating Scale for Feedback Loop Description

22. The second trait… • Student A correctly identified the events of the feedback cycle as monitoring, evaluating, and adjusting while Student B failed to mention the events at all. • Again – these become the end points of the evaluating scale and a midpoint is described. • Table 2 shows this information in table form.

23. Table 2 – Rating Scale for Events of the Feedback Loop Cycle

24. The last trait… • The last trait refers to the primary components of a feedback loop. • Student A correctly identified those components while Student B attempted to, but did so inaccurately. • Table 3 represents that part of the rubric scale.

25. Table 3 - Rating Scale for Primary Components of the Feedback Loop

26. Putting it all together… At this point the rubric can be assembled into one table and might look like Table 4.

27. Table 4 – Scoring Rubric for Describe a feedback loop and list its components

28. Quantifying the written answer… • Based on the scoring rubric just created, the short answer questions are worth a maximum of 6 points (3 traits worth maximum of 2 points each). • Student scores would be expected range from 0 to 6.

29. Step Three – Evaluate student performance against the established criteria • The next step would be to try using the scoring rubric to grade student assignments. You might make changes to the scales and traits based on actual application of the tool. • Walvoord & Anderson (1998) indicate that not only do PTA scales tend to be revised, but they should be. Specifically, the thinking that goes into such changes can lead to changes in teaching and ultimately student learning.

30. Practice Print out slide 27 so that you have the rubric in hand. Next, look at the following student response to the short answer question and grade it according to the rubric (don’t look at the next slide until you practice grading this answer). Describe a feedback loop and list its components. A feedback loop is a device used by the body to maintain equilibrium. It has a control center and signals that go in and out.

31. Awarding Points… • If you’ve graded the preceding answer with the rubric, go ahead and look at the following suggestions. • For the feedback loop trait, an instructor might award 1 point to the student. • For the events of cycle trait, the student did not describe the events so 0 points would be awarded. • For the primary components trait, the student described them with only partial accuracy and would thus earn 1 point. • The student score would be 2 of 6 and both the instructor and student could easily identify the areas of weakness in the student’s response.

32. How did it go? • Was the rubric written in such a way that a non-biology teacher could use it effectively? • Could a non-biology teacher distinguish between partial accuracy and inaccuracy? • While rubrics aren’t necessarily developed to allow non-specialty faculty to be able to utilize them, thinking along those lines does make for more explicit descriptions within each of the boxes.

33. Tinkering with the Rubric Itself • After using the rubric, perhaps the information in the boxes could be changed to improve the rubric. • For example, the 1 point response for primary components might be changed from: • Components are described with partial accuracy to • Only some of the 3 components are named or they are described out of sequence. • Another consideration would be to split that trait into two separate traits – • one to name the components and • another to put them in the proper sequence. • Otherwise, it is tempting to assign half-points.

34. Analysis • After utilizing the rubric for multiple student answers, an instructor will have accumulated data that is very useful if analyzed. • For example, imagine that the instructor kept track of rubric scores and compiled the results from 15 students as described in Table 5.

35. Table 5 – A Scoring Rubric Analysis

36. What to Do With the Analysis? • Simply examining the totals within each column can be revealing or it may be useful to calculate an average rubric score for each trait as was done in Table 5. • It is easy to pick out strengths and weaknesses by examining the averages.

37. What to Do With the Analysis? • The instructor in this situation might make the following decisions: • Provide more emphasis within lecture on the basic definition of a feedback loop (as a cycle of events). • Provide more emphasis within lecture on the events that take place within that repeating cycle. • Give students a homework assignment that includes a question very similar to the one asked here and let them practice and get feedback in this area prior to having to answer the question in a quiz or exam setting. • Provide the students with the rubric prior to the exam.

38. Are You Convinced? At this point, go back to the section on Benefits of Rubrics (slide 5 and 6) and see if this exercise and the resulting fictitious list of recommendations verifies that the benefits of using rubrics do in fact exist!