Do Longer Papers Mean Higher Scores?

1. Hello from Ohio! Tom Beery beery.t@rhodesstate.edu and John Fallon fallon.j@rhodesstate.edu English Faculty Rhodes State College, Lima, OH USA Do Longer Papers Mean Higher Scores?

2. Quick Exercise: You Be the Grader!

3. Directions • Read the prompt and take two minutes to grade the paper • Holistic scoring: Give the paper one score on a scale 1 (very low) – 8 (very high); no half scores. • When scoring, you may generally consider such traits as: • Organization • Development • Critical thinking • Style • Mechanics

4. Now for our session In 2005, SAT and ACT add a writing sample to their tests. Writing experts say this is no way to test writing ability.

5. One-time writing assessments • Are not representative of the writing “process”: brainstorming, drafting, revising, editing, and proofreading • Are not supported by the Conference on College Composition and Communication

6. CCCC Position Statement on Writing Assessment (revised 2009) “Any individual's writing ability is a sum of a variety of skills employed in a diversity of contexts….” “One piece of writing—even if it is generated under the most desirable conditions—can never serve as an indicator of overall writing ability, particularly for high-stakes decisions.”

7. Adding a one-time writing sample to the SAT and ACT… • Adds time to the test • Fosters changes in classroom pedagogy • Doesn’t equate to writing ability (CCCC) • Leads to inaccurate evaluations (e.g., test procedures)

8. Scoring the SAT: Perelman Dr. Les Perelman found a 90% correlation between paper length and score: “I have never found a quantifiable predictor in 25 years of grading that was anywhere as near as strong as this one is.” “Write long, badly, and prosper!”

9. Perelman’s recommendations • High stakes testing is too important to leave to private testing agencies • Have students write two essays over the course of a day • Gather graders together for a weekend to grade (e.g., calibrate graders on the rubric)

10. Was Perelman Right? Is paper length correlated to paper score?

11. Our Study • We needed someone to take the test • Regular and CP high school juniors from Ohio Hi Point Career Center in Bellefontaine, OH • We needed someone to grade the test • 20 teachers: 8 college and 12 high school

12. Next Step: Training Day • Give 12 teachers two hours of rubric training • Divide into 6 teams – 3 high school and 3 college • Assign each team one rubric trait • Discuss proposed revisions • Rewrite rubric to reflect consensus changes

13. Our Rubric vs. Theirs: Essentially, the Same

14. Procedures All 10 teams scored all 42 papers. Did longer papers receive higher scores? Yes, longer papers received higher scores.

15. Score by Word Count

16. Scatterplot: Word Count & Score The more the points cluster tightly about the line, the higher the magnitude of the correlation (between word count and score). We have a tight cluster!

17. Scoring Exercise for Paper #7

18. Paper #7: A Microcosm • If a paper even looks longer, it will receive a higher grade. • Trained readers (6 teams) scored this paper 17.3 • Untrained readers (4 teams) scored this paper 23.25 • Rubric-trained teams and rubric untrained teams scored very differently.

20. Trained vs. Untrained Levels of Magnitude .1 small .3 medium .5 large This chart looks at the strength of the relationship between word count and paper score. A correlation of + 1 is a perfect positive correlation.

21. Options for Rubric Training Rubrics may be used for two entirely separate activities: evaluation and assessment. • Evaluation: Use of a trait-scoring rubric would help eliminate biases in teacher evaluation (risking one trait being too prominent in how a reader scores the whole paper, e.g., the students can’t spell; therefore, they can’t write). • Assessment: Using a trait rubric for assessment in the classroom would promote discussion for what specifically makes “good” writing before the students actually write.

22. Final Observations • Grading with a rubric does not ensure standards. • Even just a couple hours of rubric training is enough to norm evaluators to a standard. • Moreover, rubrics can be used for two separate purposes: evaluation and assessment. • Perelman was right. Word count does influence paper score; however, so does rubric training.

23. Questions? Thank you!