1 / 12

From Individual Ratings to Consensus Ratings of L2 Essays

From Individual Ratings to Consensus Ratings of L2 Essays. Vivian Lindhardsen Copenhagen Business School Denmark Ph.D. Student Work in progress. From Individual Ratings to Consensus Ratings Vivian Lindhardsen, Copenhagen Business School.

sabina
Download Presentation

From Individual Ratings to Consensus Ratings of L2 Essays

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. From Individual Ratings to Consensus Ratings of L2 Essays Vivian Lindhardsen Copenhagen Business School Denmark Ph.D. Student Work in progress

  2. From Individual Ratings to Consensus RatingsVivian Lindhardsen, Copenhagen Business School • What are the decision-making strategies that experienced raters employ when going through individual ratings and consensus ratings to reach a final score?

  3. From Individual Ratings to Consensus RatingsVivian Lindhardsen, Copenhagen Business School • 20 highly experienced raters • 15 L2 essays (exit exam of upper secondary school, business line)

  4. From Individual Ratings to Consensus RatingsVivian Lindhardsen, Copenhagen Business School • Introspection • Observation • Retrospection

  5. From Individual Ratings to Consensus RatingsVivian Lindhardsen, Copenhagen Business School • Official criteria • Volume, Structure, Content and use of source materials, Language, Style, Other/General • Self-monitoring or validation strategies

  6. From Individual Ratings to Consensus RatingsVivian Lindhardsen, Copenhagen Business School Individual Ratings First impression: Scan for length, title, overall structure Thorough interpretation and assessment: Read, classify errors, edit text, summarize ideas, etc. Preliminary score

  7. From Individual Ratings to Consensus RatingsVivian Lindhardsen, Copenhagen Business School Consensus Ratings Exchange preliminary scores If different or not pass, then Validation Exemplify directly from text, define or revise own assessment strategies, consider exam or task requirements, compare essays, deliberate own teaching practices Final score

  8. From Individual Ratings to Consensus RatingsVivian Lindhardsen, Copenhagen Business School MG: “Essay 6. I gave it a 9” GP: “Well, I actually gave it a 10 because the language is fluent and he is very good at using the source texts. And it is logical. Impressive vocabulary EMBARKING ON A CRUSADE”. MG: “I see what you mean, but they have a couple of hours for this so I think there should be more than one page. We can expect that from them”. GP: “Yes, it is difficult. I think that had it been my student I would have told him to write some more and keep on what he’s doing”. But you’re right. 9 is probably more appropriate. It is too short”. Note: Average score for essay 6: 8.9

  9. From Individual Ratings to Consensus RatingsVivian Lindhardsen, Copenhagen Business School Reference to Official Criteria

  10. From Individual Ratings to Consensus RatingsVivian Lindhardsen, Copenhagen Business School • Score preference • 92% preferred consensus score • 8% preferred individual score

  11. From Individual Ratings to Consensus RatingsVivian Lindhardsen, Copenhagen Business School • Dominance? • To be finalized

  12. From Individual Ratings to Consensus RatingsVivian Lindhardsen, Copenhagen Business School • Implications • Rater training • Validation of consensus score

More Related