Essay assessors

1. Essay assessors CPI 494, April 14, 2009 Kurt VanLehn

2. Analysis ofETS’ Criterion video demo

3. Enough essay topics available? • Saw 20 for 8th? • Fit of categories with standards? • Fit with the teacher’s preferences • How to insert graphics, videos, etc. into the essay questions?

4. Plan tools • How to require? • What if students choose the wrong plan? • What about an explanation of each plan? Writer’s handbook? • 10 min of planning sufficient? • Assuem that this is practice for standardized test? • Conversion to paragraphs – OK? • How about compare & contrast or other non-linear plans?

5. Error reports • ITS-wide feature: Good idea to present error categories that didn’t have any errors in this essay. Could encourage students • X axis redundant • Yes, clicking on category does show instances • Would be nice to see non-errors too

6. Repeated submission of an essay • Does restricting the number of submissions prevents gaming? • Does it compare submissions? • Does it make you type them all in? • Can instructors see submission history? • Does it detect copying of essays when two students submit the same essay (even when they attend different schools). Similar essays?

7. Correcting errors • Not easy to correct run on sentence • Telling students to check handbook may not suffice • Need a way to submit changes on a single sentence to check again. • Need faster access to writer’s handbook. • Maybe teacher is necessary for interpretation of general advice in specific context?

8. The organization & development tool • How can it be used to improve the essay? • Only starts an analysis; doesn’t complete it. • Probably is saying, “according to the position in the essay, this should be the conclusion. Does it really read as a conclusion?” Student has to do most of the critiquing work • How does it work with a longer essay? • What if essay is not a 5 paragraph essay? • Does it pay any attention to the student’s plan? • Can it provide a plan that comes from the essay?

9. Good ideas for all ITS(Most should be in an LMS) • Inserting teacher comments in to the marked up solution. • May need a way to highlight a section of text • Library of comments • Students can insert comments into a solution • And it gets flagged in the gradebook! • Should be tied in with LMS’s forums and email system • Threading • Report the most common errors of students in a class so teacher can plan lessons • Report generation for administrators e.g., portfolio management; accountability

10. Authoring tools • Advice to authors about what kind of task to create • E-writer needs a corpus of good and bad essays in order to calibrate it, so it will not work on an instructor’s essay question • until the instructor has hand-scored several hundred student essays • The Critique Writing Analysis Tools are general and appear not to need such calibration, so they should work on the author’s essay questions

11. Answer based feedback? • First sentence says its best for learning writing • Is step-based feedback reasonable? How? • Can it be incorporated into the planning tools?

12. Word weight? • Given a topic, probably means frequency(on-topic essays) / frequency(all essays, including off topic ones)

13. E-raters’ Topic vs. argument topic? • What is the difference? • Why have both?

14. Why a different regression for each essay question? • Writing gendres?