1 / 20

James Christie

James Christie. Automated Essay Marking Does it work?. Essay Definition 1 of 2.

elom
Download Presentation

James Christie

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. James Christie Automated Essay Marking Does it work?

  2. Essay Definition 1 of 2 … requires a response composed by the examinee, usually in the form of one or more sentence, of a nature that no single response or pattern of responses can be listed as correct, and the accuracy and quality of which can be judged subjectively only by one skilled or informed in the subject, …

  3. Essay Definition 2 of 2 … but even an expert cannot usually classify a response as categorically right or wrong. Rather, there are different degrees of quality or merit which can be recognized. … attributed to Stalnaker, 1951

  4. Possible criteria for automated essay marking • Ease of creating a scoring schema • Ability to score on various mark regimes • Ease of identification on non-scoring elements • Ease of modification • should scoring error(s) occur • Consistent and reproducible scoring • Acceptability of results to • human markers, essayists, … • Defensibility • Accuracy and precision • Coachability avoidance • Cost

  5. Sample of a schema for Robert Gordon, founder of the Robert Gordon University. Essay is marked out of 25, while the sum for individual items is 35 ~ not a mistake! Mark Item 0 Robert Gordon 3 born 1668 Castlegate Aberdeen 3 inherited 1680 £1,000 5 graduated 1689 Marischal College Aberdeen 3 retired 1720 Aberdeen 1 died 1731 [max:15] 1 Father 2 Arthur Gordon 2 Edinburgh advocate [max:5] 1 Mother 2 Isabella Gordon 2 nee Isabella Menzies [max:5] 1 Grandfather 2 Robert Gordon 1 Catographer 3 Blaeu's Atlas 1654 [max:7] 1 Trader 2 Danzig Baltic [max:3] [max:25]

  6. Model Essay The cat sat on the mat.

  7. Marking Schema Mark Item 3 cat sat mat [max:3]

  8. Essay Set • ALPHA • The cat sat on the mat. • BRAVO • The cat sat on the floor. • CHARLIE • The dog lay on the floor. • MODEL • The cat sat on the mat.

  9. Schema Report Essay set ............. catmat Schema Report using ... catmat Entities 1 - 4 : Entity ID : 1 2 3 4 Entity Type : a cat sat mat Part ID's : z Essay Name : ALPHA.EXT: y y y y BRAVO.EXT: _ y y _ CHARLIE.EXT: _ _ _ _ MODEL.EXT: y y y y

  10. Content Report Essay set .............. catmat Content Report using ... catmat Essay Name : Words Sentences Usage[%] Coverage[%]Part: z[ 3] Mark[ 3] %[100] ALPHA.EXT: 6 1 50.00 100.00 3 3 100.00 BRAVO.EXT: 6 1 33.33 50.00 0 0 0.00 CHARLIE.EXT: 6 1 0.00 0.00 0 0 0.00 MODEL.EXT: 6 1 50.00 100.00 3 3 100.00 Started on Thursday, February 06 2003 at 15:59:05 Finished on Thursday, February 06 2003 at 15:59:06 Marked 4 file(s): scanned 4 file(s)

  11. Usage & Coverage • Usage • How much of the Essay matched with the Content Schema • Coverage • How much of the Content Schema matched with the Essay

  12. Marking Performance Pearson / Spearman Significance **= 0.01 *= 0.05

  13. Commercial packages – v – SEAR

  14. Future work [content] • maximise use of active and passive voices • cope with spelling & grammar errors • increased coverage of Bloom’s Taxonomy • include non-textual feature(s) • develop • better feedback to the essayist • better feedback to the examiner • plagiarism detection mechanism(s)

  15. Future work [style] • obtain marked essays for style marking • plain ASCII essays • using a common set of metrics • word-processed essays • using a common set of metrics • augmented with word-processing based metrics

  16. If manual marking equalsDa Vinci’s Helical Screw,

  17. then does SEAR equal the first powered flight?

  18. Is the future equal to the ISS?

  19. Is this the future [for style and content]?

  20. Automated Essay Marking Does it work? Yes, but … http://www.rgu.ac.uk/staff/jrc j.christie@rgu.ac.uk T: +44 [0]1224 262725 F: +44 [0]1224 262725

More Related