1 / 21

James Christie

James Christie. Automated Marking for Essay Content ~ does it work?. Essay Definition 1 of 2.

eshe
Download Presentation

James Christie

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. James Christie Automated Marking for Essay Content ~ does it work?

  2. Essay Definition 1 of 2 … requires a response composed by the examinee, usually in the form of one or more sentence, of a nature that no single response or pattern of responses can be listed as correct, and the accuracy and quality of which can be judged subjectively only by one skilled or informed in the subject, …

  3. Essay Definition 2 of 2 … but even an expert cannot usually classify a response as categorically right or wrong. Rather, there are different degrees of quality or merit which can be recognized. … attributed to Stalnaker, 1951

  4. Possible criteria for automated essay marking • Ease of creating a scoring schema • Ability to score on various mark regimes • Ease of identification on non-scoring elements • Ease of modification • should scoring error(s) occur • Consistent and reproducible scoring • Acceptability of results to • human markers, essayists, … • Defensibility • Accuracy and precision • Coachability avoidance • Cost

  5. Model Essay The cat sat on the mat.

  6. Marking Schema Mark Item 3 cat sat mat [max:3]

  7. Content Data Structure 1 z a a 0 2 3 4 2 z f cat 1 0 0 0 3 z f sat 1 0 0 0 4 z f mat 1 0 0 0

  8. Essay Set • ALPHA • The cat sat on the mat. • BRAVO • The cat sat on the floor. • CHARLIE • The dog lay on the floor. • MODEL • The cat sat on the mat.

  9. Process interface What LEVEL of Diagnostics to use [0 ... 3] : 0 What ESSAY SET to use .................... : catmat Enter SCHEMA to use ...................... : catmat ALPHA.EXT . BRAVO.EXT . CHARLIE.EXT . MODEL.EXT . Started on Thursday, February 06 2003 at 15:59:05 Finished on Thursday, February 06 2003 at 15:59:06

  10. Schema Report Essay set ............. catmat Schema Report using ... catmat Entities 1 - 4 : Entity ID : 1 2 3 4 Entity Type : a f f f Part ID's : z Essay Name : ALPHA.EXT: y y y y BRAVO.EXT: _ y y _ CHARLIE.EXT: _ _ _ _ MODEL.EXT: y y y y

  11. Content Report Essay set .............. catmat Content Report using ... catmat Essay Name : Words Sentences Usage[%] Coverage[%]Part: z[ 3] Mark[ 3] %[100] ALPHA.EXT: 6 1 50.00 100.00 3 3 100.00 BRAVO.EXT: 6 1 33.33 50.00 0 0 0.00 CHARLIE.EXT: 6 1 0.00 0.00 0 0 0.00 MODEL.EXT: 6 1 50.00 100.00 3 3 100.00 Started on Thursday, February 06 2003 at 15:59:05 Finished on Thursday, February 06 2003 at 15:59:06 Marked 4 file(s): scanned 4 file(s)

  12. Marking Performance Pearson / Spearman Significance **= 0.01 *= 0.05

  13. Commercial packages – v – SEAR

  14. Future work [content] • maximise use of active and passive voices • cope with spelling (and grammar) errors • increased coverage of Bloom’s Taxonomy • include non-textual feature(s) • develop • better feedback to the essayist • better feedback to the examiner • plagiarism detection mechanism(s)

  15. If manual marking equalsDa Vinci’s Helical Screw,

  16. then does SEAR equal the first powered flight?

  17. Is the future equal to the ISS?

  18. Is this the future [for style and content]?

  19. Future work [style] • obtain marked essays for style marking • plain ASCII essays • using a common set of metrics • word-processed essays • using a common set of metrics • augmented with word-processing based metrics

  20. James R Christie Does it work? Yes, but … http://www.jkp.christie.btinternet.co.uk james.christie@btinternet.com

More Related