NLP Research for Commercial Development of Writing Evaluation Capabilities

1. NLP Research for Commercial Development of Writing Evaluation Capabilities Jill Burstein Educational Testing Service Presentation for OTSLAC Columbia University December 2, 2004

2. CriterionSM, E-rater®, Critique, C-rater, & more … Jill Burstein Claudia Leacock Thomas Morton Educational Testing Service Martin Chodorow Hunter College, CUNY Susanne Wolff Princeton University

3. Let’s Talk About Writing & Assessment

4. Educators’ Vision: Writing Skill Development • Master basic skills in K-12 • Grammar, spelling, punctuation, etc. • Perfected the 5-paragraph essay • U.S. Concept • Thesis, 3 Main Points, Conclusion • Writing within and beyond discipline • Address different audiences • Generate various genres: persuasive, compare-and–contrast, research writing within discipline

5. Evaluation of Vision Through Writing Assessments • High stakes: Undergraduate & Graduate • Admissions • Placement • No/Low stakes: K-12 • Statewide and national assessments • Classroom instruction

6. What do essays look like?

7. The Reality of Writing Quality • Timed Assessments • Up to 500 words (grade-level dependent) • Not Literary Essays !creative !irony !metaphor • Instructional Uses • Maybe Longer Essays • Better Quality with Revision Facility

8. Most essays look like this

9. And others like this… “You are stupid. You are stupid because you can't read. You are also stupid becuase you don't speak English and because you can't add. Your so stupid, you can't even add! Once, a teacher give you a very simple math problem; it was 1+1=?. Now keep in mind that this was in fourth grade, when you should have known the answer. You said it was 23! I laughed so hard I almost wet my pants! How much more stupid can you be?! So have I proved it? Don't you agree that your the stupidest person on earth? I mean, you can't read, speak English, or add. Let's face it, your a moron, no, an idiot, no, even worse, you're an imbosol.”

10. And this …. “I THINK THAT EVERYONE SHOULD BE ABLE TO WEAR WHATEVER THE HELL THEY WANT TO WEAR. “

11. And this… “I don't know how to explain this question because I took a nap while listening. Sorry. “

12. And this … This is my topic on presidents. The one i will be talking about is Bill Clinton. He like most of us have fingers, with fingernails. He also has two arms were his fingers are on, were teh fingernails connect.What can a arm be without a hand? nothing thats the answer, so he obviously has two hands in witch the fingers are on, with the fingernails connect too. He also has eyes.....EYES o yeah even him the big cheese has eyes, its weird cuz not many people have eyes but this good president does, and he can SEE you SEE you, you might not be able to see him , but he can SEE you.One time we were on AOL chat and i was talking to him he said he like to climb trees in his underwear well his arms were covered with sauce....pizza sauce. He said it makes him feel free and good about himself. Also in the chat he said he has a pigeon for a pet and the things name is Frances, he said they like to make bacon together in the mourning and at night.And they eat at his friends Y place mostly everyday.

13. And this … “Is true that in so many jobs people have to wear dress codes for so many reazons.Like in the restaurant the workers are obligaded to use a drees code because they have to look differently and have a good looking to impresionate the cutmers.Not necesesary at all the schools have drees codes but the ones that havet is because Arriba todos mis compas ya llego el rey del cristal, y yo mismo lo cosino para mejor calidad por esos mismos motivos me busca la federal la dea de estados unidos tambien me quiere agarar. si ellos me buscan por tierra yo me les pelo por mar si piensan que ando colima yo me paseo en michoacan por la ruana y poir tepeque aguililla y cuancoman ,cuantas libras va a llevarse “ [Descriptive Translation: “It’s a rhyme about a drug-runner … the guy is basically saying that he's the king of Cristal meth, wanted by the DEA and the Feds. ”]

14. Human Scoring Algorithm

15. essay human reader score (S1) human reader score (S2) YES NO Is |S1 - S2| > 1 ? Final Score = mean Final Score = mode, or mean of closest expert human reader score (S3)

16. Building Automated Essay Scoring Capabilities

17. Some History • PEG – 1960’s essay scoring (Page, 1966) • Transformation of essay length • Some syntactic analysis • Convincing results • Writer’s Workbench (Cherry et al, 1982) • Editing tool for students • Diction, style, spelling • Discourse structure (‘topic sentence’ identification) • Intelligent Essay Assessor (Landauer et al, 1998) • Essay scoring with latent semantic analysis (LSA) • Style and mechanics measures

18. Assessment Market Technologically Ready • Increase in Internet & Computer Access • Instructional computers with Internet access in public schools: 12.1 to 1 in1998 & 4.8 to 1 in 2001 (NCES, 2002) • Web resources used in over 40% of college courses (Campus Computing Project, 2000) • 99% of public schools have internet access (NCES report, 2002) • State Assessments: Increase in computer-based delivery • Largest Test Publishers offer 850+ digital textbook titles

19. Motivation in Assessment • Cost Savings for Large-Scale Assessments • Classroom Integration for Instruction • More practice writing possible • Electronic writing portfolios • Individual performance assessment • Classroom assessments

20. Educators’ Questions About Innovation • Reliability: Can automated essay assessments increase scoring consistency for authentic assessments? • Assessment Type: Can automated scoring introduce more varied high-stakes assessments? • Costs/Performance: Can scoring costs be reduced, but scoring performance maintained?

21. Starting Development

22. What should a good essay look like? • Clearly states the author's position, and effectively persuades the reader of validity of author's argument. • Well organized, with strong transitions helping to link words and ideas. • Develops its arguments with specific, well-elaborated support. • Varies sentence structures and makes good word choices; very few errors in spelling, grammar, or punctuation

23. Mapping Writing Features to NLP Tools

24. E-rater (2/99 – 9/04) • 50+ Writing-Relevant Features • Syntactic Structure Features: clause types • Discourse Structure Features: cue words & terms • Content: Content vector analysis • Lexical Complexity: e.g., word length, unique words • NLP Tools • Syntactic Parses • High-level discourse analyzer • tf*idf (essay level & argument level) • Topic-Specific Models • Training with Human-Scored Essays • Stepwise Linear Regression (Variable Feature Set & Weights) • System Performance • Agreement with Humans • Comparable to Two Humans • E-rater/Human agreement : 59% exact; 98% exact + adjacent

25. essay human reader score (S1) E-rater score (S2) YES NO Is |S1 - S2| > 1 ? Final Score = mean Final Score = mode, or mean of closest expert human reader score (S3)

26. Outcomes of Early Success

27. NY Times Headline Phobia Can you spell imbecile?: E-rater® Gives Good Score to Bad Essay By A. Reporter ETS’s automated scoring system thinks that this essay should get something like a “B.” Would you want your child to do well on this kind of writing? You be the judge. “You are stupid. You are stupid because you can't read. You are also stupid becuase you don't speak English and because you can't add. Your so stupid, you can't even add! Once, a teacher give you a very simple math problem; it was 1+1=?. Now keep in mind that this was in fourth grade, when you should have known the answer. You said it was 23! I laughed so hard I almost wet my pants! How much more stupid can you be?! So have I proved it? Don't you agree that your the stupidest person on earth? I mean, you can't read, speak English, or add. Let's face it, your a moron, no, an idiot, no, even worse, you're an imbosol.”

28. Anomalous Essay Detection Statistical evaluation of word usage to flag anomalous essays • “Your essay does not resemble others being written on this topic.” • “Your essay might not be relevant to assigned topic.” • “Your essay appears to be restatement of the topic with a few additional concepts.” • “Compared to other essays written on this topic, your essay contains more repetition of words.” • “Your essay shows less development of a theme than other essays written on this topic.”

29. What teachers reallywanted:Qualitative Feedback

30. Learning from Assessment Experts • Holistic scores not meaningful to students Score 3:  • While a position may be stated, either it is unclear OR undeveloped. • May have organization in parts, but lacks organization in other parts. • The support of the position may be brief, repetitive, or irrelevant. • Inconsistent control of sentence structure, and incorrect word choices; errors in spelling, grammar, or punctuation occasionally interfere with reader understanding. • Demos for focus groups with teachers, policy makers, assessment experts • Errors in grammar, usage, mechanics, and style • Organization & Development

31. More Innovation – More Questions • Meaningfulness: Is the feedback consistently related to a clearly-defined standard? • Self-Evaluation: Can instructional software help students understand evaluation of their writing? • Improvement: Can writing practice with immediate feedback help students?

32. CriterionSM Online Essay Evaluation Service • Critique writing analysis tools • Grammar • Usage • Mechanics • Style • Organization & Development • E-rater

33. Motivation For New Capability Development • What’s free for commercial use • Spelling • What’s not …free … • Grammar • Usage • Mechanics • Style • What doesn’t exist • Organization & Development

34. Methods

35. Bigrams for Grammar, Usage & Mechanics Errors (Leacock & Chodorow) • Corpus of well-formed text: 30 million words from newspapers • Features: function words and part-of-speech tags a_AT good_JJ job_NN during_IN • Collect frequencies for: • Unigrams of tags and function words • bigrams of tags and function words a_JJ AT_JJ JJ_NN NN_during NN_IN • Method: pointwise mutual information and log likelihood ratio used to detect unexpected sequences – likely violations of English grammar

36. Grammar, Usage and Mechanics Errors • Harvest low probability bigrams from a set of essays. • Low probability bigrams: • DTS_NN, AT_NNS • *these pencil, *every teenagers • Write Filters: • *These pencil is yellow. • but not These pencil erasers are dirty.

37. Grammar • Fragments • Garbled Sentences • Subject-Verb Agreement: the motel are … • Verb form: They are need to distinguish … • Pronoun Errors: Them are my reasons … • Possessive Errors: the studentsgrades • Wrong or Missing Word: The should take the student • Proofread This!: I think my throughproblems

38. Usage • Article Errors: I like these song • Confused Words: Because of there differentgenres … • Wrong Form of Word: the right choose • Faulty Comparison: It is more easier • Nonstandard Verb or Word Form

39. Mechanics • Spelling • Missing Capitalization • Missing Initial Capitalization • Missing Question Mark • Missing Final Punctuation • Missing Apostrophe: Thats about the only thing • Missing Comma • Missing Hyphen: a well deserved vacation • Fused Words • Compound Words • Duplicate Words: escape to the another town

40. Style • Short sentences, unusually long sentences, and passives? • Automatic detection of repeated words • 300 essays manually annotated for repetition • Word-based text features with C5.0 • proportion of word use in essays • distance between repeated word occurrences • pronoun? • word length

41. How Do We Identify Organization & Development in Essay Writing? • Discourse Theories • Lacks Essay-Based Discourse Function • Cue-word & term detection (Cohen, Hirschberg & Litman, Hovy et al, Knott, Mann & Thompson, Vander Linden & Martin, Sidner, & Quirk, et al) • Topical Coherence, Not Discourse Function • TexTiling – (Hearst & Plaunt) • LSA (Landauer et al) • Select-A-Kibbitzer(Weimer-Hastings & Graesser) • Not Student Friendly • RST Trees (Mann & Thompson) • Essay-Based Discourse Analyzer (Burstein, Marcu, & Knight) • Background, Thesis, Main Points, Supporting Ideas, and Conclusion

42. Organization and Development:Essay-Based Discourse Analyzer • 1400+ essays manually annotated with pre-defined labels • Voting Between 3 Systems: 2 Probabilistic & 1 Decision-Based • Probable discourse label sequences • Essay sublanguage: agree, should, would, opinion, for example, because, however... • RST relations: contrast, elaboration, antithesis... • Syntactic structures: infinitive, complement, subordinating clauses... • Identifies background text, thesis statement, main ideas, supporting ideas, & conclusion statement

43. Evaluating Capabilities • Precision, Recall, & F-measure • Trade-off Precision for Recall • Better not to show falsely identified errors • Grammar, Usage, & Mechanics (Bigrams) • Minimum Precision for Deployment • Style & Organization & Development • Human-annotated data • Develop baseline comparisons • Precision, Recall, F-measure outperform baselines & approach human agreement

44. Some Numbers • Grammar, Usage, & Mechanics • (Minimum) Overall System Precision = 0.90 • Discourse Capability (Org & Dev) • Baseline Precision = 0.71 • Overall System Precision = 0.85 • Human agreement = 0.95 • Repetitive Word Use (Style) • Baseline Precision: 0.27 • Overall System Precision = 0.95 • Human Agreement: 0.55

45. E-rater v.2.0 – September 2004 • 12 Features: Relevant to Writing Standards • Grammar, Usage, and Mechanics : Error Types • Style: Sentence Type, Sentence Length, Repeated Words • Organization: Thesis, Main Points, Support, and Conclusion • Content: Vocabulary Usage • Topic-Specific & Grade-Specific Models • Training with Human-Scored Essays • Multiple Regression (Standardized Feature Set & Variable Weights) • System Performance • Agreement with Humans • Comparable to Two Humans • E-rater/Human agreement : up to 62% exact (from 59%) ; 98% exact + adjacent

46. CriterionSM Online Essay Evaluation

47. More toPrevent System Gaming: Word Salad Detector (Morton) • Rare p-o-s tag sequences (mixing up content!) quick The the over brown dogs fox. jumped lazy • Abnormal p-o-s tag distributions kfdl afjidaoi djfd &&&&**