Database of Narrative Schemas for Event Understanding

1. A Database of Narrative Schemas Nate Chambers and Dan Jurafsky Stanford University

2. Two Joint Tasks Events in a Narrative Semantic Roles suspect, criminal, client, immigrant, journalist, government, … police, agent, officer, authorities, troops, official, investigator, …

3. Scripts Schank and Abelson. 1977. Scripts Plans Goals and Understanding. Lawrence Erlbaum. Mooney and DeJong. 1985. Learning Schemata for NLP. IJCAI-85. • Background knowledge for language understanding Restaurant Script • Hand-coded • Domain dependent

4. Applications • Coreference • Argument prediction • Summarization • Inform sentence selection with event confidence scores • Textual Inference • Does a document infer other events • Selectional Preferences • Use chains to inform argument types • Aberration Detection • Detect surprise/unexpected events in text • Story Generation

5. The Protagonist protagonist: (noun) • the principal character in a drama or other literary work • a leading actor, character, or participant in a literary work or real event

6. Narrative Event Chains • ACL-2008 • Narrative Event Chains • Narrative relations • Single arguments • Temporal ordering

7. Inducing Narrative Relations Chambers and Jurafsky. Unsupervised Learning of Narrative Event Chains. ACL-08 • Dependency parse a document. • Run coreference to cluster entity mentions. • Count pairs of verbs with coreferring arguments. • Use pointwise mutual information to measure relatedness. Narrative Coherence Assumption Verbs sharing coreferring arguments are semantically connected by virtue of narrative discourse structure.

9. Chain Example (ACL-08)

10. Schema Example Police, Agent, Authorities Judge, Official Prosecutor, Attorney Suspect, Criminal, Terrorist, … Plea, Guilty, Innocent

11. Narrative Schemas E = {arrest, charge, plead, convict, sentence}

12. Learning Schemas

13. Training Data • NYT portion of the Gigaword Corpus • David Graff. 2002. English Gigaword. Linguistic Data Consortium. • 1.2 million documents • Stanford Parser • http://nlp.stanford.edu/software/lex-parser.shtml • OpenNLP coreference • http://opennlp.sourceforge.net • Lemmatize verbs and noun arguments.

14. Viral Example virus, disease, bacteria, cancer, toxoplasma, strain mosquito, aids, virus, tick, catastrophe, disease

15. Authorship Example book, report, novel, article, story, letter, magazine company, author, group, year, microsoft, magazine

16. Temporal Ordering • Supervised classifier for before/after relations • Chambers and Jurafsky, EMNLP 2008. • Chambers et al., ACL 2007. • Classify all pairs of verbs in Gigaword • Record counts of before and after relations

17. The Database • Narrative Schemas (unordered) • Various sizes of schemas (6, 8, 10, 12) • 1813 base verbs • Temporal Orderings • Pairs of verbs • Counts of before and after relations • http://cs.stanford.edu/people/nc/schemas/

18. Evaluations • The Cloze Test • Chambers, Jurafsky. ACL-2008. • Chambers, Jurafsky. ACL-2009. • Comparison to FrameNet • Chambers, Jurafsky. ACL-2009. • Corpus Coverage • LREC 2010

19. Comparison to FrameNet • Narrative Schemas • Focuses on events that occur together in a narrative. • FrameNet (Baker et al., 1998) • Focuses on events that share core roles.

20. Comparison to FrameNet • Narrative Schemas • Focuses on events that occur together in a narrative. • Schemas represent larger situations. • FrameNet (Baker et al., 1998) • Focuses on events that share core roles. • Frames typically represent single events.

21. Comparison to FrameNet • How similar are schemas to frames? • Find “best” FrameNet frame by event overlap • How similar are schema roles to frame elements? • Evaluate argument types as FrameNet frame elements.

22. FrameNet Schema Similarity • How many schemas map to frames? • 13 of 20 schemas mapped to a frame • 26 of 78 (33%) verbs are not in FrameNet • Verbs present in FrameNet • 35 of 52 (67%) matched frame • 17 of 52 (33%) did not match

23. FrameNet Schema Similarity • Why are 33% unaligned? • FrameNet represents subevents as separate frames • Schemas model sequences of events. One Schema Multiple FrameNet Frames Exchange Narrative Relation trade rise fall slip quote Change Position on a Scale Undressing n/a

24. Evaluations • The Cloze Test • Chambers, Jurafsky. ACL-2008. • Chambers, Jurafsky. ACL-2009. • Comparison to FrameNet • Chambers, Jurafsky. ACL-2009. • Corpus Coverage • LREC 2010

25. Corpus Coverage Evaluation • Narrative Schemas are generalized knowledge structures. • Newspaper articles discuss specific scenarios. • How many events in an article’s description are stereotypical events in narrative schemas?

26. Coverage Example He is painfully aware that if he sold his four-bedroom brick suburban home for the $220,000 that he thinks he can get for it and then paid off his mortgage, he would walk away with, as he putsit, …. He is painfully aware that if he sold his four-bedroom brick suburban home for the $220,000 that he thinks he can get for it and then paid off his mortgage, he would walk away with, as he puts it, …. • Article Text • aware • sell • think • pay • walk • put • Investing Schema • invest • sell • take_out • buy • pay • withdraw • pull • pull_out • put • put_in

27. Coverage Score • Largest Connected Component • Largest subset of vertices such that a path exists between all vertices • Events are connected if there exists some schema such that both events are members. • Article Text • aware • sell • think • pay • walk • put 3 of the 6 are connected 50% coverage

28. Coverage Results • 69 documents • 740 events • Macro-average document coverage • Final coverage: 34% “One third of a document’s events are part of a self-contained narrative schema.”

29. Evaluation Results • The Cloze Test • Schemas improve 36% on event prediction over verb-based similarity • Comparison to FrameNet • 65% of schemas match FrameNet frames • 33% of schema events are novel to FrameNet • Corpus Coverage • 96% of events are connected in the space of events • 34% of events are connected by self-contained schemas

30. Schemas Online • Narrative Schemas • http://cs.stanford.edu/people/nc/schemas/ • Coverage Evaluation (Cloze Test) • http://cs.stanford.edu/people/nc/data/chains/ Nate Chambers and Dan Jurafsky

32. FrameNet Argument Similarity • Argument role mapping to frame elements. • 72% of arguments appropriate as frame elements FrameNet frame: Enforcing Frame element: Rule law, ban, rule, constitutionality, conviction, ruling, lawmaker, tax INCORRECT

33. Coverage Evaluation • Choose a news article at random. • Identify the protagonist. • Extract the narrative event chain. • Match the chain to the best narrative schema with the largest event overlap.

34. Coverage Dataset • NYT portion of the Gigaword Corpus • Randomly selected the year 2001 • 69 random newspaper articles within 2001 • 100 initially chosen, 31 removed that are not news • Identified the protagonist and events by hand

35. The Resource • 3.5% of events in new documents are disconnected from the space of narrative relations • Chambers and Jurafsky. ACL 2008. • 66% of events in new documents are not clustered into generalized narrative schemas. • The extent to which a document discusses new variants of known schemas remains for future work.