1 / 66

Practical Applications of Temporal and Event Reasoning

Practical Applications of Temporal and Event Reasoning. James Pustejovsky, Brandeis Graham Katz, Osnabrück Rob Gaizauskas, Sheffield ESSLLI 2003 Vienna, Austria August 25-29, 2003. Course Outline. Monday - Theoretical and Computational Motivations Overview of Annotation Task

zanta
Download Presentation

Practical Applications of Temporal and Event Reasoning

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Practical Applications of Temporal and Event Reasoning James Pustejovsky, Brandeis Graham Katz, Osnabrück Rob Gaizauskas, Sheffield ESSLLI 2003 Vienna, Austria August 25-29, 2003

  2. Course Outline • Monday- • Theoretical and Computational Motivations • Overview of Annotation Task • Events and Temporal Expressions • Tuesday • Anchoring Events to Times • Relations between Events • Wednesday • Syntax of TimeML Tags • Semantic Interpretations of TimeML • Relating Annotations • Temporal Closure • Thursday • Automatic Identification of Expressions • TimeBank Corpus • TANGO • Automatic Link Construction • Friday- • Outstanding Problems

  3. Thursday Topics • Automatic Identification of Expressions • TimeBank Corpus • TANGO • Automatic Link Construction • Developmental Complexity Models of Narratives

  4. Annotating the Corpus Distinguishing features of TimeML are: • It builds on TIMEX2 (Ferro et al., 2001), but introduces new features such as temporal functions to allow intensionally specified expressions like three years ago. • It identifies signals determining interpretation of temporal expressions, such as temporal prepositions (for, during) and temporal connectives (before, after). • It identifies a wide range of classes of event expressions, such as tensed verbs (hasleft), stative adjectives (sunken), and event nominals (merger). • It creates dependencies between events and times or other events, such as anchoring (John left on Monday), ordering (The party happened after midnight), and embedding (John said Mary left).

  5. The Conceptual and Linguistic Basis • TimeML presupposes the following temporal entities and relations. • Events are taken to be situations that occur or happen, punctual or lasting for a period of time. They are generally expressed by means of tensed or untensed verbs, nominalisations, adjectives, predicative clauses, or prepositional phrases. • Times may be either points, intervals, or durations. They may be referred to by fully specified or underspecified temporal expressions, or intensionally specified expressions. • Relations can hold between events and events and times. They can be temporal, subordinate, or aspectual relations.

  6. Annotating Events • Events are marked up by annotating a representative of the event expression, usually the head of the verb phrase. • The attributes of events are a unique identifier, the event class, tense, and aspect. • Fully annotated example: All 75 passengers <EVENT eid="1" class="OCCURRENCE" tense="past" aspect="NONE"> died </EVENT> • See full TimeML spec for handling of events conveyed by nominalisations or stative adjectives.

  7. Annotating Times • Annotation of times designed to be as compatible with TIMEX2 time expression annotation guidelines as possible. • Fully annotated example for a straightforward time expression: <TIMEX3 tid="1" type="DATE" value="1966-07" temporalFunction="false"> July 1966 </TIMEX3> • Additional attributes are used to, e.g. anchor relative time expressions and supply functions for computing absolute time values (last week).

  8. Annotating Signals • The SIGNAL tag is used to annotate sections of text, typically function words, that indicate how temporal objects are to be related to each other. • Also used to mark polarity indicators such as not, no, none, etc., as well as indicators of temporal quantification such as twice, three times, and so forth. • Signals have only one attribute, a unique identifier. • Fully annotated example: Two days <SIGNAL sid="1" before </SIGNAL> the attack …

  9. Annotating Relations (1) • To annotate the different types of relations that can hold between events and events and times, the LINK tag has been introduced. • There are three types of LINKs: TLINKs, SLINKs, and ALINKs, each of which has temporal implications. • A TLINKor Temporal Link represents the temporal relationship holding between events or between an event and a time. • It establishes a link between the involved entities making explicit whether their relationship is: before, after, includes, is_included, holds, simultaneous, immediately after, immediately before, identity, begins, ends, begunby, ended by.

  10. Annotating Relations (2) • An SLINK or Subordination Link is used for contexts introducing relations between two events, or an event and a signal. • SLINKs are of one of the following sorts: Modal, Factive, Counter-factive, Evidential, Negative evidential, Negative. • An ALINK or Aspectual Link represents the relationship between an aspectual event and its argument event. • The aspectual relations encoded are: initiation, culmination, termination, continuation.

  11. Annotating Relations (2) Annotated examples: • TLINK: John taught on Monday <TLINK eventInstanceID="2" relatedToTime="4" signalID="4" relType="IS_INCLUDED"/> • SLINK: John said he taught <SLINK eventInstanceID="3" subordinatedEvent="4" relType="EVIDENTIAL"/> • ALINK:John started to read <ALINK eventInstanceID="5" relatedToEvent="6" relType="INITIATES"/>

  12. The Corpus: Text Sources The 300 texts in the TIMEBANK corpuswere chosen to cover a wide variety of media sources from the news domain: • DUC(TIPSTER) texts from the Document Understanding Conference corpus cover areas like biography, single and multiple events (for example dealing with news about earthquakes and Iraq). This covers 12% of the corpus; • Texts from the Automatic Content Extraction (ACE) program come from transcribed broadcast news (ABC, CNN, PRI, VOA) and newswire (AP, NYT). These comprise 17% and 16% of the corpus, respectively. • Propbank (Treebank2) texts are Wall Street Journal newswire texts, making up55%of the corpus.

  13. The Annotation Effort • The annotation of each document involves: • anautomatic pre-processing step in which some of the events and temporal, modal and negative signals are tagged; • a human annotation step which • checks the output of the pre-processing step; • introduces other signals and events, time expressions, and the appropriate links among them. • The average time to annotate a document of 500 words by a trained annotator is 1 hour. • The annotators came from a variety of backgrounds. • 70% of the corpus annotated by TimeML developers; • 30% annotated by students from Brandeis University.

  14. The Annotation Tool (1) • To help the annotators with the annotation effort, a modified version of the Alembic Workbench (Vilain and Day 1996) was developed. • When a text is loaded into the tool: • the text is shown in one window with the results of the pre-processing shown via coloured tags. These tags can be edited or deleted, and new tags can be introduced. • linksare shown in a second window These links can be created by selecting tags in the text window and inserting these into the link window.

  15. The Annotation Tool (2)

  16. The Annotation Tool (3)

  17. Key: EVENT TIMEX STATE

  18. O’SMACH, Cambodia (AP) - The top commander of a Cambodian resistance force saidThursday he has sent a team to recover the remains of a British mine removal expert kidnapped and presumedkilled by Khmer Rouge guerrillas almosttwo years ago. February 19, 1998 irrealis before before kidnapped killed sent said recover eventArg Is_included Thursday Cambodian Duration=almosttwo years Is_included relType=after Signal=ago British DCT Within-sentence annotation presumed time

  19. Gen. Nhek Bunchhay, a loyalist of oustedCambodian Prime Minister Prince Norodom Ranariddh, said in an interview with The Associated Press at his hilltop headquarters that he hopes to recover the remains of Christopher Howes within the next two weeks. irrealis before recover ousted hopes Signal=within Is_included the next two weeks said loyalist Is_included before Cambodian DCT Within-sentence annotation

  20. Howes had been working for the Britain-based Mines Advisory Group when he was abducted with his Cambodian interpreter Houn Hourth inMarch 1994. There were many conflicting accounts of his fate. working Signal=when ibefore abducted Is_included Signal=in March 1994 DCT Within-sentence annotation

  21. Howes’ team was clearing mines 17 kilometers (10 miles) from Angkor Wat, the fabled11th Century temple that is Cambodia’s main tourist attraction, when it was attacked. clearing Signal=when ibefore attacked DCT Within-sentence annotation

  22. before kidnapped killed irrealis hopes before ousted irrealis working ID before sent said recover clearing recover ibefore Is_included said abducted Is_included Is_included ibefore Thursday Is_included March 1994 Is_included Is_included the next two weeks after attacked before DCT Within-document annotation (four sentences)

  23. TANGO Demo • Performing link analysis on a text

  24. Closure: lessons from TANGO • Discovery aspects less important The spatial metaphor of TANGO guides the annotator to an event graph that requires less user prompting in order to get to a complete annotation. • Closure makes a consistent and complete annotation possible Closure is still needed to infer implicit relations and to have prior choices of links restrict the relation type of other links.

  25. Domains and Data Sets • Document Collection (300): • ACE • DUC • PropBank (WSJ) • Query Corpus Collection: • Excite query logs • MITRE Corpus • TREC8/9/10 • Queries from TIMEBANK

  26. Corpus Statistics (1) • The statistics collected so far give: • the proportion of tagged text in the corpus • the distribution of: • event classes • TIMEX3 types • LINK types • Information like this gives a useful starting point when analysing the mechanisms used to convey temporal information. • For example, 62% of links were TLINKs, indicating the importance of this link type. • Further analysis of the TLINK will reveal the proportion of explicitly expressed temporal relations (i.e. a signal is used) to implicitly expressed temporal relations (no signal is used).

  27. Corpus Statistics (2) • For example, here is the distribution of tag types:

  28. The Question Corpus • TimeML aims to contribute to Question Answering (QA) – temporal question answering in particular. • Temporal questions can be broadly classified into two categories: • Questions that ask for a temporal expression as an answer, like When was Clinton president of the United States? When was Lord of the Rings: The Two Towers released? We call this type explicit. • Questions that either use temporal expression to ask for a non-temporal answer or that ask about the relations holding between events. Who was president of the United States in 1990? Did world steel output increase during the 1990s? We call this type of temporal question implicit.

  29. The Question Corpus (2) • To evaluate the usefulness of TimeML for (temporal) QA, a question corpus of 50 questions has been created. • This corpus was annotated according to a specially developed annotation scheme. This scheme allows features such as: • the type of the expected answer • the volatility of the answer (i.e. how often it changes) to be annotated. • The questions contained in the corpus cover both types mentioned above. Examples of questions in the corpus are: When did the war between Iran and Iraq end? When did John Sununu travel to a fundraiser for John Ashcroft? How many Tutsis were killed by Hutus in Rwanda in 1994? Who was Secretary of Defense during the Gulf War?

  30. Conclusion • There has as yet been no time to analyse the corpus • the statistics collected so far do not represent such an analysis, but only a very preliminary scoping. • We anticipate that the corpus will allow a new range of explorations both theoretical and practical. For example: • Theoretical: can study to what extent temporal ordering of events is conveyed • explicitly through signals, such as temporal subordinating conjunctions, versus • implicitly through the lexical semantics of the verbs or nominalizations expressing the events. • Practical: can trainand evaluate algorithms to determine event ordering and time-stamping, and explore their utility in QA.

  31. Tempex • Wilson and Mani (2002) MITRE • Timex2 parsing • Direct Interpretation to ISO value

  32. What is TempEx? • Perl module that implements the TIDES Temporal • Annotation Guidelines • Handles many formats • - Feb. 10, Feb. 10th, February Tenth • Some parts of standard not fully implemented • - Embedded Expressions: Two weeks ago tomorrow • - Unknown Components: June 10 (VAL = XXXX0610) • Some very small extensions • - Easter gets an ALT_VAL

  33. Sample OutputPOS Tags removed I got up <TIMEX2 TYPE="DATE" VAL="20010216TMO" MOD="EARLY">early this morning</TIMEX2>. I ate lunch <TIMEX2 TYPE="TIME" VAL="20010216T1207">an hour and a half ago</TIMEX2>. In <TIMEX2 TYPE="DATE" VAL="FUTURE_REF">the future</TIMEX2>, I will know better. I went to Hong Kong <TIMEX2 TYPE="DATE" VAL="2000W40">the week of October third</TIMEX2>. I went to Hong Kong <TIMEX2 TYPE="DATE" VAL="2000W42">the third week of October</TIMEX2>. Reference Date: 02/16/2001 13:37:00

  34. Performance Interannotator agreement TIMEX VAL MOD Human x Human 0.789 0.889 0.871 TempEx x Human 0.624 0.705 0.301 Speed - 0.5Megabyte/Minute Demo: Tempex

  35. TIMEX3 Parser Objects (T3PO) Automatic TimeML Markup • Extends TIDES TIMEX2 annotation: • Broader Coverage of temporal expressions • Larger lexicon of temporal triggers • Delays Computation of Temporal Math: • Annotation with Temporal Functions • Import Hobbs’ Semantic Web Temporal System • Distinct Cascaded Processes: • TIMEX3 and signal recognizer; • Event Predicate recognizer • LINK creation transducer.

  36. T3PO Overview • Preprocessing: • POS, Shallow Parsing • Three Finite State modules: • Temporal Expressions • Events • Signals • Links • Discourse Information

  37. Temporal Expressions • Extension to Timex2 • Coverage • Absolute ISO Values • Signals • Functional Representation: • Anchor Resolution • Suite of Temporal Functions

  38. Event Recognition • In Verbal uses VG chunks: • Encodes Tense and Aspect information • Nominal Events using: • Morphological information • POS ambiguity • Signals • Semantic Information

  39. Link Recognition • Event -Timex Links Use of heuristics. Extra-sentential (Event-DCT Links) • Event-Event Links: • Intrasentential • SLINKS (evidential) • SLINKS (infinitivals) • Extrasentential

  40. Preliminary Tests Estimation(6 documents with human annotated version)

  41. Mani et al. (2003) • A variety of theories have been proposed as to the roles of semantic and pragmatic knowledge in event ordering • Very little prior work on corpus-based methods for event ordering • They carried out a pilot experiment with 8 subjects who provided event-ordering judgments for 280 clause pairs. Results revealed that: • A. Narrative convention applied only 47% of the time in ordering events in 131 pairs of successive past-tense clauses • B. ~75% of clauses lack explicit time expressions • Suggests that anchoring events only to explicit times wouldn’t be sufficient

  42. Motivation • Question Answering from News • When do particular events occur • When did the war between Iran and Iraq end? • Which events occur in a temporal relation to a given event • What is the largest U.S. military operation since Vietnam? • Multi-Document News Summarization • Event chronologies (e.g., timelines) are used widely in everyday news • Need to know when events occur, to avoid inappropriate merging of distinct events

  43. Problem Characteristics • In news, events aren’t usually described in the (narrative) order in which they occur • Temporal structure dictated by perceived news value • Latest news usually presented first • News sometimes expresses multiple viewpoints, with commentaries, eyewitness recapitulations, etc., • Temporal ordering appears to involve a variety of knowledge sources • Tense & aspect • Max entered the room. Mary stood up/was seated on the desk. • Temporal adverbials • Simpson made the call at 3. Later, he was spotted driving towards Westwood. • Rhetorical relations and World Knowledge • Narration Max stood up. John greeted him. • Cause/Explanation Max fell. John pushed him. • Background Boutros-Ghali Sunday opened a meeting in Nairobi of ....He arrived in Nairobi from South Africa.

  44. Event Ordering and Reference Time • Reference Time (Reichenbach 47) – provides temporal anchoring for events uI hadr mailedethe letter (when John came and told me the news). Past Perfect: e < r < u Past: e =r < u • Movement of Reference Time depends on tense, aspect, rhetorical relations, world knowledge, etc. u1John pickedr1,e1 up the phone (at 3 pm) u2He hadr2 tolde2 Mary he would call her Assuming r2 = e1 (stative), e2 < e1 (Hwang & Schubert 92) u1,u2 r1=3pm e1 r2 e2

  45. Two Clause Interpretation Past2Past • Max stood up. John greeted him • AFTER relation • Max fell. John pushed him. • BEFORE relation • Max entered the room. Mary was seated behind the desk. • Equal (SIMULTANEOUS or INCLUDE) relation Past2PastPerfect • Max entered the room. He had drunk a lot of wine • BEFORE relation PastPerfect2Past • Max had been in Boston. He arrived late. • AFTER relation

  46. Factors That Determine Relation • Aspect: • Progressive or not • Order: • The iconic order in text • Tempex: • The existence of a temporal expression • Tense: • Past vs. Past Perfect • Meaning: • Lexical or constructional semantics of the sentence.

  47. Event Ordering: Human Experiment Foreign Minister John Chang confirmed to reporters that Lien, during a Sunday stopover in New York, had made a detour to a “third country'' with which Taiwan has no diplomatic ties and would not return to Taipei as scheduled on Monday. But Chang and other Taiwan spokesmen pointedly refused to confirm local media reports that Lien was in Europe, much less to confirm that he had flown to France. Since a civil war divided them in 1949, China has regarded Taiwan as a rebel province ineligible for sovereign foreign relations. In mid-1995, a furious Beijing downgraded ties with Washington and froze talks with Taiwan after President Lee Teng-hui made a private visit to the United States. ‘meets’ and ‘during/includes’ not used

  48. Results on Human Event Ordering 5 subjects X 48 exs = 240 exs 131 109 While shallow features can be leveraged in ordering, meaning and commonsense knowledge also play a crucial role Narrative convention applies in less than half of the Past to Past cases and less than two-thirds of the Past Perfect to Past cases

  49. Overall: 24/40 = 60% Removing Unclears: 24/33 = 72% Unclears Breakdown: Clear: 1; POS error: 1; Not enough context: 5 Other Disagreements: Polar Opposition: 4 (1 difficult) Entirely Before vs Equal: (1) In an interview with Barbara Walters to be shown on ABC’s “Friday nights”, Shapiro said he tried on the gloves and realized they would never fit Simpson’s larger hands. Entirely Before vs Upto: (2) They had contested the 1992 elections separately and won just six seats to 70 for MPRP. Based on 3 subjects on a common set of 40 examples Fine-grained decisions about temporal ordering, are difficult Subjects show an acceptable level of agreement on more coarse-grained ordering (collapsing Entirely Before and Upto) Inter-annotator Agreementon Temporal Ordering

  50. Automatic Link Identification in Text • Mani, Schiffman, and Zhang (2003)

More Related