Annotating Committed Belief

1. Annotating Committed Belief Mona Diab, Lori Levin, Teruko Mitamura, Owen Rambow CMU/Columbia-CCLS diab,rambow@ccls.columbia.edu lsl,teruko@cs.cmu.edu

2. Motivating Example 1 • Suppose: IR System • Query: show documents discussing instances of peasants being robbed of their land • Document found 1: The people robbing Iraqi peasants of their land should be punished • Document found 2: Robbing Iraqi peasants of their land would be bad Relevant Irrelevant

3. Motivating Example 2 • Suppose: QA System • Question: Did the humanitarian crisis in Iraq end? • Text found 1: He arrived on Tuesday, bringing an end to the humanitarian crisis in Iraq. • Text found 2: He arrived on Tuesday, calling for an end to the humanitarian crisis in Iraq Answer: Yes Answer: I do not know

4. What is “Committed Belief”? • Model: two discourse participants • speaker/writer (S) • hearer/reader (H) • S and H have beliefs about the world • Note: knowledge = belief + encyclopedic truth; not relevant here • Note: discourse participants also have desires and intentions; not relevant here

5. What is “Committed Belief”? • A lot of (but not all) communication is about S informing H of S’s beliefs • Patrick is a nice man: H believes nice(Patrick) • I hope Patrick is a nice man: H has no belief about Patrick being nice or not, only a desire • Typically, S’s goal is that H adopt those beliefs • Beliefs about the world can be of different strengths; linguistic means can signal strength: • I know Patrick is a nice man: Committed Belief • I think Patrick is a nice man: Non-Committed Belief • Can use numerical weights to represent strength; we use 2 discrete categories • Committed belief • Non-committed belief

6. Annotation Categories • We annotate propositions on the verb or noun • Committed belief: S indicates in this utterance that S believes the proposition • I know Mark and Sandra have eloped. • Non-committed belief: S identifies the proposition as something which S could believe, but S happens not to have a strong belief in the proposition • Mark and Sandra may have eloped. • Not applicable: for S, the proposition is not of type in which S is expressing a belief, or could express a belief. Usually, this is because the proposition does not have a truth value in this world. • I wish Mark and Sandra would finally elope.

7. Why Is RecognizingCommitted Belief Important? • Committed-Belief Annotation Distinguishes • Propositions that are asserted as true (CB) • Propositions that are asserted but speculative (NCB) • Propositions that are not asserted at all (NA) • Important whenever we need to identify facts • IR • QA • Summarization • …

8. Map from Committed Belief Detection to Useful Functionality Traditional IR System Or QA System Automatic Committed Belief Filter Response including unasserted propositions Response based on asserted propositions only

9. Committed Belied is not Tense CB = committed belief, NA = No asserted belief • We have a special feature to indicate future tense on CB (committed belief) and NCB (non-committed belief)

10. Committed Belied is not Factivity CB = committed belief, NA = No asserted belief • Committed-belief annotation and factivity annotation are complementary

11. Some Examples • Word File (English) • Mona on Arabic

12. Status • Have complete manual • Have core corpus annotated (including Arabic), working on 15% double annotation

13. Proposed Annotation Work • Perhaps annotate desire in same manner (much of NA is desire) • Work on one language until prediction (across genres!) works, then move to next language • We do not know how much data is needed • We do not know how much variability in expressing committed beliefs between languages there is