Building a System that can Learn by Reading

1. Building a System that canLearn by Reading Kevin Livingston PhD Candidate Cognitive Systems Division EECS Department Northwestern University Presented at University of Dayton November 3, 2006 As part of the Computer Science Research Colloquium Series

2. Text Understanding Systems Text Understanding System Question Answering Knowledge Base Explanation Reasoning

3. Examples of Intelligent Systems • Digital Assistants • Conversational Agents • Imagine a video game character that actually talked to you • Intelligent Games • Search • Question Answering

4. Modeling and Using Knowledge • Frames (Minsky) • Scripts (Shank) • Qualitative Reasoning (Forbus)

5. Semantic Memory • Ontology • Doctors are people • Bombings are events • Semantics • Doctors have patients • Bombings have targets

6. Episodic Memory • Memory Instances • Madrid is a city • Madrid is in Spain • The Madrid Bombings is a terrorist attack • The Madrid Bombings occurred on March 11, 2004 • Al Qaida is a terrorist organization

7. Modeling Reasoning • Logic Based Systems • Cyc (Cycorp; Austin, TX) • Fire (Qualitative Reasoning Group (QRG); Northwestern) • Statistical • Bayesian • Markovian • Neural Nets

8. Text Knowledge Bases Text Understanding System Question Answering Knowledge Base Explanation Reasoning

9. Available Knowledge Bases • Information Retrieval (IR) techniques • mine information from Internet (MUC and TREC) • Open Mind Common Sense (OMCS) • Sentences collected from Internet contributors • Mined for knowledge • Knowledge Machine (KM) • Frame based • ResearchCyc • Latest release ~3,000,000 assertions • Predicate Logic in CycL • First order and some second order constructs

10. ResearchCyc • Bombings are Attacks and Attacks are Events (genls Bombing AttackOnObject) (genls AttackOnObject Event) • The Madrid terrorist attack was a bombing (isa TerroristAttack-September-8-2003-Madrid Bombing) • Al Qaida is an Islamist Terrorist Group (isa AlQaida TerroristGroup-Islamist)

11. ResearchCyc (cont.) • Location of an event (eventOccursAt TerroristAttack-September-8-2003-Madrid CityOfMadridSpain) • Perpetrator of an event (perpetrator TerroristAttack-September-8-2003-Madrid AlQaida)

12. ResearchCyc (cont.) • Person being killed (organismKilled SpaceShuttleChallengerDisaster ChristaMcAuliffe) • Deaths caused by an attack (deathToll TerroristAttack-September-8-2003-Madrid Person 190)

13. Building Knowledge Bases • Slow and Tedious • years to grow Cyc from 1.2M to 3M assertions • Requires Training • measured in weeks+, for GUI tools (SHAKEN) • Expensive • Project Halo estimates $10,000 per page! of AP level Chemistry content

14. Available Information • Encyclopedias • Newspapers • Online sources • Print

15. A Better Way?Teach the Computer to Read

16. What we want to Read • Episodic Knowledge • New people • New events • General Knowledge • “the heart is a pump”

17. Text Standard Model forNatural Language Processing Tagged Text Syntactic Parser POS Tagging Dictionary Grammar Semantic Interpreter Semantics

18. “Time flies like an arrow.”

19. “Time flies like an arrow.” • Time moves quickly just like an arrow does

20. “Time flies like an arrow.” • Time moves quickly just like an arrow does • (You should) time flies like you would an arrow

21. “Time flies like an arrow.” • Time moves quickly just like an arrow does • (You should) time flies like you would an arrow • Time flies in the same way that an arrow would (time them)

22. “Time flies like an arrow.” • Time moves quickly just like an arrow does • (You should) time flies like you would an arrow • Time flies in the same way that an arrow would (time them) • Time those flies that are like arrows;

23. “Time flies like an arrow.” • Time moves quickly just like an arrow does • (You should) time flies like you would an arrow • Time flies in the same way that an arrow would (time them) • Time those flies that are like arrows; • A type of flying insect, "time-flies," enjoy arrows (compare Fruit flies like a banana.)

24. Picture of George Burns And Grace Allen

25. Grace, Those are beautiful flowers. Picture of George Burns And Grace Allen

26. Grace, Those are beautiful flowers. Picture of George Burns And Grace Allen Where did they come from?

27. Don’t you remember, George? Picture of George Burns And Grace Allen

28. You said that if I went to visit Clara Bagley in the hospital I should be sure to take her flowers. Picture of George Burns And Grace Allen So when she wasn’t looking, I did.

29. Picture of George Burns And Grace Allen

30. Common Sense • “You said that if I went to visit Clara Bagley in the hospital I should be sure to take her flowers. So, when she wasn't looking, I did.” • take flowers from her • take flowers to her

31. Picture of Elevator Operator

32. Down? Picture of Elevator Operator

33. That Way! Picture of Elevator Operator

34. Context in the Environment “Down?” What does the question mean? • Which way is down? • Are you going down?

35. Language Understanding Goals include understanding: • The meaning of the text • How it fits into what is known • The purpose of being told Requires: • Common sense knowledge • Awareness of context Our model is to get to knowledge as quickly as possible, with as few intermediate steps as possible.

37. Reader Example An attack occurred in Madrid The bombing killed 190 people The bombing was perpetrated by Al-Qaida (eventOccursAt TerroristAttack-September-8-2003-Madrid CityOfMadridSpain) (perpetrator TerroristAttack-September-8-2003-Madrid AlQaida) (deathToll TerroristAttack-September-8-2003-Madrid Person 190) (isa TerroristAttack-September-8-2003-Madrid Bombing)

38. Lexical Processing “An attack occurred in Madrid.” • “attack” (singular Attack-TheWord “attack”) (denotation Attack-TheWord CountNoun 0 AttackOnObject) (isa ?x AttackOnObject) • “Madrid” (placeName-Standard CityOfMadridSpain "Madrid") (isa CityOfMadridSpain City)

39. Rule Pattern • Pattern: (isa ?event Event) Occur-TheWord In-TheWord (isa ?location GeographicLocation) • Results: (eventOccursAt ?event ?location)

40. Pattern Matching • Pattern: (isa ?event Event) Occur-TheWord In-TheWord (isa ?location GeographicLocation) • Input: • “An” • “attack” (isa ?x AttackOnObject) • “occurred” Occur-TheWord • “in” In-TheWord • “Madrid” (isa CityOfMadridSpain City)

41. Rule Completion “An attack occurred in Madrid.” • Pattern: (isa ?event Event) Occur-TheWord In-TheWord (isa ?location GeographicLocation) • Results: (eventOccursAt ?event ?location) • Constraints: (isa ?event AttackOnObject) • Bindings from Reading: ((?location . CityOfMadridSpain))

42. Remindings “An attack occurred in Madrid.” • Pattern: (isa ?event Event) Occur-TheWord In-TheWord (isa ?location GeographicLocation) • Results: (eventOccursAt ?event ?location) • Constraints: (isa ?event AttackOnObject) • Bindings from Reading: ((?location . CityOfMadridSpain)) • Remindings from Memory: ((?event . TerrorAttack-Sept8-2003-Madrid))

43. Coreference Resolution The bombing was perpetrated by Al-Qaida. An attack occurred in Madrid.

44. Coreference Resolution The bombing was perpetrated by Al-Qaida Results: (perpetrator ?action ?agent) Constraints: (isa ?action Bombing) Bindings from Reading: ((?agent . AlQaida)) An attack occurred in Madrid Results: (eventOccursAt ?event ?location) Constraints: (isa ?event AttackOnObject) Bindings from Reading: ((?location . CityOfMadridSpain)) Remindings from Memory: ((?event . TerrorAttack-Sept8-2003-Madrid))

45. Coreference Resolution The bombing was perpetrated by Al-Qaida Results: (perpetrator ?action ?agent) Constraints: (isa ?action Bombing) Bindings from Reading: ((?agent . AlQaida)) An attack occurred in Madrid Results: (eventOccursAt ?event ?location) Constraints: (isa ?event AttackOnObject) Bindings from Reading: ((?location . CityOfMadridSpain)) Remindings from Memory: ((?event . TerrorAttack-Sept8-2003-Madrid))

46. Coreference Resolution The bombing was perpetrated by Al-Qaida Results: (perpetrator ?action ?agent) Constraints: (isa ?action Bombing) Bindings from Reading: ((?agent . AlQaida)) An attack occurred in Madrid Results: (eventOccursAt ?event ?location) Constraints: (isa ?event AttackOnObject) Bindings from Reading: ((?location . CityOfMadridSpain)) Remindings from Memory: ((?event . TerrorAttack-Sept8-2003-Madrid))

47. Coreference Resolution The bombing was perpetrated by Al-Qaida Results: (perpetrator ?action ?agent) Constraints: (isa ?action Bombing) Bindings from Reading: ((?agent . AlQaida)) An attack occurred in Madrid Results: (eventOccursAt ?event ?location) Constraints: (isa ?event AttackOnObject) Bindings from Reading: ((?location . CityOfMadridSpain)) Remindings from Memory: ((?event . TerrorAttack-Sept8-2003-Madrid))

48. Coreference Resolution The bombing was perpetrated by Al-Qaida Results: (perpetrator ?action ?agent) Constraints: (isa ?action Bombing) Bindings from Reading: ((?agent . AlQaida)) An attack occurred in Madrid Results: (eventOccursAt ?event ?location) Constraints: (isa ?event AttackOnObject) Bindings from Reading: ((?location . CityOfMadridSpain)) Remindings from Memory: ((?event . TerrorAttack-Sept8-2003-Madrid))

49. Coreference • Refer to a more general or specific type • “bombing” and “attack” • Consistent with being the same, have a known shared instance in memory • “doctor” and “father of four”

50. Tracking Ambiguity • Words are ambiguous • “Bush” • A shrubbery? • A president? Which one? • Sentences and Phrases • “take her flowers” • “Iraq borders Iran on the North.” • Intension • “Down?” • “Where is Baghdad?”