AQUAINT Phase 2 Advanced Techniques for Multimodal Question Answering

1. AQUAINT Phase 2Advanced Techniques for Multimodal Question Answering Language Computer Corporation www.languagecomputer.com Richardson, TX Dan Moldovan, Ph.D.

2. Phase2 Tasks Task 1 Inference – Based Question Answering Task 2 Applying Semantic Relations to Answer Judgment Questions Task 3 Multimedia Question Answering

4. COGEX QLF Q Logic Forms Axiom Building Logic Prover Answer Ranking ALF Text Ranked Answer Axioms XWN Lex ch Relaxation Linguistic Axioms

5. Semantically Enhanced COGEX Semantic Calculus Logic Selector Q QLF Contexts Semantic Parser Logic Forms Axiom Building Logic Prover Answer Ranking Text ALF Ranked Answer Axioms XWN Lex ch World K Axioms Relaxation Linguistic Axioms

6. LCC’s List of Semantic Relations

7. Polaris Semantic Parser Bin Laden reportedly purchased anthrax a half decade ago from a supplier in North Korea. Human-generated relations AGENT(Bin Laden, purchased) THEME(anthrax, purchased) RECEIVE-FROM(a supplier in North Korea, purchased) MANNER(reportedly, purchased) TEMPORAL(a half decade ago, purchased) MEASURE(a half, decade) SOURCE-FROM(from a supplier in North Korea, anthrax) LOCATION(in North Korea, a supplier) System output AGENT(purchased, Bin Laden) THEME(purchased, anthrax) LOCATION(purchased, from a supplier in North Korea) TEMPORAL(purchased, a half decade ago) PROPERTY(half, decade) LOCATION(a supplier, in North Korea)

8. Polaris Semantic Parser An Israeli helicopter fired missiles at a car in Gaza City on Wednesday, killing two senior Hamas militants and five other people, witnesses and doctors said. Human-generated relations POSSESSION(helicopter, Israeli) AGENT(An Israeli helicopter, fired) THEME(missiles, fired) LOCATION(at a car, fired) LOCATION(in Gaza City, fired) LOCATION(in Gaza City, a car) TEMPORAL(on Wednesday, fired) CAUSE(fired, killing) THEME(two senior Hamas militants, killing) MEASURE(two, senior Hamas militants) PROPERTY(senior, Hamas militants) IS-A(Hamas militants, militants) PART-WHOLE(militants, Hamas) THEME(five other people, killing) MEASURE(five, other people) AGENT(witness, said) AGENT(doctor, said) TOPIC(said, An Israeli helicopter fired missiles at a car in Gaza City on Wednesday , killing two senior Hamas militants and five other people) System output PART-WHOLE(helicopter, Israeli) AGENT(fired, An Israeli helicopter) THEME(fired, missiles) LOCATION(fired, at a car in Gaza City) TEMPORAL(fired, on Wednesday) THEME(killing, two senior Hamas militants and five other people) PROPERTY(senior, militants) PROPERTY(other, people) THEME(said, An Israeli helicopter fired missiles at a car in Gaza City on Wednesday , killing two senior Hamas militants and five other people)

9. Applying Semantic Relations to Answer Judgment Questions Approach: Layered Semantics Text Word Senses Learning Semantic Relations Learning, SR Calculus, Lex Chains High Level Semantics ? Text Understanding WSD Question Complexity JQ JQ JQ

10. SR Calculus Goal : Given Semantic Relations Ri,...,Rk and a function f that combines Ri,...,Rk determine the semantics of the new relation R = f (R i,…, Rk)

11. 1 IS-A ENTAIL 2 1 ENTAIL 3 IS-A 3 2 1 3 IS-A ENTAIL IS-A 2 2 2 2 3 1 3 ENTAIL ENTAIL IS-A 2 IS-A ENTAIL 3 3 2 2 1 3 IS-A IS-A 1 ENTAIL ENTAIL 3 IS-A IS-A 3 1 1 ENTAIL 2 3 1 3 2 ENTAIL 1 3 IS-A IS-A 1 2 IS-A IS-A ENTAIL 1 2 IS-A 1 2 3 ENTAIL ENTAIL ENTAIL ENTAIL 3 1 1 2 2 Sixteen Possible Pairs of IS-A, Entail, and their reverses IS-A ENTAIL R_IS-A R_ENTAIL IS-A ENTAIL 1 IS-A R_IS-A ENTAIL R_ENTAIL

12. SR Calculus IS-A ENTAIL R_IS-A R_ENT IS-A ENTAIL R_IS-A R_ENT

13. SR Calculus • CAUSE(x,y) & CAUSE(y,z)  CAUSE(x,z) • ACCOMPANIMENT(x,y)  ACCOMPANIMENT(y,x) • ACCOMPANIMENT(x,y) & LOCATION(y,z)  LOCATION(x,z) • INFLUENCE(x,y) & CAUSE(y,z)  INFLUENCE(x,z) • ISA(x,y) & PURPOSE(y,z)  PURPOSE(x,z) • LOCATION(x,y) & MAKEPRODUCE(y,z)  PURPOSE(x,z)

14. SR Calculus An example of Rule 4 • The criminal apologized. • He confessed his crime. In WordNet Entail admit apologize IS-A confess Q. Explain how did the criminal apologize ?

15. New High Level Semantic Relation based on Rule 4 Explains Confess apologize. Q. Explain how did the criminal apologize ? A. He confessed SR Calculus An example of Rule 4 • The Criminal apologized. • He confessed his crime. In WordNet Entail admit apologize IS-A explain confess

16. SR Calculus - Definitions and Operations • R-1i = the reverse of Ri If A R i B then B R-1i A • Ri o Rj the composition of Riand Rj If A Ri B and B Rj C then Ri o Rjis the semantic relation that holds between A and C • Ri⊳ R j Ri dominates Rjif Ri = Ri o Rj and Ri = Rj o Ri Left dominanceRight dominance Ri ⊳L R j if R i = Ri o Rj Ri ⊳R Rj if R i = Rj o Ri

17. SR Calculus Definitions and Operations • Ri = Rj ＵRk iff A Ri B  A Rj B or A Rk B • Ri = Rj Ո Rk iff A Ri B  A Rj B and A Rk B • ⊥denotes OTHER semantic relations for any concepts A and B If A ⊥ B then ¬Ǝ Ri such that A Ri B • Ri is symmetric iff Ri = R-1i • Ri is transitive iff Ri o Ri = Ri

18. Properties of the Operations 1(R¡ -1) -1 = Ri 2 Ri o R i = (R i-1 o R i -1) -1 3 Ri o R j = (Rj -1 o R i -1) -1 4 Ri o Rj -1 = (Rj o R i -1) -1 5 R i -1 o R j = (Rj -1 o R i ) -1 6 Ri -1 o R j -1 = (Rj o R i ) -1 7 R i -1inherits all the properties of R I 8 ⊥-¹ = ⊥ 9 R іо⊥ = ⊥о R і = ⊥(i.e. OTHERS dominates every semantic relation)

19. Properties of the Operations 10 RіU⊥ = ⊥U R і= ⊥ 11 By definition, the semantic union operation is commutative 12if R iis symmetric, then R i o R iis symmetric 13 if R i is symmetric, then R i o R j = R i -1 o R j 14 if R i is symmetric, then R j o R i = R i-1 15 R i ⊳LR j ⇒ R I-1 ⊳R R j -1 16 R i ⊳ RR¡ ⇒ Rі-1⊳ L R¡-1 17 Thus, R i ⊳Rj⇒ Rj-1 ⊳ Rj-1 18if R iis symmetric and R i ⊳R j, then R i ⊳ R j -1, and 19if R j is symmetric and R I ⊳ R j, then R i -1 ⊳ R j

20. High Level Semantics Goal: Given a set of HLSR, develop methods to automatically identify them in: • Questions • Free Text Approach: • Decompose HLSR into simpler relations using SR Calculus • Improve Lexical Chains finder • Machine Learning (annotations).

21. High Level Semantics LCC’s focus: Goal; intent, desire, volition, ambition, hope Implication; prediction, estimation, result, conclusion Belief; expectation, conviction, opinion, impression Relation; compare, contrast Justification; motive, reason, explanation Causality; cause, antecedent, determinant, Meaning; significance, understanding, interpretation

22. Intentions Approach to discover HLSR. Step 1.Identify patterns that express a HLSR. Looked at 2700 Semcore sentences and found 46 intentions. Step 2.Identify and extract features Formulated 6 syntactic and semantic features Pattern Example Frequency

23. Intention Step 3. Annotate data Looked at 9631 sentences; found 115 positive examples, and selected 258 negatives. Step 4.Use Machine Learning Run Support Vector Machines with radial kernel Results: Recognized intentions with 90.4% accuracy Intentions Non-Intentions Total Training (%) 92 20.8 300 30.66 69.33 Testing (%) 23 50 73 31.50 68.49 Total 115 258 373

24. Intention Example Question: What was Putin trying to achieve by increasing military cooperation with North Korea? Question Relations: AGENT(Putin, trying) -> AGENT(Putin, intend) -> INTENTION(x,Putin) MEANS (increasing military cooperation with North Korea, achieve) LOCATION (North Korea, military cooperation) Question Paraphrase: What was Putin’s intention by (or through the means of) increasing military cooperation with North Korea?

25. Intention Example Answer: Putin is attempting to restore Russia’s influence in the East Asian region. The report said , “the possibility remains that Russia could increase military cooperation with North Korea based on their treaty. Relations: • PART-WHOLE (Putin, Russia) & AGENT (Russia, increase)-> AGENT (Putin, increase) • AGENT (Putin, is attempting) -> AGENT (Putin, try) -> INTENT (restore Russia’s influence in the East Asian region, Putin) • AGENT (Putin, restore) • AGENT (Putin, attempting) • LOCATION (East Asia, North Korea) • LOCATION (North Korea, military cooperation) Answer Paraphrase: Putin’s intent is to restore Russia’s influence in the East Asian region [North Korea]. The report said, “the possibility remains that Russia [Putin] could increase military cooperation with North Korea based on their treaty.

26. Intention Example INTENTION (to restore Russia’s influence in the East Asian Region, Putin) * Perfect Match ** Concept Match

27. An Example of Context Al-Qaeda is an international terrorist network founded and led by Osama bin Laden in the late 1980s. CTemporal1 [ Al-Qaeda is an international terrorist network founded and led by Osama bin Laden ] Al-Qaeda_NN(x1) & international_JJ(x2) & terrorist_JJ(x2) & network_NN(x2) & found_VB(e1,x6,x2) & and_CC(e3,e1,e2) & lead_VB(e2,x6,x2) & by_IN(e2,x6) & Osama_NN(x3) & bin_NN(x4) & Laden_NN(x5) & nn_NNC(x6,x3,x4,x5) & in_IN(x6,x7) & late_JJ(x7) & 1980s_NN(x7) & CTemporal1(e1,e2)

28. An Example of Context After graduation from King Abdul-Aziz University in Jeddah , bin Laden left Saudi Arabia for Afghanistan to join the Mujahadeen in the fight against the Soviet occupation CTemporal2 [ bin Laden left Saudi Arabia for Afghanistan to join the Mujahadeen in the fight against the Soviet occupation ] After_IN(e1,x1) & graduation_NN(x1) & from_IN(x1,x5) & King_NN(x2) & Abdul-Aziz_NN(x3) & University_NN(x4) & nn_NNC(x5,x2,x3,x4) & in_IN(x5,x6) & Jeddah_NN(x6) & bin_NN(x7) & Laden_NN(x8) & nn_NNC(x9,x7,x8) & leave_VB(e1,x9,x12) & Saudi_NN(x10) & Arabia_NN(x11) & nn_NNC(x12,x10,x11) & for_IN(e1,x13) & Afghanistan_NN(x13) & to_TO(e1,e2) & join_VB(e2,x9,x14) & Mujahadeen_NN(x14) & in_IN(e2,x15) & fight_NN(x15) & against_IN(x15,x16) & Soviet_JJ(x16) & occupation_NN(x16) & CTemporal2(e1,e2)

29. An Example of Context In early 1980 's the US media called the Mujahadeen freedom fighters, not terrorists. COpinion1 [ Mujahadeen freedom fighters , not terrorists ] CTemporal3 [ media called the Mujahadeen freedom fighters, not terrorists [ In_IN(e1,x1) & early_JJ(x1) & 1980_CD(x1) & US_NN(x2) & media_NN(x3) & nn_NNC(x4,x2,x3) & call_VB(e2,x4,x5) & Mujahadeen_NN(x5) & freedom_NN(x6) & fighter_NN(x7) & nn_NNC(x5,x6,x7) & -( terrorist_NN(x5) ) & COpinion1(e2) & CTemporal3(e2)

30. Putting it All Together The path to robust reasoning on textual knowledge • Logic Forms • Semantic Relations • SR Calculus • High Level Semantics • Contexts Modal Epistemic Logic

31. Modal Epistemic Logic • Reasoning about the knowledge of n agents • Modal operators K₁, K₂,…,Knone per agent. • KіØ means that agent i knows (or believes) formula Ø • Lkn the language consisting of all formulas for all agents. Example K₁ K₂ p ٨ ¬ K₂ K₁ K₂ p Agent 1 knows that agent 2 knows p but agent 2 does not know that agent 1 knows that agent 2 knows p. Example “CIA doesn’t know whether bin Laden is in Afghanistan.” ¬ Kc¬ p ٨ ¬ K c¬ (¬p) where p= bin Laden is in Afghanistan; and Kc = CIA knows

32. Epistemic Structure (Kripke Structure) M= (W, K₁, K₂,…,Kn, ∏) where W is a set of possible worlds. K i is a binary relation on W called the possibility relation or accessibility relation. It is a subset of W x W (w,w') ∈ Ki if agent i considers w'a possible world in world w. ∏- is an interpretation, a function that associates truth assignments to primitive propositions p in each world w ∏ (w) (p) ∈ { true, false} In modal logic the truth of a formula depends on the world. Ex: Clinton is president in 1995, but not in 2002.

33. Properties of Knowledge K1. (Ki p٨Ki(p⇒ q)) ⇒ Ki q (Distribution axiom) K2.Ki p ⇒p (Knowledge axiom). K3.¬Ki false (Consistency axiom). K4.K i p ⇒K i Ki p (Positive Introspection axiom). K5. ¬ Ki p ⇒Ki¬ Ki p (Negative Introspection axiom) MP. From p and p ⇒ q infer q (Modus Ponens). Gen. From p infer Ki p (Knowledge Generalization).

34. An Example of Epistemic Semantics Al-Qaeda is an international terrorist network founded and let by Osama bin Laden in the late 1980’s. After graduation from Kin Abdul-Aziz University of Jedday, bin Laden left Saudi Arabia for Afghanistan to join the Mujahadeen in the fight against the Soviet occupation. In early 1980’s the US media called the Mujahadeen freedom fighters, not terrorists. Q: Is bin Laden a terrorist?

35. An Example of Epistemic Semantics K1 [Al-Qaeda is an international terrorist network founded and let by Osama bin Laden in the late 1980’s.] K2[After graduation from Kin Abdul-Aziz University of Jedday, bin Laden left Saudi Arabia for Afghanistan to join the Mujahadeen in the fight against the Soviet occupation. In early 1980’s the US media called the Mujahadeenfreedom fighters, not terrorists.] Q: Is bin Laden a terrorist? K1 Al-Queda ⇒ terrorist K2 Mujahadeen ⇒ freedom - fighter Al-Queda (bin Laden) freedom- fighter = ¬ terrorist -------------------------------- Mujahadeen (bin Laden) terrorist ( bin Laden) ------------------------------------------- ¬ terrorist (bin Laden There are two interpretations for bin Laden 1. bin Laden is terrorist 2. bin Laden is not a terrorist

36. Thank You