1 / 26

Boeing’s Language Understanding Engine (BLUE) and its performance on the STEP Shared Task

This article discusses Boeing's Language Understanding Engine BLUE and its performance on the STEP Shared Task, focusing on its semantic representation and structural reorganization capabilities. It also evaluates how well BLUE tackled anaphora and reference resolution.

arion
Download Presentation

Boeing’s Language Understanding Engine (BLUE) and its performance on the STEP Shared Task

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Boeing’s Language Understanding Engine (BLUE) and its performance on the STEP Shared Task Peter Clark Phil Harrison (Boeing Phantom Works)

  2. What is a Semantic Representation? “Leftist” “Rightist” Semantic structure close to syntactic structure, need more downstream interpretation Semantics more explicit. causes(virus01,c-cancer01) forall c isa(c,c-cancer) → exists e,v isa(e,event), isa(v,virus), involves(e,v), causes(e,c) (2.1) "Cervical cancer is caused by a virus." forall c isa(c,c-cancer) → exists v isa(v,virus), causes(v,c)

  3. Semantic Formalism • Ground (Skolemized) first-order logic assertions • No quantification (left for downstream processing) “An object is thrown from a cliff.” isa(object01,object_n1), isa(cliff01,cliff_n1), isa(throw01,throw_v1), object(throw01,object01), origin(throw01,cliff01). • Embedding used for modals (6.3) "The farm hopes to make a profit" isa(farm, farm_n1), ….. agent(hope01,farm01), object(hope01,[ agent(make02,farm01), object(make02,profit02)])

  4. BLUE’s Pipeline Throw object origin Object Cliff “An object is thrown from a cliff” Parser & LF Generator Initial Logic Generator Word Sense Disambiguation Semantic Role Labeling Coreference Resolution Metonymy Resolution Structural Reorganization Linguistic and World Knowledge isa(object01,object_n1), isa(cliff01,cliff_n1), isa(throw01,throw_v1), object(throw01,object01), origin(throw01,cliff01).

  5. 1. Parsing/Logical Form Generation • Parsing: SAPIR, a bottom-up chart parser, guided by • Hand-coded cost function • “tuples” – examples of attachment preferences (VPN "fall" "onto" "floor") (VPN "flow" "into" "body") (NN "lead" "iodide") (NPN "force" "on" "block") (NPN "speed" "at" "point") (NPN "chromosome" "in" "cell") (NPN "force" "during" *) (NPN "distance" "from" *) (NPN "distance" "to" *) • Several 100 authored by hand • 53 million generated automatically • Logical Form: Classical compositional rule-based approach

  6. 2. Initial logic generation isa(object01,object_n1), isa(cliff01,cliff_n1), isa(throw01,throw_v1), object(throw01,object01), origin(throw01,cliff01). Final Logic “An object is thrown from a cliff” (DECL ((VAR _X1 "an" "object") (VAR _X2 "a" "cliff")) (S (PRESENT) NIL "throw" _X1 (PP "from" _X2)) Logical form "object"(object01), "cliff"(cliff01), "throw"(throw01), sobject(throw01,object01), "from"(throw01,cliff01). Initial Logic

  7. 3. Subsequent Processing • Word Sense Disambiguation: • Naïve use of WordNet (70% ok) • For applications using CLib ontology, then do a mapping: CLib Ontology WordNet Physical-Object “object” Goal Concept (Word Sense) Lexical Term

  8. 3. Subsequent Processing Throw Throw sobject object “from” origin Object Object Cliff Cliff • Semantic Role Labeling: • ~40 semantic roles (from Univ Texas) • Assign using a hand-built database of (~100) rules IF X “of” Y and X is Physical-Object and Y is Material THEN X material Y • Moderate performance

  9. 3. Subsequent Processing • Coreference • Same name = same object • Same/subsuming noun phrase = same object • “The red block fell. The block that had fallen was damaged.” • Ordinal reference • “A block is on a block. The second block is larger.” • Metonymy resolution • Mainly with CLib KB “H2O reacts with NaCl” React React raw-material raw-material raw-material raw-material Chemical Chemical H2O NaCl basic-unit basic-unit H2O NaCl

  10. 3. Subsequent Processing • Structural Reorganization property Cliff Height “the height of the cliff” height Cliff Value Contain subject object “the box contains a block” Box Block content Box Block property kg 10 “10 kg” value Weight-Value (:pair 10 kg)

  11. 3. Subsequent Processing • Structural Reorganization “be” and “have” Be subject object (a) equality color Block Color Red “the color of the block is red” color Block Red (b) other relation Be subject object “the block is red” Block Red color Block Red

  12. What does BLUE do, and how did it fare?

  13. What does BLUE do, and how did it fare? “An object is thrown from a cliff.” object(throw01,object01), origin(throw01,cliff01).    (5.1) “I have a problem” has-part(i01,problem01). (4.1) “..training of dogs…” “of”(training01,dog1).  Predicate Argument Structure / Semantic Roles • Vocabulary of ~40 semantic relations (from UTexas) • Use small set of hand-coded SRL rules, e.g., • IF “of”(object,material) THEN material(object,material) • Occasionally will not make an assignment

  14. What does BLUE do, and how did it fare?  Attachment Correctness • (or structural completeness/correctness) • Use of “world knowledge” tuples to guide parsing • Frequently right, sometimes not • Occasional loss in the final logic • (5.6) "Initially it was suspected that…” “Initially” lost • (5.4) "When the gun crew was killed they were crouching unnaturally, which suggested that they knew that an explosion would happen.“ Parse but no logic out

  15. What does BLUE do, and how did it fare?  Anaphora and Reference Resolution • “an object…the object…” • Cross-POS, “falls…the fall…”, “high…the height…” • No pronoun resolution • “lacks it”→ object(lack01,it01) • Not across names “Greenslow…the farm…” • No propositional coreference • “Cancer is caused by a virus. That has been known…” • Unmentioned referent • “…a restaurant…the waiter…”      

  16. What does BLUE do, and how did it fare? Saint John  Word Sense Disambiguation • Naïve translation to WordNet (but 70% ok) (3.1) "John went into a restaurant.“ isa(John01, John_n1), agent(go01,John01), destination(go01,restaurant01).

  17. What does BLUE do, and how did it fare?  /  Quantification • Treated as ground facts • “every”, “some” in initial LF, but not final logic (“leftists” vs. “rightists”) (2.1) "Cervical cancer is caused by a virus." isa(cancer01,cancer_n1), isa(cervical01,cervical_a1), isa(virus01,virus_n1), causes(virus01,cancer01), mod(cancer01,cervical01).

  18. What does BLUE do, and how did it fare? / Negation, Modals, Conditionals, Disjunction • Negation • Sentence polarity marked • Modals • Embedded structures • Conditionals • Disjunction  (5.2) "I am not ready yet." sentence-polarity(sentence,negative), "be"(i02,ready01). (4.5) "We would like our school to work similarly..." agent(like01,we01). object(like01,[ agent(work01,school01), manner(work01,similarly01)])   

  19. What does BLUE do, and how did it fare?  (4.3) They visited places in England and France" object(visit01,place01), is-inside(place01,England01), is-inside(place01,France01). Conjunction: BLUE multiplies out coordinates  (3.4) "The atmosphere was warm and friendly" "be"(atmosphere01, warm01). "be"(atmosphere01, friendly01).

  20. What does BLUE do, and how did it fare? ? Tense and Aspect • Recorded in LF, not output in final logical form (4.5) "We would like our school to work similarly to the French ones..." ;;; Intermediate logical form (LF): (DECL ((VAR _X1 "we" NIL) (VAR _X2 "our" "school")) (S (WOULD) _X1 "like" _X2 (DECL ((VAR _X3 "the" (PLUR "one") (AN "French" "one") ….

  21. What does BLUE do, and how did it fare? • Collectives: use number-of-elements predicate (4.2) "seven people developed" isa(person01, person_n1), agent(develop01, person01), number-of-elements(person01,7).  ? Plurals • Recorded in LF, not output in final logical form (4.5) "We would like our school to work similarly to the French ones..." ;;; Intermediate logical form (LF): (DECL ((VAR _X1 "we" NIL) (VAR _X2 "our" "school")) (S (WOULD) _X1 "like" _X2 (DECL ((VAR _X3 "the" (PLUR "one") (AN "French" "one") ….

  22. What does BLUE do, and how did it fare?  Comparison Phrases • Will do “-er” comparatives (not in shared task) • Won’t do “similarly to” "The cost was greater than ten dollars" isa(cost01,cost_n1), value(quantity01,[10,dollar_n1]), greater-than(cost01,quantity01).  (4.5) "We would like our school to work similarly to the French ones..." agent(work01,school04), manner(work01,similarly01), destination(work01,one01), 

  23. What does BLUE do, and how did it fare? • But not all… (7.1) "the 1930s", (7.2) "the early 1970s", (7.4) "mid-'80s" (misparsed as an adjective), and (7.3) "the past 30 years"   Time • Tense and aspect recorded but not output • Some temporal prepositions and phrases recognized (4.4) "ensures operation until 1999" time-ends(ensure01, year1999). (5.3) "...yelled. Then the propellant exploded" next-event(yell01,explode01). (5.4) "When they were killed, they were crouching" time-at(crouch01,kill01). 

  24. What does BLUE do, and how did it fare? Questions  (1.4) "What is the duration?" query-for(duration01). Measurement Expressions  (1.1) "125 m" value(height01,[125,m_n1]). • 5 types recognized • True/false, find value, find identity, find subtypes, how many

  25. What does BLUE do, and how did it fare? Clarity (to a naïve reader) (3.1) "John went into a restaurant.“ isa(John01, John_n1), isa(restaurant01,restaurant_n1), isa(go01,go_v1), named(John01,["John"]), agent(go01,John01), destination(go01,restaurant01). (3.3) "The waiter took the order.“ isa(waiter01,waiter_n1), isa(order01,tell_v4), isa(take01,take_v1), agent(take01,waiter01), object(take01,order01).

  26. Summary BLUE: • Formalism: • Skolemized first-order logic predicates • Embedding for modals • Mechanism: • Parser → LF generator → initial logic → post-processing • Performance: • Many sentences are mostly right  • Most sentences have something wrong 

More Related