CPSC 503 Computational Linguistics

1. CPSC 503Computational Linguistics Lecture 11 Giuseppe Carenini CPSC503 Winter 2019

2. Plan for today • Finish syntax • Start semantics…. CPSC503 Winter 2019

3. Big Picture: Syntax & Parsing (2016…) NEURAL APPROCHES • The classifier is a simple MLP/FF neural network • The element of the stack / queue are vectors learned by RNNs • Bidirectional LSTM builds text span representations • FF to label each span with non-terminals • CKY for efficient global inference CPSC503 Winter 2019

4. Very recent paper (NAACL 2018) • Abstractly, our model consists of a single scoring function s(i, j, l) that assigns a real-valued score to every label for each span(i, j) in an input sentence. • We take the set of available labels to be the collection of all non-terminals … in the training data, • To build up to spans, we first run a bidirectional LSTM over the sequence of word representations for an input sentence • we implement the label scoring function by feeding the span representation through a one layerfeedforward network whose output dimensionality equals the number of possible labels • ….we can still employ a CKY-style algorithm for efficient globally optimal inference ….. • “We find that our model implicitly learns to encode much of the same information that was explicitly provided by grammars and lexicons in the past, indicating that this scaffolding can largely be subsumed by powerful general-purpose neural machinery • Also this one does (92.08 F1 on PTB) What’s Going On in Neural Constituency Parsers? An Analysis, D.Gaddy, M. Stern, D. Klein, Computer Science ., Univ. of California, Berkeley You know the terminology and the methods ! You should be able to understand most of this ! CPSC503 Winter 2019

5. Very recent paper (NAACL 2018) What’s Going On in Neural Constituency Parsers? An Analysis, D.Gaddy, M. Stern, D. Klein, Computer Science ., Univ. of California, Berkeley CPSC503 Winter 2019

6. Big Picture: Syntax & Parsing (2016…) NEURAL APPROCHES • The classifier is a simple MLP/FF neural network • The element of the stack / queue are vectors learned by RNNs CPSC503 Winter 2019

7. Transition-BasedDependencyParsing But parsing algorithm is the same….. DeterministicDep. Parsing(slightly simplified) The oracle/classifieris a simple MLP/FF neural network Parse(w1,...,wn) c←([w0]S,[w1,...,wn]Q,{}A) whileQc isnotempty t=o(c) c = t (c ) returnG=({w0,w1,...,wn},Ac) 1 2 3 4 5 l-arc, r-arc, shift NB:w0=ROOT • The classifier is a simple MLP/FF neural network • The element of the stack / queue are vectors learned by RNNs CPSC503 Winter 2019 SortingOutDependencyParsing 14(38)

8. Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature Representations TACL 2016 Note: buffer-> queue; considers3 words at the top of stack CPSC503 Winter 2019

9. Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature Representations TACL 2016 CPSC503 Winter 2019

10. Knowledge-Formalisms Map(including probabilistic formalisms) State Machines (and prob. versions) (Finite State Automata,Finite State Transducers, Markov Models) Neural Models, Neural Sequence Modeling Morphology Logical formalisms (First-Order Logics, Prob. Logics) Machine Learning Syntax Rule systems (and prob. versions) (e.g., (Prob.) Context-Free Grammars) Semantics Pragmatics Discourse and Dialogue AI planners (MDP Markov Decision Processes, Reinforcement learning) CPSC503 Winter 2019

11. Next three classes • What meaning is and how to represent it • Semantic Analysis: How to map sentences into their meaning • (syntax driven) Semantic Parsing • “Shallow” version: Semantic Role Labeling • Meaning of individual words (lexical semantics) • Computational Lexical Semantics Tasks • Word sense disambiguation • Word Similarity CPSC503 Winter 2019

12. Today Feb 9 • Semantics / Meaning /Meaning Representations • Linguistically relevant Concepts in FOPC/FOL • Semantic Analysis CPSC503 Winter 2019

13. Semantics Def.Semantics: The study of the meaning of words, intermediate constituents and sentences Def1.Meaning: a representation that links the linguistic input to knowledge of the world Def2. Meaning: a representation that expresses the linguistic input in terms of objects, actions, events, time, space… beliefs, attitudes...relationships Language independent CPSC503 Winter 2019

14. Semantic Relations involving Sentences ? Same truth conditions Paraphrase: have the same meaning • I gave the apple to John vs. I gave John the apple • I bought a car from you vs. you sold a car to me • The thief was chased by the police vs. …… • Entailment: “implication” • The park rangers killed the bear vs. The bear is dead • Nemo is a fish vs. Nemo is an animal Contradiction: I am in Vancouver vs. I am in India CPSC503 Winter 2019

15. Meaning Structure of Language • How does language convey meaning? • Grammaticization • Display a basic predicate-argument structure (e.g., verb complements) • Display a partially compositional semantics • Words CPSC503 Winter 2019

16. Words from Nonlexical categories • Obligation • Possibility • Definite, Specific • Indefinite, Non-specific • Disjunction • Negation • Conjunction • must • may • the • a • or • not • and Grammaticization Concept Affix • -ed • -s • re- • in-, un-, de- • Past • More than one • Again • Negation CPSC503 Winter 2019

17. Predicate-Argument Structure • Represent relationships among concepts • Some words act like arguments and some words act like predicates: • Nounsasconcepts or arguments: red(ball) • Adj, Adv, Verbs as predicates: red(ball) • Sub-categorization frames for verbs specify number, position, and syntactic category of arguments • Examples: give NP1 NP2, find NP, sneeze [] CPSC503 Winter 2019

18. Semantic (Thematic) Roles This can be extended to the realm of semantics • Semantic Roles: Participants in an event • Agent: George hit Bill. Bill was hit by George • Theme: George hit Bill. Bill was hit by George Source, Goal, Instrument, Force… Arguments in surface structure can be linked with their semantic roles • Mary gave/sent/read a book to Ming • Agent Theme Goal • Mary gave/sent/read Ming a book • Agent Goal Theme CPSC503 Winter 2019

19. Requirements for Meaning Representations? CPSC503 Winter 2019

20. First Order Predicate Calculus (FOPC) • FOPC provides sound computational basis for verifiability, inference, expressiveness… • Supports determination of truth • Supports Canonical Form • Supports question-answering (via variables) • Supports (tractable!) inference • Argument-Predicate structure • Supports compositionality of meaning CPSC503 Winter 2019

21. Common Meaning Representations I have a car FOPC Semantic Nets Frames Common foundation: structures composed of symbols that correspond to objects and relationships CPSC503 Winter 2019

22. Today Feb 9 • Semantics / Meaning /Meaning Representations • Linguistically relevant Concepts in FOPC/FOL • Semantic Analysis CPSC503 Winter 2019

23. Categories & Events • Categories: • VegetarianRestaurant (Joe’s) - relation vs. object • MostPopular(Joe’s,VegetarianRestaurant) • Events: can be described in NL with different numbers of arguments… • I ate • I ate a turkey sandwich • I ate a turkey sandwich at my desk • I ate at my desk • I ate lunch • I ate a turkey sandwich for lunch • I ate a turkey sandwich for lunch at my desk • ISA (Joe’s,VegetarianRestaurant) • AKO (VegetarianRestaurant,Restaurant) Reification CPSC503 Winter 2019

24. Reification Again “I ate a turkey sandwich for lunch” $w: Isa(w,Eating)Ù Eater(w,Speaker) Ù Eaten(w,TurkeySandwich) Ù MealEaten(w,Lunch) • Reification Advantage: • No need to specify fixed number of arguments to represent a given sentence in NL CPSC503 Winter 2019

25. On October 30, 1989, one civilian was killed in a reported FMLN attack in El Salvador. MUC-4 Example INCIDENT: DATE 30 OCT 89 INCIDENT: LOCATION EL SALVADOR INCIDENT: TYPE ATTACK INCIDENT: STAGE OF EXECUTION ACCOMPLISHED INCIDENT: INSTRUMENT ID INCIDENT: INSTRUMENT TYPEPERP: INCIDENT CATEGORY TERRORIST ACT PERP: INDIVIDUAL ID "TERRORIST" PERP: ORGANIZATION ID "THE FMLN" PERP: ORG. CONFIDENCE REPORTED: "THE FMLN" PHYS TGT: ID PHYS TGT: TYPEPHYS TGT: NUMBERPHYS TGT: FOREIGN NATIONPHYS TGT: EFFECT OF INCIDENTPHYS TGT: TOTAL NUMBERHUM TGT: NAMEHUM TGT: DESCRIPTION "1 CIVILIAN"HUM TGT: TYPE CIVILIAN: "1 CIVILIAN"HUM TGT: NUMBER 1: "1 CIVILIAN"HUM TGT: FOREIGN NATIONHUM TGT: EFFECT OF INCIDENT DEATH: "1 CIVILIAN"HUM TGT: TOTAL NUMBER CPSC503 Winter 2019

26. Representing Time (14.4.1) • Events are associated with points or intervals in time. • We can impose an ordering on distinct events using the notion of precedes. • Temporal logic notation: $e $t : Isa(e,Arriving) ÙTimeOf(e,t) • Constraints on variable tI arrived in New York$e $t : Isa(e,Arriving) ÙTimeOf(e,t) ÙAgent(e, Speaker) ÙDest(e,NewYork) Ù precedes(t,Now) CPSC503 Winter 2019

27. Interval Events • Needtstart and tend “She was driving to New York until now” • $ tstart,tend,e, i • ISA(e,Drive) Driver(e, She) • Dest(e,NewYork) Ù IntervalOf(e,i) • Endpoint(i, tend) Startpoint(i, tstart) • Precedes(tstart,Now) Ù • Equals(tend,Now) CPSC503 Winter 2019

28. Relation Between Tenses and Time Relation between verb tenses and points in time is not straightforward • Present tense used like future: • We fly from Baltimore to Boston at 10 • Complex tenses: • Flight 1902 arrived late • Flight 1902 had arrived late Representing them in the same way seems wrong…. CPSC503 Winter 2019

29. Reference Point • Reichenbach (1947) introduced notion of Reference point(R), separated out from Utterance time (U) and Event time (E) • Example: • When Mary's flight departed, I ate lunch • When Mary's flight departed, I had eaten lunch • Departure event specifies reference point. CPSC503 Winter 2019

30. CPSC503 Winter 2019

31. Language and time • Much more than how past, present, and futureare signaled by various English verb tenses • Languages have many other more direct and more specific ways to convey temporal information, including the use of a wide variety of temporal expressions… in the morning, somewhere around noon, throughout my life… • Course project ? (see project in 2016, Error Analysis of temporal tagger) CPSC503 Winter 2019

32. Today Feb 9 • Semantics / Meaning /Meaning Representations • Linguistically relevant Concepts in FOPC / FOL • Semantic Analysis (aka Semantic Parsing) CPSC503 Winter 2019

33. Map NL queries into FOPC so that answers can be effectively computed Practical Goal for (Syntax-driven) Semantic Analysis • What African countries are not on the Mediterranean Sea? • Was 2007 the first El Nino year after 2001? CPSC503 Winter 2019

34. Practical Goal for (Syntax-driven) Semantic Analysis Referring to physical objects - Executing instructions CPSC503 Winter 2019

35. Semantic Analysis I am going to SFU on Tue Sentence Meanings of grammatical structures The garbage truck just left Syntax-driven Semantic Analysis Meanings of words Literal Meaning I N F E R E N C E Common-Sense Domain knowledge Further Analysis Discourse Structure Intended meaning Context Shall we meet on Tue? CPSC503 Winter 2019 What time is it?

36. Compositional Analysis • Principle of Compositionality • The meaning of a whole is derived from the meanings of the parts • What parts? • The constituents of the syntactic parse of the input CPSC503 Winter 2019

37. Compositional Analysis: Example • AyCaramba serves meat CPSC503 Winter 2019

38. Abstractly Augmented Rules • Augment each syntactic CFG rule with a semantic formation rule • i.e., The semantics of A can be computed from some function applied to the semantics of its parts. • The class of actions performed by f will be quite restricted. CPSC503 Winter 2019

39. A FOL sentence with variables in it that are to be bound. Simple Extension of FOL: Lambda Forms • Lambda-reduction: variables are bound by treating the lambda form as a function with formal arguments CPSC503 Winter 2019

40. PropNoun -> AyCaramba MassNoun -> meat Attachments {AyCaramba} {MEAT} Augmented Rules: Example • Concrete entities assigning FOL constants • Simple non-terminals copying from daughters up to mothers. • Attachments • {PropNoun.sem} • {MassNoun.sem} • NP -> PropNoun • NP -> MassNoun CPSC503 Winter 2019

41. Verb -> serves {VP.sem(NP.sem)} {Verb.sem(NP.sem) Augmented Rules: Example Semantics attached to one daughter is applied to semantics of the other daughter(s). • S -> NP VP • VP -> Verb NP lambda-form CPSC503 Winter 2019

42. y y Example AC MEAT ……. AC MEAT • S -> NP VP • VP -> Verb NP • Verb -> serves • NP -> PropNoun • NP -> MassNoun • PropNoun -> AyCaramba • MassNoun -> meat • {VP.sem(NP.sem)} • {Verb.sem(NP.sem) • {PropNoun.sem} • {MassNoun.sem} • {AC} • {MEAT} CPSC503 Winter 2019

43. Semantic Parsing (via ML) CPSC503 Winter 2019

44. Semantic Parsing (via ML) CPSC503 Winter 2019

45. Semantic Parsing (via ML) CPSC503 Winter 2019

46. References (Project?) • Text Book: Representation and Inference for Natural Language : A First Course in Computational Semantics Patrick Blackburn and Johan Bos, 2005, CSLI • J. Bos (2011): A Survey of Computational Semantics: Representation, Inference and Knowledge in Wide-Coverage Text Understanding. Language and Linguistics Compass 5(6): 336–366. • Semantic parsing via Machine Learning: The Cornell Semantic Parsing Framework (Cornell SPF) is an open source research software package. It includes a semantic parsing algorithm, a flexible meaning representation language and learning algorithms. http://yoavartzi.com/ CPSC503 Winter 2019

47. Next Class • Computing with Word Senses: Word Sense Disambiguation WSD; WordNet; Ontologies (Lexical Semantics); Intro to Semantic Role Labeling (Chp. 18 3rd Ed. ) • Assignment-2 due Now ! CPSC503 Winter 2019

48. Just a sketch: to provide some context for some concepts / techniques discussed in 422 CPSC503 Winter 2019

49. Non-Compositionality • Unfortunately, there are lots of examples where the meaning of a constituent can’t be derived from the meanings of the parts • - metaphor, (e.g., corporation as person) • metonymy, (??) • idioms, • irony, • sarcasm, • indirect requests, etc CPSC503 Winter 2019

50. English Idioms • Lots of these… constructions where the meaning of the whole is either • Totally unrelated to the meanings of the parts (“kick the bucket”) • Related in some opaque way (“run the show”) • “buy the farm” • “bite the bullet” • “bury the hatchet” • etc… CPSC503 Winter 2019