Learning Semantic Parsers: An Important But Under-Studied Problem

1. Learning Semantic Parsers: An Important But Under-Studied Problem Raymond J. Mooney Dept. of Computer Sciences University of Texas at Austin "The fish trap exists because of the fish. Once you've gotten the fish you can forget the trap. The rabbit snare exists because of the rabbit. Once you've gotten the rabbit, you can forget the snare. Words exist because of meaning. Once you've gotten the meaning, you can forget the words. Where can I find a man who has forgotten words so I can talk with him?" -- The Writings of Chuang Tzu, 4th century B.C.

2. Natural Language Learning • Most computational research in natural-language learning has addressed “low-level” syntactic processing. • Morphology (e.g. past-tense generation) • Part-of-speech tagging • Shallow syntactic parsing • Syntactic parsing

3. Semantic Language Learning • Learning for semantic analysis has been restricted to relatively small, isolated tasks. • Word sense disambiguation (e.g. SENSEVAL) • Semantic role assignment (determining agent, patient, instrument, etc., e.g. FrameNet) • Information extraction

4. Cognitive Modeling • Most computational research on natural-language learning is focused on engineering performance on large corpora. • Very little attention is paid to modeling human-language acquisition.

5. Language Learning Training Data • Most computational language-learning systems assume detailed, supervised training data that is unavailable to children acquiring language. • Sentences paired with phonetic transcriptions • Words paired with their past tense. • Sentences with a POS tag for each word • Sentences paired with parse trees (treebank) • Sentences with words tagged with proper senses • Sentences with semantic roles tagged • Documents tagged with named entities and semantic relations.

6. Semantic Parsing • A semantic parser maps a natural-language sentence to a complete, detailed semantic representation (logical form). • Semantic Parser Acquisition attempts to automatically induce such parsers from corpora of sentences each paired only with a semantic representation. • Assuming a child can infer the likely meaning of an utterance from context, this is more cognitively plausible training data.

7. CHILL(Zelle & Mooney, 1992-96) • Semantic parser acquisition system using Inductive Logic Programming (ILP) to induce a parser written in Prolog. • Starts with a parsing “shell” written in Prolog and learns to control the operators of this parser to produce the given I/O pairs. • Requires a semantic lexicon, which for each word gives one or more possible semantic representations. • Parser must disambiguate words, introduce proper semantic representations for each, and then put them together in the right way to produce a proper representation of the sentence.

8. CHILL Example • U.S. Geographical database • Sample training pair • What is the capital of the state with the highest population? • answer(C, (capital(S,C), largest(P, (state(S), population(S,P))))) • Sample semantic lexicon • what: answer(_,_) • capital: capital(_,_) • state: state(_) • highest: largest(_,_) • population: population(_,_)

9. WOLFIE(Thompson & Mooney, 1995-1999) • Learns a semantic lexicon for CHILL from the same corpus of semantically annotated sentences. • Determines hypotheses for word meanings by finding largest isomorphic common subgraphs shared by meanings of sentences in which the word appears. • Uses a greedy-covering style algorithm to learn a small lexicon sufficient to allow compositional construction of the correct representation from the words in a sentence.

10. WOLFIE Lexicon Learner Semantic Lexicon Logical Form CHILL Parser Learner Natural Language Semantic Parser WOLFIE + CHILLSemantic Parser Acquisition NLLF Training Exs

11. U.S. Geography Corpus • Queries for database of about 800 facts on U.S. geography. • Collected 250 sample questions from undergraduate students in a German class. • Questions annotated with correct logical form in Prolog. • Questions also translated into Spanish, Turkish, and Japanese.

12. Experimental Evaluation • 10-fold cross-validation over sentences in the corpus. • Generate learning curves by training on increasing fractions of the total training set. • Test accuracy by determining if, when executed in Prolog, the generated logical query generates the same answer from the database as the correct logical query. • Compared performance to manually-written NL interface called Geobase. • Compared performance with manually-written vs. learned semantic lexicon.

13. CHILL + WOLFIE Learning Curves

14. Cognitively Plausible Aspects of CHILL/WOLFIE • Deterministic parsing that processes sentence one word at a time. • Evidence from garden path sentences • Dynamically integrates syntactic and semantic cues during parsing. • Evidence from real-time studies of language comprehension (Tanenhaus et al.) • Learns from plausible input of sentences paired only with semantic form.

15. Cognitively Implausible Aspectsof CHILL/WOLFIE • Batch training on complete corpora. • Incremental training that processes one sentence at a time (Siskind, 1996). • Lexicon learning is disconnected from parser learning. • Assumes each sentence is annotated with a single, correct semantic form.

16. Interactions between Syntax and Lexicon Acquisition • Syntactic Bootstrapping allows children to use verb syntax to help acquire verb meanings (Gleitman, 1990) • “Big Bird and Elmo are gorping.” • “Big Bird is gorping Elmo”

17. Contextually Ambiguous Sentence Meaning • Sentences are uttered in complex situations composed of numerous potential meanings. • Could assume each sentence is annotated with multiple possible meanings inferred from context (Siskind, 1996). • Multiple instance learning (Dietterich et al., 1997) • Assuming context meaning is represented as a semantic network, sentence meaning could be assumed to be any connected subgraph of the context.

18. Sample Ambiguous Context “Juvenile caresses canine.” HasColor attr Dog obj Black isa HasColor patient Spot attr Petting obj Possess agent Blonde agent patient Chewing agent isa Hair patient Mary Thing2 Barbie obj part isa isa Thing1 HasPart Child Doll isa Bone

19. Sample Ambiguous Context “Juvenile caresses canine.” HasColor attr Dog obj Black isa HasColor patient Spot attr Petting obj Possess agent Blonde agent patient Chewing agent isa Hair patient Mary Thing2 Barbie obj part isa isa Thing1 HasPart Child Doll isa Bone

20. Sample Ambiguous Context “Juvenile caresses canine.” HasColor attr Dog obj Black isa HasColor patient Spot attr Petting obj Possess agent Blonde agent patient Chewing agent isa Hair patient Mary Thing2 Barbie obj part isa isa Thing1 HasPart Child Doll isa Bone

21. Sample Ambiguous Context “Juvenile caresses canine.” HasColor attr Dog obj Black isa HasColor patient Spot attr Petting obj Possess agent Blonde agent patient Chewing agent isa Hair patient Mary Thing2 Barbie obj part isa isa Thing1 HasPart Child Doll isa Bone

22. Sample Ambiguous Context “Juvenile caresses canine.” HasColor attr Dog obj Black isa HasColor patient Spot attr Petting obj Possess agent Blonde agent patient Chewing agent isa Hair patient Mary Thing2 Barbie obj part isa isa Thing1 HasPart Child Doll isa Bone

23. Sample Ambiguous Context “Juvenile caresses canine.” HasColor attr Dog obj Black isa HasColor patient Spot attr Petting obj Possess agent Blonde agent patient Chewing agent isa Hair patient Mary Thing2 Barbie obj part isa isa Thing1 HasPart Child Doll isa Bone

24. Sample Ambiguous Context “Juvenile caresses canine.” HasColor attr Dog obj Black isa HasColor patient Spot attr Petting obj Possess agent Blonde agent patient Chewing agent isa Hair patient Mary Thing2 Barbie obj part isa isa Thing1 HasPart Child Doll isa Bone

25. Sample Ambiguous Context “Juvenile caresses canine.” HasColor attr Dog obj Black isa HasColor patient Spot attr Petting obj Possess agent Blonde agent patient Chewing agent isa Hair patient Mary Thing2 Barbie obj part isa isa Thing1 HasPart Child Doll isa Bone

26. Sample Ambiguous Context “Juvenile caresses canine.” HasColor attr Dog obj Black isa HasColor patient Spot attr Petting obj Possess agent Blonde agent patient Chewing agent isa Hair patient Mary Thing2 Barbie obj part isa isa Thing1 HasPart Child Doll isa Bone

27. Issues in Engineering Motivation • Most computational language-learning research strives for broad coverage while sacrificing depth. • “Scaling up by dumbing down” • Realistic semantic parsing currently entails domain dependence. • Domain-dependent natural-language interfaces have a large potential market. • Learning makes developing specific applications more tractable. • Training corpora can be easily developed by tagging existing corpora of formal statements with natural-language glosses.

28. Robocup Coach Competition • Simulated robot soccer competition. • Coachable teams can take advice on how to play the game. • Coaching instructions are provided in a formal language called CLANG.

29. Project on Interactive Learning from Language Advice and Reinforcements Broadening the Communication Channel Between Machine Learners and their Human Teachers Raymond J. Mooney, Jude Shavlik Rich Maclin, Peter Stone, Risto Miikkulainen DARPA Machine Learning Seedling Project

30. CLANG Corpus • We collected 500 examples of CLANG statements written by humans for the Robocup Coach Competition. • Each statement was annotated with a synonymous English sentence.

31. Samples from English/CLANG Corpus • If player 4 has the ball, it should pass the ball to player 2 or 10. ((bowner our {4}) (do our {4} (pass {2 10})))) • No one pass to the goalie. ((bowner our {0}) (dont our {0} (pass {1})))) • If the ball is in the left upper half of the field, and player 2 is within distance 5 of the ball, it should intercept the ball. ((and (bpos (rec (pt -52.50 -34.00) (pt 0.00 34.00))) (ppos our { 2 } 1 11 (arc (pt ball) 0.00 5.00 0.00 360.00))) (do our { 2 } (intercept)))) • If players 9, 10 or 11 have the ball, they should shoot and should not pass to players 2-8. ((bowner our {9 10 11}) (do our {9 10 11} (shoot)) (dont our {9 10 11} (pass {2 3 4 5 6 7 8}))))

32. New Approaches to Semantic Parsing • Directly mapping NL sentences to logical form using string-to-tree transduction rules. • Mapping NL syntactic parse trees to logical form using tree-to-tree transduction rules. • Integrated syntactic/semantic parsing using syntactic and semantic knowledge-bases.

33. RULE CONDITION DIRECTIVE bowner our do our UNUM_SET UNUM_SET UNUM_SET ACTION pass 2 2 10 String-to-Tree Transduction • This approach exploits the CLANG grammar but not an English grammar. • Sample production rules of CLANG grammar: ACTION (pass UNUM_SET) CONDITION (bowner our UNUM_SET) CONDITION (play_m PLAY_MODE) • Each CLANG Statement can be unambiguously parsed ((bowner our{2}) (do our {2} (pass {10})))

34. Transduction Rules • Rules convert sub-strings of natural language sentences into CLANG production instances in the resulting parse tree. “ player N has ball” => CONDITION (bowner our {N}) “the ball is with player N” => CONDITION (bowner our {N}) • Replace the matched sub-strings by the non-terminals and apply rules for higher level CLANG production rules “If CONDITION then DIRECTIVE” => RULE (CONDITION DIRECTIVE)

35. CONDITION (bowner our{2}) Example Sentence: If player 2 has the ball, player 2 should pass the ball to player 10. CLANG representation:((bowner our{2}) (do our {2} (pass {10}))) Rule: player N has [1] ball => CONDITION (bowner our {N}) If , player 2 should pass the ball to player 10. player 2 has the ball player 2 has the ball

36. ACTION CONDITION (bowner our{2}) (pass {10}) Example contd. Sentence: If CONDITION, player 2 should pass the ball to player 10. Rule: pass [1] ball to player N => ACTION(pass {N}) If , player 2 should . pass the ball to player 10 pass the ball to player 10

37. ACTION ACTION CONDITION DIRECTIVE DIRECTIVE (bowner our{2}) (pass {10}) (pass {10}) (do our {2} (pass {10})) (do our {2} (pass {10})) Example contd. Sentence: If CONDITION, player 2 should ACTION. Rule: player N should ACTION => DIRECTIVE(do our {N} ACTION) If , . player 2 should player 2 should

38. RULE CONDITION CONDITION ((bowner our {2}) (do our {2} (pass {10}))) DIRECTIVE DIRECTIVE (bowner our{2}) (bowner our{2}) (do our {2} (pass {10})) (do our {2} (pass {10})) Example contd. Sentence: If CONDITION, DIRECTIVE. Rule: If CONDITION [1] DIRECTIVE . => RULE(CONDITION DIRECTIVE) If , . If , .

39. Experiment with Manually-built Rules • To test the feasibility of this approach, rules were manually written to cover 40 examples. • When tested on previously unseen 63 examples, the rules covered 18 examples completely and the remaining examples partially. • Good indication that the approach can work if manually-built rules can be automatically learned.

40. Learning Transduction Rules • Parse all the CLANG examples • For every production rule in the CLANG grammar: • Call those sentences positiveswhose CLANG representation’s parses has that production rule • Call the remaining sentences negatives • Learn rules using Information Extraction system ELCS (Extraction using Longest Common Subsequences) which we developed for extracting protein-protein interactions from biological text • Given examples of positive and negative sentences, ELCS repeatedly generalizes positive sentences to form rules until the rules become overly general and start matching negative examples

41. Generalization Method: Longest Common Subsequence Whenever the ball is in REGION player 6 should be positioned at REGION . If the ball is in the near quarter of the field , player 2 should position itself at REGION . the [0] ball [0] is [0] in [7] player [1] should [2] at [0] REGION

42. The ball is in REGION , our player 7 is in REGION and no opponent is around our player 7 within 1.5 distance. If the ball is in REGION and not in REGION then player 3 should intercept the ball. During normal play if the ball is in the REGION then player 7 , 9 and 11 should dribble the ball to the REGION . If our team has the ball and the ball is in REGION , then it should be passed to REGION . If the ball is in REGION then position player 7 at REGION . When the play mode is normal and the ball is in the REGION then our player 2 should pass the ball to the REGION . All players except the goalie should pass the ball to RP12 if it is in RP18. If the ball is inside rectangle ( -54 , -36 , 0 , 36 ) then player 10 should position itself at REGION with a ball attraction of REGION . Player 2 should pass the ball to REGION if it is in REGION . If our player 6 has the ball then he should take a shot on goal. If player 4 has the ball , it should pass the ball to player 2 or 10. If the condition DR5C3 is true , then player 2 , 3 , 7 and 8 should pass the ball to player 3. If "DR6C11" , player 10 , 3 , 4 , or 5 should the ball to player 11. if DR1C7 then players 10 , 3 , 4 and 5 should pass to player 5. During play on , if players 6 , 7 or 8 is in REGION , they should pass the ball to players 9 , 10 or 11. If "Clear_Condition" , players 2 , 3 , 7 or 5 should clear the ball REGION . If it is before the kick off , after our goal or after the opponent's goal , position player 3 at REGION . If the condition MDR4C9 is met , then players 4-6 should pass the ball to player 9. If Pass_11 then player 11 should pass to player 9 and no one else. ball [0] is [2] REGION => the [0] ball [0] in [0] REGION => CONDITION (bpos REGION) CONDITION (bpos REGION) Example of Learned Rules CONDITION (bpos REGION) positives negatives ELCS ball [0] is [2] REGION the [0] ball [0] in [0] REGION

43. Resources Required for Progressin Semantic Parser Acquisition • More corpora of sentences annotated with logical form. • More researchers studying the problem. • More algorithms for addressing the problem. • More ideas, constraints, and methods from psycholinguistics that can be exploited. • More psycholinguistic issues that can be explored.