Natural Language Generation with Tree Conditional Random Fields

Natural Language Generation with Tree Conditional Random Fields Wei Lu, Hwee Tou Ng, Wee Sun Lee Singapore-MIT Alliance National University of Singapore

Natural Language Generation • Generating Natural Language (NL) paraphrases for Meaning Representations (MR) How many states do not have rivers ? … … … … … … … Natural Language Sentence Meaning Representation Lu, Ng & Lee

Meaning Representation (MR) QUERY:answer(NUM) NUM:count(STATE) STATE:exclude(STATE STATE) STATE:state(all) STATE:loc_1(RIVER) RIVER:river(all) how many states do not have rivers ? Lu, Ng & Lee

Previous Work • Chart Generation for Surface Realization • Head-Driven Phrase Structure Grammar (HPSG) (Carroll et al., 1999; Carroll and Oepen, 2005; Nakanishi et al., 2005) • Combinatory Categorial Grammar (CCG) (White and Baldridge, 2003; White, 2004). • WASP-1++ by Wong and Mooney (2007) • View the problem as a statistical machine translation task • Inversion of a semantic parser called WASP, with incorporation of models borrowed from PHARAOH Lu, Ng & Lee

Hybrid Tree Framework • Aims to bridge natural language sentences and their underlying meaning representations • On top of the framework, we built a generative model that jointly generates both natural language and MR tree • Details presented in our EMNLP 2008 paper for semantic parsing • Wei Lu, HweeTou Ng, Wee Sun Lee, and Luke S. Zettlemoyer. 2008. A Generative Model for Parsing Natural Language to Meaning Representations. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing (EMNLP 2008), pages 783–792. Lu, Ng & Lee

Hybrid Tree NL-MR Pair QUERY:answer(NUM) NUM:count(STATE) STATE:exclude(STATE STATE) Hybrid sequences STATE:state(all) STATE:loc_1(RIVER) RIVER:river(all) how many states do not have rivers ? Lu, Ng & Lee

The Joint Generative Model • Assume the MR tree and NL sentence are jointly generated with a top-down recursive Markov process • Able to handle re-ordering of nodes (MR productions) in MR tree during generation process • Generation process results in a hybrid tree • Shown effective in semantic parsing Lu, Ng & Lee

NLG with Hybrid Trees • The most probable NL w for a given MR m is: 1. Find the most probable hybrid tree T: • T* = argmaxT p(T|m) 2. The most probable NL sentence w is the yield of the hybrid tree T*: • w* = yield(T*) • Different assumptions can be made for finding the most probable hybrid tree T • Two models: • Baseline: Direct Inversion Model • Tree Conditional Random Field Model Lu, Ng & Lee

Direct Inversion Model QUERY:answer(NUM) NUM:count(STATE) NUM ? STATE STATE:exclude(STATE STATE) how many STATE1 STATE:state(all) do not STATE:loc_1(RIVER) STATE2 states RIVER:river(all) have RIVER rivers Lu, Ng & Lee

Direct Inversion Model • Direct inversion of the semantic parser • The distance dβ(w1,w2) = -log θ(w2| β,w1), which is non-negative • Find the most probable sequence from BEGIN to END • Problem equivalent to shortest path problem β: NUM:count(STATE) dβ(number,of) dβ(the,number) dβ(of,STATE1) dβ(STATE1,END) STATE1 the number of BEGIN how many STATE1 END dβ(BEGIN,the) dβ(BEGIN,how) dβ(how,many) dβ(many,STATE1) dβ(STATE1,END) … … Lu, Ng & Lee

Direct Inversion Model • Problems with the Direct Inversion Model • Strong independence assumptions • Always generates the same phrase below the same MR production, regardless of context MR productions • Modeling dependencies at word level only • Need to model dependencies between adjacent hybrid sequences Lu, Ng & Lee

NLG with Hybrid Trees • Tree Conditional Random Fields (CRF) Model • Generate complete phrases instead of words • Explicitly model dependencies between adjacent phrases STATE:loc_1(RIVER) NUM:count(STATE) BEGIN how many STATE1 END BEGIN have RIVER1 END STATE:exclude(STATE STATE) STATE:state(all) BEGINSTATE1 do not STATE2 END BEGIN states END Lu, Ng & Lee

Tree CRF Model QUERY:answer(RIVER) what is RIVER1 RIVER:longest(RIVER) the longest RIVER1 RIVER:exclude(RIVER1,RIVER2) RIVER1 that does not RIVER2 RIVER:river(all) RIVER:traverse(STATE) river run through STATE1 STATE:stateid(STATENAME) STATENAME1 STATENAME:texas Four sets of features: Hybrid Sequence Features Two-level Hybrid Sequence Features Three-level Hybrid Sequence Features Adjacent Hybrid Sequence Features texas Lu, Ng & Lee

Features for Tree CRF Model QUERY:answer(RIVER) what is RIVER1 RIVER:longest(RIVER) the longest RIVER1 RIVER:exclude(RIVER1,RIVER2) RIVER1 that does not RIVER2 RIVER:river(all) RIVER:traverse(STATE) river run through STATE1 STATE:stateid(STATENAME) STATENAME1 STATENAME:texas Hybrid Sequence Features texas Lu, Ng & Lee

Features for Tree CRF Model QUERY:answer(RIVER) what is RIVER1 RIVER:longest(RIVER) the longest RIVER1 RIVER:exclude(RIVER1,RIVER2) RIVER1 that does not RIVER2 RIVER:river(all) RIVER:traverse(STATE) river run through STATE1 STATE:stateid(STATENAME) STATENAME1 STATENAME:texas Two-level Hybrid Sequence Features texas Lu, Ng & Lee

Features for Tree CRF Model QUERY:answer(RIVER) what is RIVER1 RIVER:longest(RIVER) the longest RIVER1 RIVER:exclude(RIVER1,RIVER2) RIVER1 that does not RIVER2 RIVER:river(all) RIVER:traverse(STATE) river run through STATE1 STATE:stateid(STATENAME) STATENAME1 STATENAME:texas Three-level Hybrid Sequence Features texas Lu, Ng & Lee

Features for Tree CRF Model QUERY:answer(RIVER) what is RIVER1 RIVER:longest(RIVER) the longest RIVER1 RIVER:exclude(RIVER1,RIVER2) RIVER1 that does not RIVER2 RIVER:river(all) RIVER:traverse(STATE) river run through STATE1 STATE:stateid(STATENAME) STATENAME1 STATENAME:texas Adjacent Hybrid Sequence Features texas Lu, Ng & Lee

Strengths of Tree CRF Model • Allows features that specifically model the dependencies between neighboring hybrid sequences in the tree to be used • Can efficiently capture long range dependencies between MR productions and hybrid sequences since each hybrid sequence is allowed to depend on the entire MR tree. Lu, Ng & Lee

Tree CRF Model • Candidate hybrid sequences • Each MR production is associated with a set of candidate hybrid sequences • Tree CRF: the correct hybrid sequence for each MR production is hidden NUM:count(STATE) BEGIN how many STATE1 END BEGIN how many STATE1 are there END BEGIN what is the number of STATE1 END BEGIN count the number of STATE1 END BEGIN give me the number of STATE1 END …… …… Lu, Ng & Lee

Tree CRF Model • Candidate hybrid sequences • Training set consists of hybrid trees which are determined with Viterbi algorithm • Candidate hybrid sequences for each MR production are extracted from these training hybrid trees NUM:count(STATE) BEGIN how many STATE1 END BEGIN how many STATE1 are there END BEGIN what is the number of STATE1 END BEGIN count the number of STATE1 END BEGIN give me the number of STATE1 END …… …… Lu, Ng & Lee

Evaluations (I) • Comparison over two models • Two benchmark corpora: Geoquery and Robocup • The tree CRF model performs better than the direct inversion model • Validates the belief that some long range dependencies are important for NLG task • While the direct inversion model performs well on ROBOCUP, it performs substantially worse on GEOQUERY Lu, Ng & Lee

Sample Outputs Lu, Ng & Lee

Evaluations (II) • Comparison with the previous state-of-the-art model • Previous model optimizes evaluation metrics directly • However, on both corpora, the tree CRF model performs better than the previous model • Confirms that longer range dependencies and phrase-level dependencies are more important Lu, Ng & Lee

Evaluations (III) • Comparison on other languages (GEOQUERY-250) • Achieves better performance than the previous state-of-the-art system on all languages Lu, Ng & Lee

Conclusions • Built two novel models for NLG, on top of the hybrid tree framework • Direct Inversion Model • Tree CRF Model • Evaluation shows the tree CRF model performs better than the direct inversion model • Further evaluation shows the proposed tree CRF model performs better than a previous state-of-the-art system reported in the literature Lu, Ng & Lee

Questions? Lu, Ng & Lee

Natural Language Generation with Tree Conditional Random Fields