Constraints Driven Structured Learning with Indirect Supervision

1. Constraints Driven Structured Learningwith Indirect Supervision Dan Roth Department of Computer Science University of Illinois at Urbana-Champaign With thanks to: Collaborators:Ming-Wei Chang, James Clarke, Dan Goldwasser, Lev Ratinov, Vivek Srikumar, Many others Funding: NSF: ITR IIS-0085836, SoD-HCER-0613885, DHS; DARPA: Bootstrap Learning & Machine Reading Programs DASH Optimization (Xpress-MP) April 2010 Carnegie Mellon University

2. Nice to Meet You

3. A process that maintains and updates a collection of propositions about the state of affairs. Comprehension (ENGLAND, June, 1989) - Christopher Robin is alive and well. He lives in England. He is the same person that you read about in the book, Winnie the Pooh. As a boy, Chris lived in a pretty home called Cotchfield Farm. When Chris was three years old, his father wrote a poem about him. The poem was printed in a magazine for others to read. Mr. Robin then wrote a book. He made up a fairy tale land where Chris lived. His friends were animals. There was a bear called Winnie the Pooh. There was also an owl and a young pig, called a piglet. All the animals were stuffed toys that Chris owned. Mr. Robin made them come to life with his words. The places in the story were all near Cotchfield Farm. Winnie the Pooh was written in 1925. Children still love to read about Christopher Robin and his animal friends. Most people don't know he is a real person who is grown now. He has written two books of his own. They tell what it is like to be famous. 1. Christopher Robin was born in England. 2. Winnie the Pooh is a title of a book. 3. Christopher Robin’s dad was a magician. 4. Christopher Robin must be at least 65 now. This is an Inference Problem

4. Coherency in Semantic Role Labeling Predicate-arguments generated should be consistent across phenomena The touchdown scoredby Bettis cemented the victoryof the Steelers. Linguistic Constraints: A0: the Steelers Sense(of): 11(6) A0:Bettis Sense(by): 1(1)

5. Semantic Parsing Y: largest( state( next_to( state(NY) AND next_to (state(MD)))) X :“What is the largest state that borders New York and Maryland ?" • Successful interpretation involves multiple decisions • What entities appear in the interpretation? • “New York” refers to a state or a city? • How to compose fragments together? • state(next_to()) >< next_to(state())

6. Learning and Inference • Natural Language Decisions are Structured • Global decisions in which several local decisions play a role but there are mutual dependencies on their outcome. • It is essential to make coherent decisions in a way that takes the interdependencies into account. Joint, Global Inference. • But: Learning structured models requires annotating structures.

7. Penalty for violating the constraint. Weight Vector for “local” models How far y is from a “legal” assignment Features, classifiers; log-linear models (HMM, CRF) or a combination Constrained Conditional Models (aka ILP Inference) (Soft) constraints component CCMs can be viewed as a general interface to easily combine domain knowledge with data driven statistical models How to solve? This is an Integer Linear Program Solving using ILP packages gives an exact solution. Search techniques are also possible How to train? Training is learning the objective function How to exploit the structure to minimize supervision?

8. Example: Semantic Role Labeling Who did what to whom, when, where, why,… I left my pearls to my daughter in my will . [I]A0left[my pearls]A1[to my daughter]A2[in my will]AM-LOC . • A0 Leaver • A1 Things left • A2 Benefactor • AM-LOC Location I left my pearls to my daughter in my will . Overlapping arguments If A2 is present, A1 must also be present.

9. Semantic Role Labeling (2/2) • PropBank [Palmer et. al. 05] provides a large human-annotated corpus of semantic verb-argument relations. • It adds a layer of generic semantic labels to Penn Tree Bank II. • (Almost) all the labels are on the constituents of the parse trees. • Core arguments: A0-A5 and AA • different semantics for each verb • specified in the PropBank Frame files • 13 types of adjuncts labeled as AM-arg • where arg specifies the adjunct type

10. I left my nice pearls to her I left my nice pearls to her I left my nice pearls to her I left my nice pearls to her [ [ [ [ [ [ [ [ [ [ ] ] ] ] ] ] ] ] ] ] Identify Vocabulary Algorithmic Approach candidate arguments • Identify argument candidates • Pruning [Xue&Palmer, EMNLP’04] • Argument Identifier • Binary classification (SNoW) • Classify argument candidates • Argument Classifier • Multi-class classification (SNoW) • Inference • Use the estimated probability distribution given by the argument classifier • Use structural and linguistic constraints • Infer the optimal global output Inference over (old and new) Vocabulary Ileftmy nice pearlsto her

11. Semantic Role Labeling (SRL) I left my pearls to my daughter in my will . Page 11

12. Semantic Role Labeling (SRL) I left my pearls to my daughter in my will . Page 12

13. Semantic Role Labeling (SRL) I left my pearls to my daughter in my will . One inference problem for each verb predicate. Page 13

14. Integer Linear Programming Inference • For each argument ai • Set up a Boolean variable: ai,tindicating whether ai is classified as t • Goal is to maximize • i score(ai = t ) ai,t • Subject to the (linear) constraints • If score(ai = t ) = P(ai = t ), the objective is to find the assignment that maximizes the expected number of arguments that are correct and satisfies the constraints. The Constrained Conditional Model is completely decomposed during training

15. Constraints Any Boolean rule can be encoded as a (collection of) linear constraints. • No duplicate argument classes aPOTARG x{a = A0} 1 • R-ARG  a2POTARG , aPOTARG x{a = A0}x{a2 = R-A0} • C-ARG • a2POTARG , (aPOTARG)  (a is before a2 )x{a = A0}x{a2 = C-A0} • Many other possible constraints: • Unique labels • No overlapping or embedding • Relations between number of arguments; order constraints • If verb is of type A, no argument of type B If there is an R-ARG phrase, there is an ARG Phrase If there is an C-ARG phrase, there is an ARG before it Universally quantified rules LBJ: allows a developer to encode constraints in FOL; these are compiled into linear inequalities automatically. Joint inference can be used also to combine different (SRL) Systems.

16. Learning and Inference • Natural Language Decisions are Structured • Global decisions in which several local decisions play a role but there are mutual dependencies on their outcome. • It is essential to make coherent decisions in a way that takes the interdependencies into account. Joint, Global Inference. • But: Learning structured models requires annotating structures.

17. Prediction result of a trained HMM Lars Ole Andersen . Program analysis and specialization for the C Programming language . PhD thesis . DIKU , University of Copenhagen , May 1994 . [AUTHOR] [TITLE] [EDITOR] [BOOKTITLE] [TECH-REPORT] [INSTITUTION] [DATE] Information extraction without Prior Knowledge Lars Ole Andersen . Program analysis and specialization for the C Programming language. PhD thesis. DIKU , University of Copenhagen, May 1994 . Violates lots of natural constraints! Page 17

18. Examples of Constraints Easy to express pieces of “knowledge” Non Propositional; May use Quantifiers Each field must be aconsecutive list of words and can appear at mostoncein a citation. State transitions must occur onpunctuation marks. The citation can only start withAUTHORorEDITOR. The wordspp., pagescorrespond toPAGE. Four digits starting with20xx and 19xx areDATE. Quotationscan appear only inTITLE …….

19. Information Extraction with Constraints • Adding constraints, we getcorrectresults! • Without changing the model • [AUTHOR]Lars Ole Andersen . [TITLE]Program analysis andspecialization for the C Programming language . [TECH-REPORT] PhD thesis . [INSTITUTION] DIKU , University of Copenhagen , [DATE] May, 1994 . Page 19

20. Guiding Semi-Supervised Learning with Constraints • In traditional Semi-Supervised learning the model can drift away from the correct one. • Constraints can be used to generate better training data • At training to improve labeling of un-labeled data (and thus improve the model) • At decision time, to bias the objective function towards favoring constraint satisfaction. Constraints Model Un-labeled Data Decision Time Constraints

21. Constraints Driven Learning (CoDL) [Chang, Ratinov, Roth, ACL’07;ICML’08,Long’10] (w0,½0)=learn(L)‏ For N iterations do T= For each x in unlabeled dataset h à argmaxy wTÁ(x,y) - ½k dC(x,y) T=T  {(x, h)} (w,½) =  (w0,½0) + (1- ) learn(T) Supervised learning algorithm parameterized by (w,½). Learning can be justified as an optimization procedure for an objective function Inference with constraints: augment the training set Learn from new training data Weigh supervised & unsupervised models. Excellent Experimental Results showing the advantages of using constraints, especially with small amounts on labeled data [Chang et. al, Others] Page 21

22. Constraints Driven Learning (CODL) [Chang et. al 07,08; others] • Semi-Supervised Learning Paradigm that makes use of constraints to bootstrap from a small number of examples Objective function: Learning w 10 Constraints Poor model + constraints Constraints are used to: • Bootstrap a semi-supervised learner • Correct weak models predictions on unlabeled data, which in turn are used to keep training the model. Learning w/o Constraints: 300 examples. # of available labeled examples

23. Learning and Inference • Natural Language Decisions are Structured • Global decisions in which several local decisions play a role but there are mutual dependencies on their outcome. • It is essential to make coherent decisions in a way that takes the interdependencies into account. Joint, Global Inference. • But: Learning structured models requires annotating structures. • Interdependencies among decision variables should be exploited in learning. • Goal: use minimal, indirect supervision • Amplify it using interdependencies among variables

24. Two Ideas • Idea1: Simple, easy to supervise, binary decisions often depend on the structure you care about. Learning to do well on the binary task can drive the structure learning. • Idea2: Global Inference can be used to amplify the minimal supervision. • Idea 2 ½:There are several settings where a binary label can be used to replace a structured label. Perhaps the most intriguing is where you use the world response to the model’s actions.

25. Outline • Inference • Semi-supervised Training Paradigms for structures • Constraints Driven Learning • Indirect Supervision Training Paradigms for structure • Indirect Supervision Training with latent structure [NAACL’10] • Transliteration; Textual Entailment; Paraphrasing • Training Structure Predictors by Inventing (easy to supervise) binary labels [ICML’10] • POS, Information extraction tasks • Driving supervision signal from World’s Response [CoNLL’10] • Semantic Parsing

26. Textual Entailment x3 x4 x1 x2 x3 x4 x1 x5 x2 x7 x6 Former military specialist Carpenter took the helm at FictitiousCom Inc. after five years as press official at the United States embassy in the United Kingdom. Jim Carpenter worked for the US Government. • Entailment Requires an Intermediate Representation • Alignment based Features • Given the intermediate features – learn a decision • Entail/ Does not Entail But only positive entailments are expected to have a meaningful intermediate representation

27. Paraphrase Identification Given an input x 2 X Learn a model f : X ! {-1, 1} • Consider the following sentences: • S1: Druce will face murder charges, Conte said. • S2: Conte said Druce will be charged with murder . • Are S1 and S2 a paraphrase of each other? • There is a need for an intermediate representation to justify this decision • We need latent variables that explain: • why this is a positive example. Given an input x 2 X Learn a model f : X ! H ! {-1, 1}

28. Algorithms: Two Conceptual Approaches • Two stage approach (typically used forTE and paraphrase identification) • Learn hidden variables; fix it • Need supervision for the hidden layer (or heuristics) • For each example, extract features over x and (the fixed) h. • Learn a binary classier • Proposed Approach: Joint Learning • Drive the learning of h from the binary labels • Find the best h(x) • An intermediate structure representation is good to the extent is supports better final prediction. • Algorithm?

29. Learning with Constrained Latent Representation (LCLR): Intuition • If x is positive • There must exist a good explanation (intermediate representation) • 9 h, wTÁ(x,h) ¸ 0 • or, maxh wTÁ(x,h) ¸ 0 • If x is negative • No explanation is good enough to support the answer • 8 h, wTÁ(x,h) · 0 • or, maxh wTÁ(x,h) · 0 • Altogether, this can be combined into an objective function: Minw¸/2 ||w||2 + Ci l(1-zimaxh 2 C wT{s} hsÁs (xi)) • Why does inference help? New feature vector for the final decision. Chosen hselects a representation. Inference: best h subject to constraints C

30. Optimization • Non Convex, due to the maximization term inside the global minimization problem • In each iteration: • Find the best feature representation h* for all positive examples (off-the shelf ILP solver) • Having fixed the representation for the positive examples, update w solving the convex optimization problem: • Not the standard SVM/LR: need inference • Asymmetry: Only positive examples require a good intermediate representation that justify the positive label. • Consequently, the objective function decreases monotonically

31. Iterative Objective Function Learning Inference for h subj. to C Prediction with inferred h Initial Objective Function Training w/r to binary decision label Generate features • Formalized as Structured SVM + Constrained Hidden Structure • LCRL: Learning Constrained Latent Representation Update weight vector

32. Learning with Constrained Latent Representation (LCLR): Framework • LCLR provides a general inference formulation that allows that use of expressive constraints • Flexibly adapted for many tasks that require latent representations. • Paraphrasing: Model input as graphs, V(G1,2), E(G1,2) • Four Hidden variables: • hv1,v2 – possible vertex mappings; he1,e2 – possible edge mappings • Constraints: • Each vertex in G1 can be mapped to a single vertex in G2 or to null • Each edge in G1 can be mapped to a single edge in G2 or to null • Edge mapping is active iff the corresponding node mappings are active LCLR Model Declarative model

33. Experimental Results Transliteration: Recognizing Textual Entailment: Paraphrase Identification:*

34. Outline • Inference • Semi-supervised Training Paradigms for structures • Constraints Driven Learning • Indirect Supervision Training Paradigms for structure • Indirect Supervision Training with latent structure • Transliteration; Textual Entailment; Paraphrasing • Training Structure Predictors by Inventing (easy to supervise) binary labels • POS, Information extraction tasks • Driving supervision signal from World’s Response • Semantic Parsing

35. Structured Prediction • Before, the structure was in the intermediate level • We cared about the structured representation only to the extent it helped the final binary decision • The binary decision variable was given as supervision • What if we care about the structure? • Information Extraction; Relation Extraction; POS tagging, many others. • Invent a companion binary decision problem! • Parse Citations: Lars Ole Andersen . Program analysis and specialization for the C Programming language. PhD thesis. DIKU , University of Copenhagen, May 1994 . • Companion:Given a citation; does it have a legitimate parse? • POS Tagging • Companion:Given a word sequence, does it have a legitimate POS tagging sequence?

36. Predicting phonetic alignment (For Transliteration) Target Task I t a l y Yes/No Companion Task I l l i n o i s ט י ל י א ה ל י י נ א ו י Why it is a companion task? • Target Task • Input: an English Named Entity and its Hebrew Transliteration • Output:Phonetic Alignment (character sequence mapping) • A structured output prediction task (many constraints), hard to label • Companion Task • Input: an English Named Entity and an Hebrew Named Entity • Companion Output: Do they form a transliteration pair? • A binary output problem, easy to label • Negative Examples are FREE, given positive examples

37. Companion Task Binary Label as Indirect Supervision • The two tasks are related just like the binary and structured tasks discussed earlier • All positive examples must have a good structure • Negative examples cannot have a good structure • We are in the same setting as before • Binary labeled examples are easier to obtain • We can take advantage of this to help learning a structured model • Here: combine binary learning and structured learning Positive transliteration pairs must have “good” phonetic alignments • Negative transliteration pairs cannot have “good” phonetic alignments

38. Learning Structure with Indirect Supervision • In this case we care about the predicted structure • Use both Structural learning and Binary learning Predicted Correct Negative examples cannot have a good structure The feasible structures of an example Negative examples restrict the space of hyperplanes supporting the decisions for x

39. Joint Learning with Indirect Supervision (J-LIS) Target Task I t a l y Companion Task I l l i n o i s Yes/No ט י ל י א ה ל י י נ א ו י Loss function: LB, as before; LS, Structural learning Key: the same parameter wfor both components Loss on Target Task Loss on Companion Task Joint learning : If available, make use of both supervision types

40. Experimental Result Very little direct (structured) supervision. (Almost free) Large amount binary indirect supervision

41. Experimental Result Very little direct (structured) supervision. (Almost free) Large amount binary indirect supervision

42. Relations to Other Frameworks • B=Á, l=(squared) hinge loss: Structural SVM • S=Á, LCLR • Related to Structural Latent SVM (Yu & Johachims) and to Felzenszwalb. • If S=Á, Conceptually related to Contrastive Estimation • No “grouping” of good examples and bad neighbors • Max vs. Sum: we do not marginalize over the hidden structure space • Allows for complex domain specific constraints • Related to some Semi-Supervised approaches, but can use negative examples (Sheffer et. al)

43. Outline • Inference • Semi-supervised Training Paradigms for structures • Constraints Driven Learning • Indirect Supervision Training Paradigms for structure • Indirect Supervision Training with latent structure • Transliteration; Textual Entailment; Paraphrasing • Training Structure Predictors by Inventing (easy to supervise) binary labels • POS, Information extraction tasks • Driving supervision signal from World’s Response • Semantic Parsing

44. Connecting Language to the World Can I get a coffee with no sugar and just a bit of milk Great! Arggg Semantic Parser MAKE(COFFEE,SUGAR=NO,MILK=LITTLE) Can we rely on this interaction to provide supervision?

45. Real World Feedback Traditional approach: learn from logical forms and gold alignments EXPENSIVE! Our approach:use only the responses x NL Query “What is the largest state that borders NY?" Logical Query Query Response: r y largest( state( next_to( const(NY)))) Pennsylvania Binary Supervision r Semantic parsing isa structured prediction problem: identify mappings from text to a meaning representation Expected : Pennsylvania Predicted : Pennsylvania Positive Response Expected : Pennsylvania Predicted : NYC Negative Response Train a structured predictor with this binary supervision ! Supervision = Expected Response Check if Predicted response == Expected response Interactive Computer System Query Response: Pennsylvania

46. Learning Structures with a Binary Signal “What is the smallest state?" “What is the largest state that borders NY?" “What is the largest state that borders NY?" - - + + b largest(state(next_to(const(NJ)))) largest(state(next_to(const(NY)))) state(next_to(const(NY)))) • Protocol I: Direct learning with binary supervision • Uses predicted structures as examples for learning a binary decision • Inference used to predict the query: (y,z) = argmaxy,zwTÁ(x,y,z) • Positive feedback: add a positive binary example • Negativefeedback: add a negative binary example • Learned parameters form the objective function; iterate

47. Learning Structures with a Binary Signal + “What is the largest state that borders NY?" largest( state( next_to( const(NY)))) • Protocol II: Aggressive Learning with Binary supervision • Positive feedback IFF the structure is correct • (y,z) = argmaxy,zwTÁ(x,y,z) • Train a structured predictor from these structures • Positive feedback: add a positive structured example • Iterate until no new structures are found Interactive Computer System Correct Response! Pennsylvania

48. Empirical Evaluation *[WM] Y.-W. Wong and R. Mooney. 2007. Learning synchronous grammars for semantic parsing with lambda calculus. ACL. • Key Question: Can we learn from this type of supervision?

49. Summary • Constrained Conditional Model: Computation Framework for global interference and an vehicle for incorporating knowledge • Direct supervision for structured NLP tasks is expensive • Indirect supervision is cheap and easy to obtain • We suggested learning protocols for Indirect Supervision • Make use of simple, easy to get, binary supervision • Showed how to use it to learn structure • Done in the context of Constrained Conditional Models • Inference is an essential part of propagating the simple supervision • Learning Structures from Real World Feedback • Obtain binary supervision from “real world” interaction • Indirect supervision replaces direct supervision

50. Features Versus Constraints Mathematically, soft constraints are features IfÁ(x,y) = Á(x) – constraints provide an easy way to introduce dependence on y • Ái : X £ Y ! R; Ci : X £ Y ! {0,1}; d: X £ Y ! R; • In principle, constraints and features can encode the same properties • In practice, they are very different • Features • Local , short distance properties – to support tractable inference • Propositional (grounded): • E.g. True if “the followed by a Noun occurs in the sentence” • Constraints • Global properties • Quantified, first order logic expressions • E.g.True iff “all yis in the sequence y are assigned different values.”