The Leaf Projection Path View of Parse Trees: Exploring String Kernels for HPSG Parse Selection

1. The Leaf Projection Path View of Parse Trees: Exploring String Kernels for HPSG Parse Selection Kristina Toutanova, Penka Markova, Christopher Manning Computer Science Department Stanford University

2. Motivation: the task “I would like to meet with you again on Monday” Input: a sentence Classify to one of the possible parses focus on discriminating among parses

3. to meet meet on Motivation: traditional representation of parse trees • Features are pieces of local rule productions with grand-parenting When using plain context free rules most features make no reference to the input string – naive for a discriminative model! Lexicalization with the head word introduces more connection to the input

4. Motivation: traditional representation of parse trees • All subtrees representation: features are (a restricted kind) of subtrees of the original tree must choose features or discount larger trees

5. General idea: representation Trees are lists of leaf projection paths • Provides broader view of tree contexts • Increases connection to the input string (words) • Captures examples of non-head dependencies like in “more careful than his sister” (Bod 98) Non-head path is included in addition to the head path Each node is lexicalized with all words dominated by it Trees must be binarized

6. General idea: tree kernels • Often only a kernel (a similarity measure) between trees is necessary for ML algorithms. Measure the similarity between trees by the similarity between projection paths of common words/pos tags in the trees.

7. S VP VP-NF VP VP-NF VP-NF VP-NF VP-NF meet General idea: tree kernels from string kernels • Measures of similarity between sequences (strings) have been developed for many domains. • use string kernels between projection paths and combine them into a tree kernel via a convolution • this gives rise to interesting features and more global modeling of the syntactic environment of words S VP VP-NF VP-NF VP-NF VP VP-NF VP-NF meet SIM

8. Overview • HPSG syntactic analyses representation • Illustration of the leaf projection paths representation • Comparison to traditional rule representation • experimental results • Tree kernels from string kernels on projection paths • Experimental results

9. IMPER HCOMP HCOMP HCOMP LET_V1 US PLAN_ON_V2 HCOMP plan ON let us THAT_DEIX on that HPSG tree representation: derivation trees HPSG – Head Driven Phrase Structure Grammar; lexicalized unification based grammar ERG grammar of English Node labels are rule names such as head-complement and head-adjunct The inventory of rules is larger than in traditional HPSG grammars Full HPSG signs can be recovered from the derivation trees using the grammar We use annotated derivation trees as the main representation for disambiguation

10. IMPER : verb HCOMP : verb HCOMP HCOMP : verb : verb LET_V1 US PLAN_ON_V2 HCOMP : prep* plan ON let us THAT_DEIX on that HPSG tree representation: annotation of nodes Annotation with the value of synsem.local.cat.head Its values are a small set of part-of-speech tags

11. HPSG tree representation: syntactic word classes Our representation heavily uses word classes to backoff from words word types lexical item ids ON THAT_DEIX LET_V1 US PLAN_ON_V2 plan let us on that v_empty_prep_intrans v_sorb n_pers_pro p_reg n_deictic_pro The word classes are around 500 types in the HPSG type hierarchy. They show detailed syntactic information including e.g. subcategorization.

12. END IMPER: verb IMPER : verb HCOMP: verb HCOMP HCOMP: verb : verb HCOMP HCOMP END LET_V1 verb : verb : verb START START LET_V1 US PLAN_ON_V2 HCOMP : prep* let v_sorb let v_sorb n_pers_pro v_empty_prep_ intrans v_sorb plan ON let us THAT_DEIX p_reg on that n_deictic_pro Leaf projection paths representation • The tree is represented as a list of paths from the words to the top. • The paths are keyed by words and corresponding word classes. • The head and non-head paths are treated separately.

13. IMPER : verb END END HCOMP HCOMP: verb : verb IMPER: verb HCOMP HCOMP PLAN_ON: verb HCOMP: verb : verb : verb START START LET_V1 US PLAN_ON HCOMP : prep* plan v_empty_prep_intrans plan v_empty_prep_intrans n_pers_pro v_empty_prep_ intrans v_sorb plan ON let us THAT_DEIX p_reg on that n_deictic_pro Leaf projection paths representation • The tree is represented as a list of paths from the words to the top. • The paths are keyed by words and corresponding word classes. • The head and non-head paths are treated separately.

14. IMPER : verb HCOMP : verb HCOMP HCOMP : verb : verb LET_V1 US PLAN_ON HCOMP : prep* n_pers_pro v_empty_prep_ intrans v_sorb plan ON let us THAT_DEIX p_reg on that n_deictic_pro Leaf projection paths representation Can recover local rules by annotation of nodes with sister and parent categories Now extract features from this representation for discriminative models

15. Overview • HPSG syntactic analyses representation • Illustration of the leaf projection paths representation • Comparison to traditional rule representation • experimental results • Tree kernels from string kernels on projection paths • Experimental Results

16. Machine learning task setup Given m training sentences Sentence si has pi possible analyses and ti,1 is the correct analysis Learn a parameter vector and choose for a test sentence the tree t with the maximum score Linear Models e.g. (Collins 00)

17. Choosing the parameter vector • Previous formulations (Collins 01, Shen and Joshi 03) • We solve this problem using SVMLight for ranking • For all models we extract all features from the kernel’s feature map and solve the problem with a linear kernel

18. The leaf projection paths view versus the context free rule view • Goals: • Compare context free rule models to projection path models • Evaluate the usefulness of non-head paths • Models • Projection paths: • Bi-gram model on projection paths (2PP) • Bi-gram model on head projection paths only (2HeadPP) • Context free rules: • Joint rule model (J-Rule) • Independent rule model (I-Rule)

19. IMPER : verb HCOMP : verb HCOMP HCOMP : verb LET_V1 US PLAN_ON_V2 HCOMP : prep* n_pers_pro v_empty_prep_ intrans v_sorb plan ON let us THAT_DEIX p_reg on that n_deictic_pro The leaf projection paths view versus the context free rule view • 2PP has as features bi-grams from the projection paths. • Features of 2PP including the node HCOMP :verb • plan (headpath) • [v_empty_prep_intrans,PLAN_ON_V2,HCOMP,head] • [v_empty_prep_intrans,HCOMP,END,head]

20. IMPER : verb HCOMP : verb HCOMP HCOMP : verb LET_V1 US PLAN_ON_V2 HCOMP : prep* n_pers_pro v_empty_prep_ intrans v_sorb plan ON let us THAT_DEIX p_reg on that n_deictic_pro The leaf projection paths view versus the context free rule view • 2PP has as features bi-grams from the projection paths. • Features of 2PP including the node HCOMP :verb • plan (headpath) • [v_empty_prep_intrans,PLAN_ON_V2,HCOMP,head] • [v_empty_prep_intrans,HCOMP,END,head]

21. IMPER : verb HCOMP : verb HCOMP HCOMP : verb LET_V1 US PLAN_ON_V2 HCOMP : prep* n_pers_pro v_empty_prep_ intrans v_sorb plan ON let us THAT_DEIX p_reg on that n_deictic_pro The leaf projection paths view versus the context free rule view • 2PP has as features bi-grams from the projection paths. • Features of 2PP including the node HCOMP :verb • on (non-headpath) • [p_reg,START,HCOMP,non-head] • [p_reg,HCOMP,HCOMP,non-head]

22. IMPER : verb HCOMP : verb HCOMP HCOMP : verb LET_V1 US PLAN_ON_V2 HCOMP : prep* n_pers_pro v_empty_prep_ intrans v_sorb plan ON let us THAT_DEIX p_reg on that n_deictic_pro The leaf projection paths view versus the context free rule view • 2PP has as features bi-grams from the projection paths. • Features of 2PP including the node HCOMP :verb • on (non-headpath) • [p_reg,START,HCOMP,non-head] • [p_reg,HCOMP,HCOMP,non-head]

23. IMPER : verb HCOMP : verb HCOMP HCOMP : verb LET_V1 US PLAN_ON_V2 HCOMP : prep* n_pers_pro v_empty_prep_ intrans v_sorb plan ON let us THAT_DEIX p_reg on that n_deictic_pro The leaf projection paths view versus the context free rule view • 2PP has as features bi-grams from the projection paths. • Features of 2PP including the node HCOMP :verb • that (non-headpath) • [n_deictic_pro,HCOMP,HCOMP,non-head] • [n_deictic_pro,HCOMP,HCOMP,non-head]

24. IMPER : verb HCOMP : verb HCOMP HCOMP : verb LET_V1 US PLAN_ON_V2 HCOMP : prep* n_pers_pro v_empty_prep_ intrans v_sorb plan ON let us THAT_DEIX p_reg on that n_deictic_pro The leaf projection paths view versus the context free rule view • 2PP has as features bi-grams from the projection paths. • Features of 2PP including the node HCOMP :verb • that (non-headpath) • [n_deictic_pro,HCOMP,HCOMP,non-head] • [n_deictic_pro,HCOMP,HCOMP,non-head]

25. IMPER : verb HCOMP : verb HCOMP HCOMP : verb : verb LET_V1 US PLAN_ON_V2 HCOMP : prep* n_pers_pro v_empty_prep_ intrans v_sorb plan ON let us THAT_DEIX p_reg on that n_deictic_pro [v_empty_prep_intrans,PLAN_ON_V2,HCOMP,head] The leaf projection paths view versus the context free rule view • I-Rule has as features edges of the tree, annotated with the word class of the child and head vs. non-head information • Features of I-Rule including the node HCOMP

26. IMPER : verb HCOMP : verb HCOMP HCOMP : verb : verb LET_V1 US PLAN_ON_V2 HCOMP : prep* n_pers_pro v_empty_prep_ intrans v_sorb plan ON let us THAT_DEIX p_reg on that n_deictic_pro The leaf projection paths view versus the context free rule view • I-Rule has as features edges of the tree, annotated with the word class of the child and head vs. non-head information • Features of I-Rule including the node HCOMP [p_reg,HCOMP,HCOMP,non-head]

27. IMPER : verb HCOMP : verb HCOMP HCOMP : verb : verb LET_V1 US PLAN_ON_V2 HCOMP : prep* n_pers_pro v_empty_prep_ intrans v_sorb plan ON let us THAT_DEIX p_reg on that n_deictic_pro The leaf projection paths view versus the context free rule view • I-Rule has as features edges of the tree, annotated with the word class of the child and head vs. non-head information • Features of I-Rule including the node HCOMP [v_empty_prep_intrans,HCOMP,HCOMP,non-head]

28. Comparison results • Redwoods corpus 3829 ambiguous sentences; average number of words 7.8 average ambiguity 10.8 10-fold cross-validation ; report exact match accuracy Non-head paths are useful (13% relative error reduction from head only) The bi-gram model on projection paths performs better than a very similar local rule based model

29. Overview • HPSG syntactic analyses representation • Illustration of the leaf projection paths representation • Comparison to traditional rule representation • experimental results • Tree kernels from string kernels on projection paths • Experimental Results

30. String kernels on projection paths • We looked at a bi-gram model on projection paths (2PP). • This is a special case of a string kernel (n-gram kernel). • We could use more general string kernels on projection paths --- existing ones, that handle non-contiguous substrings or more complex matching of nodes. • It is straightforward to combine them into tree kernels.

31. END IMPER: verb HCOMP: verb HCOMP: verb END LET_V1 verb START START let v_sorb let v_sorb Formal representation of parse trees t key1=let (head) X1=“START LET_V1:verb HCOMP:verb HCOMP:verb IMPER:verb END” key2=v_sorb(head) X2 = X1 key3=let (non-head) X3=“START END” key4=v_sorb(non-head) X4 = X3

32. Tree kernels using string kernels on projection paths t t’

33. String kernels overview • Define string kernels by their feature map from strings to vectors indexed by feature indices Example: 1-gram kernel END IMPER HCOMP HCOMP LET_V1 START

34. Repetition kernel • General idea: Improve on the 1-gram kernel by better handling repeated symbols. He eats chocolate from Belgium with fingers . head path of eats when high attachment – (NP PP PP NP) Rather than the feature for PP having twice as much weight, there should be a separate feature indicating that there are two PPs. The feature space is indexed by strings Two discount factors for gaps and for letters NP NP PP PP

35. The Repetition kernel versus 1-gram and 2-gram 1-gram44,278 features Repetition52,994 features 2-gram104,331 features Repetition achieves 7.8% error reduction from 1-gram

36. Other string kernels • So far: 1-gram,2-gram, repetition • Next: allow general discontinuous n-grams • restricted subsequence kernel • Also: allow partial matching • wildcard kernel allowing a wild-card character in the n-gram features; the wildcard matches any character Lodhi et al. 02; Leslie and Kuang 03

37. Restricted subsequence kernel • Has parameters k – maximum size of the feature n-gram; g – maximum span in the string; λ1 - gap penalty and λ2 -letter - penalty λ2 when k=2,g=5, λ1 =.5, λ2 =1 END IMPER HCOMP HCOMP LET_V1 START

38. Varying the string kernels on word class keyed paths 1-gram (13K) 81.43 2-gram (37K) 82.70 subseq (2,3,.50,2) (81K) 83.22 subseq (2,3,.25,2) (81K) 83.48 subseq (2,4,. 5,2) (102K) 83.29 subseq (3,5,.25,2)(416K) 83.06

39. Varying the string kernels on word class keyed paths 1-gram (13K) 81.43 2-gram (37K) 82.70 subseq (2,3,.50,2) (81K) 83.22 subseq (2,3,.25,2) (81K) 83.48 subseq (2,4,.50,2) (102K) 83.29 subseq (3,5,.25,2) (416K) 83.06 Increasing the amount of discontinuity or adding larger n-gram did not help

40. Adding word keyed paths Fixed the kernel for word keyed paths to 2-gram+repetition Best previous result from a single classifier 82.7 (mostly local rule based). Relative error reduction is 13%

41. Other models and model combination • Many features are available in the HPSG signs. • A single model is likely to over-fit when given too many features. • To better use the additional information, train several classifiers and combine them by voting Best previous result from voting classifiers is 84.23% (Osborne & Balbridge 04)

42. Conclusions and future work Summary • We presented a new representation of parse trees leading to a tree kernel • It allows the modeling of more global tree contexts as well as greater lexicalization • We demonstrated gains from applying existing string kernels on projection paths and new kernels useful for the domain (Repetition kernel) • The major gains were due to the representation Future Work • Other sequence kernels better suited for the task • Feature selection: which words / word classes deserve better modeling of their leaf paths • Other corpora