Syntax

1. Syntax Sudeshna Sarkar 25 Aug 2008

2. Top-Down and Bottom-Up • Top-down • Only searches for trees that can be answers (i.e. S’s) • But also suggests trees that are not consistent with any of the words • Bottom-up • Only forms trees consistent with the words • But suggest trees that make no sense globally

3. Problems • Even with the best filtering, backtracking methods are doomed if they don’t address certain problems • Ambiguity • Shared subproblems

4. Ambiguity

5. Shared Sub-Problems • No matter what kind of search (top-down or bottom-up or mixed) that we choose. • We don’t want to unnecessarily redo work we’ve already done.

6. Shared Sub-Problems • Consider • A flight from Indianapolis to Houston on TWA

7. Shared Sub-Problems • Assume a top-down parse making bad initial choices on the Nominal rule. • In particular… • Nominal -> Nominal Noun • Nominal -> Nominal PP

8. Shared Sub-Problems

9. Shared Sub-Problems

10. Shared Sub-Problems

11. Shared Sub-Problems

12. Parsing • CKY • Earley • Both are dynamic programming solutions that run in O(n**3) time. • CKY is bottom-up • Earley is top-down

13. Sample Grammar

14. Dynamic Programming • DP methods fill tables with partial results and • Do not do too much avoidable repeated work • Solve exponential problems in polynomial time (sort of) • Efficiently store ambiguous structures with shared sub-parts.

15. CKY Parsing • First we’ll limit our grammar to epsilon-free, binary rules (more later) • Consider the rule A -> BC • If there is an A in the input then there must be a B followed by a C in the input. • If the A spans from i to j in the input then there must be some k st. i<k<j • Ie. The B splits from the C someplace.

16. CKY • So let’s build a table so that an A spanning from i to j in the input is placed in cell [i,j] in the table. • So a non-terminal spanning an entire string will sit in cell [0, n] • If we build the table bottom up we’ll know that the parts of the A must go from i to k and from k to j

17. CKY • Meaning that for a rule like A -> B C we should look for a B in [i,k] and a C in [k,j]. • In other words, if we think there might be an A spanning i,j in the input… AND • A -> B C is a rule in the grammar THEN • There must be a B in [i,k] and a C in [k,j] for some i<k<j

18. CKY • So to fill the table loop over the cell[i,j] values in some systematic way • What constraint should we put on that? • For each cell loop over the appropriate k values to search for things to add.

19. CKY Table

20. CKY Algorithm

21. CKY Parsing • Is that really a parser?

22. Note • We arranged the loops to fill the table a column at a time, from left to right, bottom to top. • This assures us that whenever we’re filling a cell, the parts needed to fill it are already in the table (to the left and below)

23. Example

24. Other Ways to Do It? • Are there any other sensible ways to fill the table that still guarantee that the cells we need are already filled?

25. Other Ways to Do It?

26. Sample Grammar

27. Problem • What if your grammar isn’t binary? • As in the case of the TreeBank grammar? • Convert it to binary… any arbitrary CFG can be rewritten into Chomsky-Normal Form automatically. • What does this mean? • The resulting grammar accepts (and rejects) the same set of strings as the original grammar. • But the resulting derivations (trees) are different.

28. Problem • More specifically, rules have to be of the form A -> B C Or A -> w That is rules can expand to either 2 non-terminals or to a single terminal.

29. Binarization Intuition • Eliminate chains of unit productions. • Introduce new intermediate non-terminals into the grammar that distribute rules with length > 2 over several rules. So… S -> A B C • Turns into S -> X C X - A B Where X is a symbol that doesn’t occur anywhere else in the the grammar.

30. CNF Conversion

31. CKY Algorithm

32. Example Filling column 5

33. Example

34. Example

35. Example

36. Example

37. END

38. Statistical parsing • Over the last 12 years statistical parsing has succeeded wonderfully! • NLP researchers have produced a range of (often free, open source) statistical parsers, which can parse any sentence and often get most of it correct • These parsers are now a commodity component • The parsers are still improving year-on-year.

39. Classical NLP Parsing • Wrote symbolic grammar and lexicon • S  NP VP NN  interest • NP  (DT) NN NNS  rates • NP  NN NNS NNS  raises • NP  NNP VBP  interest • VP  V NP VBZ  rates • … • Used proof systems to prove parses from words • This scaled very badly and didn’t give coverage • Minimal grammar on “Fed raises” sentence: 36 parses • Simple 10 rule grammar: 592 parses • Real-size broad-coverage grammar: millions of parses

40. Classical NLP Parsing:The problem and its solution • Very constrained grammars attempt to limit unlikely/weird parses for sentences • But the attempt make the grammars not robust: many sentences have no parse • A less constrained grammar can parse more sentences • But simple sentences end up with ever more parses • Solution: We need mechanisms that allow us to find the most likely parse(s) • Statistical parsing lets us work with very loose grammars that admit millions of parses for sentences but to still quickly find the best parse(s)

41. The rise of annotated data:The Penn Treebank ( (S (NP-SBJ (DT The) (NN move)) (VP (VBD followed) (NP (NP (DT a) (NN round)) (PP (IN of) (NP (NP (JJ similar) (NNS increases)) (PP (IN by) (NP (JJ other) (NNS lenders))) (PP (IN against) (NP (NNP Arizona) (JJ real) (NN estate) (NNS loans)))))) (, ,) (S-ADV (NP-SBJ (-NONE- *)) (VP (VBG reflecting) (NP (NP (DT a) (VBG continuing) (NN decline)) (PP-LOC (IN in) (NP (DT that) (NN market))))))) (. .)))

42. The rise of annotated data • Going into it, building a treebank seems a lot slower and less useful than building a grammar • But a treebank gives us many things • Reusability of the labor • Broad coverage • Frequencies and distributional information • A way to evaluate systems

43. Human parsing • Humans often do ambiguity maintenance • Have the police … eaten their supper? • come in and look around. • taken out and shot. • But humans also commit early and are “garden pathed”: • The man who hunts ducks out on weekends. • The cotton shirts are made from grows in Mississippi. • The horse raced past the barn fell.

44. Phrase structure grammars = context-free grammars • G = (T, N, S, R) • T is set of terminals • N is set of nonterminals • For NLP, we usually distinguish out a set P  N of preterminals, which always rewrite as terminals • S is the start symbol (one of the nonterminals) • R is rules/productions of the form X  , where X is a nonterminal and  is a sequence of terminals and nonterminals (possibly an empty sequence) • A grammar G generates a language L.

45. Probabilistic or stochastic context-free grammars (PCFGs) • G = (T, N, S, R, P) • T is set of terminals • N is set of nonterminals • For NLP, we usually distinguish out a set P  N of preterminals, which always rewrite as terminals • S is the start symbol (one of the nonterminals) • R is rules/productions of the form X  , where X is a nonterminal and  is a sequence of terminals and nonterminals (possibly an empty sequence) • P(R) gives the probability of each rule. • A grammar G generates a language model L.

46. Soundness and completeness • A parser is sound if every parse it returns is valid/correct • A parser terminates if it is guaranteed to not go off into an infinite loop • A parser is complete if for any given grammar and sentence, it is sound, produces every valid parse for that sentence, and terminates • (For many purposes, we settle for sound but incomplete parsers: e.g., probabilistic parsers that return a k-best list.)

47. Top-down parsing • Top-down parsing is goal directed • A top-down parser starts with a list of constituents to be built. The top-down parser rewrites the goals in the goal list by matching one against the LHS of the grammar rules, and expanding it with the RHS, attempting to match the sentence to be derived. • If a goal can be rewritten in several ways, then there is a choice of which rule to apply (search problem) • Can use depth-first or breadth-first search, and goal ordering.

48. Top-down parsing

49. Bottom-up parsing • Bottom-up parsing is data directed • The initial goal list of a bottom-up parser is the string to be parsed. If a sequence in the goal list matches the RHS of a rule, then this sequence may be replaced by the LHS of the rule. • Parsing is finished when the goal list contains just the start category. • If the RHS of several rules match the goal list, then there is a choice of which rule to apply (search problem) • Can use depth-first or breadth-first search, and goal ordering. • The standard presentation is as shift-reduce parsing.

50. Shift-reduce parsing: one path cats scratch people with claws cats scratch people with claws SHIFT N scratch people with claws REDUCE NP scratch people with claws REDUCE NP scratch people with claws SHIFT NP V people with claws REDUCE NP V peoplewith claws SHIFT NP V N with claws REDUCE NP V NP with claws REDUCE NP V NP with claws SHIFT NP V NP P claws REDUCE NP V NP P claws SHIFT NP V NP P N REDUCE NP V NP P NP REDUCE NP V NP PP REDUCE NP VP REDUCE S REDUCE What other search paths are there for parsing this sentence?