Competitive Grammar Writing

1. Competitive Grammar Writing VP Jason Eisner Noah A. Smith Johns Hopkins Carnegie Mellon TeachCL Workshop @ ACL – June 20, 2008

2. every summer since 2002 thank you JHU,NSF, and NAACL … 1. Welcome to the lab exercise! • Please form teams of ~3 people … • Programmers, get a linguist on your team • And vice-versa • Undergrads, get a grad student on your team • And vice-versa We always run this exercise on the 1st day of the Johns Hopkins Summer School in Human Language Technology We’ve also run variants in our JHU & CMU classes

3. 2. Okay, team, please log in • The 3 of you should use adjacent workstations • Log in as individuals • Your secret team directory: cd …/03-turbulent-kiwi • You can all edit files there • Publicly readable & writeable • No one else knows the secret directory name Minimizes permissions fuss

4. 3. Now write a grammar of English • You have 2 hours.  Actually, as the deadline approaches, the teams usually vote to stay an extra hour

5. What’s a grammar? 3. Now write a grammar of English Here’s one to start with. • You have 2 hours.  • 1 S1  NP VP . • 1 VP  VerbT NP • 20 NP  Det N’ • 1 NP  Proper • 20 N’  Noun • 1 N’  N’ PP • 1 PP  Prep NP

6. 3. Now write a grammar of English Plus initial terminal rules. Here’s one to start with. • 1 Noun  castle • 1 Noun  king … • 1 Proper  Arthur • 1 Proper  Guinevere … • 1 Det  a • 1 Det  every … • 1 VerbT  covers • 1 VerbT  rides … • 1 Misc  that • 1 Misc  bloodier • 1 Misc  does … • 1 S1  NP VP . • 1 VP  VerbT NP • 20 NP  Det N’ • 1 NP  Proper • 20 N’  Noun • 1 N’  N’ PP • 1 PP  Prep NP Any PCFG is okay

7. NP VP . 3. Now write a grammar of English Here’s one to start with. • 1 S1  NP VP . • 1 VP  VerbT NP • 20 NP  Det N’ • 1 NP  Proper • 20 N’  Noun • 1 N’  N’ PP • 1 PP  Prep NP S1 1 Sample a sentence on the blackboard Any PCFG is okay

8. 20/21 Det N’ 1/21 3. Now write a grammar of English Here’s one to start with. S1 • 1 S1  NP VP . • 1 VP  VerbT NP • 20 NP  Det N’ • 1 NP  Proper • 20 N’  Noun • 1 N’  N’ PP • 1 PP  Prep NP NP VP . Sample a sentence on the blackboard Any PCFG is okay

9. Det N’ drinks [[Arthur [across the [coconut in the castle]]] [above another chalice]] Noun every castle 3. Now write a grammar of English Here’s one to start with. S1 • 1 S1  NP VP . • 1 VP  VerbT NP • 20 NP  Det N’ • 1 NP  Proper • 20 N’  Noun • 1 N’  N’ PP • 1 PP  Prep NP NP VP . Sample a sentence on the blackboard Arbitrary PCFG is okay

10. How will we be testedon this? 4. Okay – go!

11. How will we be testedon this? 5. Evaluation procedure 4. Okay – go! • We’ll sample 20 random sentences from your PCFG. • Human judges will vote on whether each sentence is grammatical. • By the way, y’all will be the judges (double-blind). this is educational • You probably want to use the sampling script to keep testing your grammar along the way.

12. 1 S1  NP VP . • 1 VP  VerbT NP • 20 NP  Det N’ • 1 NP  Proper • 20 N’  Noun • 1 N’  N’ PP • 1 PP  Prep NP 5. Evaluation procedure • We’ll sample 20 random sentences from your PCFG. • Human judges will vote on whether each sentence is grammatical. • You’re right: This only tests precision. • How about recall? Ok, we’re done! All our sentences are already grammatical.

13. covered by initial grammar Development set You might want your grammar to generate … • Arthur is the king . • Arthur rides the horse near the castle . • riding to Camelot is hard . • do coconuts speak ? • what does Arthur ride ? • who does Arthur suggest she carry ? • why does England have a king ? • are they suggesting Arthur ride to Camelot ? • five strangers are at the Round Table . • Guinevere might have known . • Guinevere should be riding with Patsy . • it is Sir Lancelot who knows Zoot ! • either Arthur knows or Patsy does . • neither Sir Lancelot nor Guinevere will speak of it . We provide a file of 27 sample sentences illustrating a range of grammatical phenomena questions, movement, (free) relatives, clefts, agreement, subcat frames, conjunctions, auxiliaries, gerunds, sentential subjects, appositives …

14. Development set You might want your grammar to generate … • the Holy Grail was covered by a yellow fruit . • Zoot might have been carried by a swallow . • Arthur rode to Camelot and drank from his chalice . • they migrate precisely because they know they will grow . • do not speak ! • Arthur will have been riding for eight nights . • Arthur , sixty inches , is a tiny king . • Arthur knows Patsy , the trusty servant . • Arthur and Guinevere migrate frequently . • he knows what they are covering with that story . • Arthur suggested that the castle be carried . • the king drank to the castle that was his home . • when the king drinks , Patsy drinks . questions, movement, (free) relatives, clefts, agreement, subcat frames, conjunctions, auxiliaries, gerunds, sentential subjects, appositives …

15. No OOVs allowedin the test set. Fixed vocabulary. How should we parse sentences with OOV words? (= productivity!!) 5’. Evaluation of recall What we could have done: Cross-entropy on a similar, held-out test set • every coconut of his that the swallow dropped sounded like a horse .

16. In Boggle, you get points for finding words that your opponents don’tfind. Use the fixed vocabulary creatively. (= productivity!!) 5’. Evaluation of recall What we could have done (good for your class?): Cross-entropy on a similar, held-out test set What we could have done: Cross-entropy on a similar, held-out test set What we actually did, to heighten competition & creativity: Test set comes from the participants! You should try to generate sentences that your opponents can’t parse.

17. Use the fixedvocabulary creatively. Initial terminal rules • 1 Noun castle • 1 Noun king … • 1 Proper Arthur • 1 Proper Guinevere … • 1 Det a • 1 Det every … • 1 VerbT covers • 1 VerbT rides … • 1 Misc that • 1 Misc bloodier • 1 Misc does … The initial grammar sticks to 3rd-person singular transitive present-tense forms. All grammatical. But we provide 183 Misc words (not accessible from initial grammar) that you’re free to work into your grammar …

18. Use the fixedvocabulary creatively. Initial terminal rules • 1 Misc that • 1 Misc bloodier • 1 Misc does … The initial grammar sticks to 3rd-person singular transitive present-tense forms. All grammatical. But we provide 183 Misc words (not accessible from initial grammar) that you’re free to work into your grammar … pronouns (various cases), plurals, various verb forms, non-transitive verbs, adjectives (various forms), adverbs & negation, conjunctions & punctuation, wh-words, …

19. (= productivity!!) 5’. Evaluation of recall What we could have done (good for your class?): Cross-entropy on a similar, held-out test set What we actually did, to heighten competition & creativity: Test set comes from the participants! In Boggle, you get points for finding words that your opponents don’tfind. You should try to generate sentences that your opponents can’t parse.

20. (= productivity!!) 5’. Evaluation of recall What we could have done (good for your class?): Cross-entropy on a similar, held-out test set What we actually did, to heighten competition & creativity: Test set comes from the participants! We’ll score your cross-entropywhen you try to parse the sentences that the other teams generate. (Only the ones judged grammatical.) You should try to generate sentences that your opponents can’t parse. • You probably want to use the parsing script to keep testing your grammar along the way.

21. 0 probability?? You get the infinite penalty. (= productivity!!) 5’. Evaluation of recall What we could have done (you could too): Cross-entropy on a similar, held-out test set What we actually did, to heighten competition & creativity: Test set comes from the participants! We’ll score your cross-entropywhen you try to parse the sentences that the other teams generate. (Only the ones judged grammatical.) What if my grammar can’t parseone of the testsentences? So don’t do that.

22. S2 • S2   • S2  _Noun • S2 _Misc • _Noun Noun • _Noun  Noun _Noun • _Noun Noun _Misc • _Misc Misc • _Misc  Misc _Noun • _Misc Misc _Misc _Verb (etc.) Verb _Misc rides Misc _Punc ‘s Punc _Noun ! Noun swallow Use a backoff grammar : Bigram POS HMM Initial backoff grammar i.e., something that starts with a Verb _Verb i.e., something that starts with a Misc Verb _Misc . . . Misc

23. S2   • S2  _Noun • S2 _Misc • _Noun Noun • _Noun  Noun _Noun • _Noun Noun _Misc • _Misc Misc • _Misc  Misc _Noun • _Misc Misc _Misc (etc.) Use a backoff grammar : Bigram POS HMM Init. linguistic grammar Initial backoff grammar • S1  NP VP . • VP  VerbT NP • NP  Det N’ • NP  Proper • N’  Noun • N’  N’ PP • PP  Prep NP

24. Initial master grammar • START  S1 • START  S2 • S2  • S2 _Noun • S2 _Misc • _Noun Noun • _Noun  Noun _Noun • _Noun Noun _Misc • _Misc Misc • _Misc  Misc _Noun • _Misc Misc _Misc (etc.) Use a backoff grammar : Bigram POS HMM Mixturemodel Choose these weights wisely! Init. linguistic grammar Initial backoff grammar • S1 NP VP . • VP  VerbT NP • NP  Det N’ • NP  Proper • N’  Noun • N’  N’ PP • PP  Prep NP

25. 6. Discussion • What did you do? How? • Was CFG expressive enough? • How would you improve the formalism? • Would it work for other languages? • How should one pick the weights? • And how could you build a better backoff grammar? • Is grammaticality well-defined? How is it related to probability? • What if you had 36 person-months to do it right? • What other tools or data do you need? • What would the resulting grammar be good for? • What evaluation metrics are most important? features, gapping

26. 7. Winners announced

27. Helps to favor backoff grammar Anyway, a lot of work! yay unreachable 7. Winners announced • Of course, no one finishes their ambitious plans. • Alternative: Allow 2 weeks (see paper) …

28. What did they do? (see paper) • More fine-grained parts of speech • do-support for questions & negation • Movement using gapped categories • X-bar categories (following the initial grammar) • Singular/plural features • Pronoun case • Verb forms • Verb subcategorization; selectional restrictions (“location”) • Comparative vs. superlative adjectives • Appositives (must avoid double comma) • A bit of experimentation with weights • One successful attempt to game scoring system (ok with us!)

29. Why do we recommend this lesson? • Good opening activity • No programming • Only very simple probability • No background beyond linguistic intuitions • Though w/ time constraints, helps to have a linguist on the team • Works great with diverse teams • Social, intense, good mixer, sets the pace http://www.clsp.jhu.edu/grammar-writing

30. Why do we recommend this lesson? • Good opening activity • Good opening activity • Introduces many topics – touchstone for later teaching • Grammaticality • Grammaticality judgments, formal grammars, parsers • Specific linguistic phenomena • Desperate need for features, morphology, gap-passing • Generative probability models: PCFGs and HMMs • Backoff, inside probability, random sampling, … • Recovering latent variables: Parse trees and POS taggings • Evaluation (sort of) • Annotation, precision, recall, cross-entropy, … • Manual parameter tuning • Why learning would be valuable, alongside expert knowledge http://www.clsp.jhu.edu/grammar-writing

31. Akin toprogramming languages A final thought • The CS curriculum starts with programming • Accessible and hands-on • Necessary to motivate or understand much of CS • In CL, the equivalent is grammar writing • It was the traditional (pre-statistical) introduction • Our contributions: competitive game, statistics, finite-state backoff, reusable instructional materials • Much of CL work still centers around grammar formalisms • We design expressive formalisms for linguistic data • Solve linguistic problems within these formalisms • Enrich them with probabilities • Process them with algorithms • Learn them from data • Connect them to other modules in the pipeline