MPS 2016

1. PARENTAL VIEW OF CONTEXT – FREE BIRTH AND EVOLUTION MPS 2016 ELI SHAMIR, HEBREW UNIVERSITY JERUSALEM

2. Formal Languages Theory- Mid 50’s Confluence of several directions: • Natural Languages [NLP], Syntax Specifications • Early Prog. Languages, Syntax Specifications • Automata & Machine, Formal Specifications • Combinatorial Math. Sets of strings • Biological: L Systems • …

3. Formal Languages Generative HierarchyChomsky+ Subsequently integrated into space/time complexity hierarchy- the backbone of theoretical computer science. * Several sub-models studied, related to compiler constructions for programming languages.

4. Context-free [CF] central position due to: equivalence of several distinct models • Algebraic equations [MPS] DUAL APPROACH IN ARGGEMENT AND PROOFS • Production rules and trees • BNF- Backus NF, Syntax of early prog. languages • Categorical grammars • Dependency structures • Lambek algebraic calculus • Pushdown Automata… Rich algebraic, combinatorial, algorithmic properties and problems, significant applications.

5. 1957- 1963: Boston- Jerusalem Correspondence Linguists: MIT N. Chomsky Y. Bar Hillel HUJI Mathem: Harvard MPS (MARCO) H. Gaifman, M. Perles, E. Shamir Paris [Math PhD students] Main articles, monographs mainly on CF [listed next: 2-5, 19]. Up to 1969, Many other researches and groups in USA, Europe, Japan joined. See publication lists [next few slides]. Inclusion as a basic topic in CS education.

6. Central Publications up to 1969 • J. Hopcroft and J. Ullman, Formal Languages and their relations to Automata, Assidon-Wesley, 1969. [Extensive reference list] • Y. Bar-Hillel, H. Gaifman and E. Shamir, On categorical and phrase structure grammars. Bulletin research council of Israel, vol. 9f (1960), 1-16. • Y. Bar-Hillel, M. Perles and E. Shamir, On formal properties of simple phrase, structure grammars, Z. Phonetik, Sprachwiss. Kommun., 14 (1961), 143-172. 2 & 3 reproduced in Y. Bar-Hillel, Language and information, Assidon-Wesley, 1964. 3 appeared as a monograph in Russian, 1964. • N. Chomsky, On certain formal properties of grammars, Inf. and Control, 2:2 (1959), 113-124. • N. Chomsky and M. P. Schutzenberger, The algebraic theory of context-free languages, Computer Programming and Formal Systems, North Holland, 1963. [Appeared as a monograph] • J. Evey, The theory and application of pushdown store machines, Doctoral Thesis, Harvard University, 1963. • R. W. Floyd, The syntax of programming languages- a survey, Professional Group Electronic Computers [PGEC], 13: 4 (1964), 346- 353.

7. S. Ginsburg, and H. G. Rice, Two families of languages related to ALGOL, JACM, 9: 3, 350-371, 1962. • S. Ginsburg, The mathematical theory of context-free languages, 1966. • S. Greibach, A new normal form theorem for context-free grammars, JACM, 12:1, 42-52, 1965. • D. E. Knuth, a characterization of parenthesis languages, Inf. and Control, 11: 3, 269-289, 1967. • P. S. Landweber, Three theorems on phrase structure grammars of type 1, Inf. and Control, 6:2, 131- 136, 1963. • M. Nivat, Transduction des langages de Chomsky, PhD Thesis. Univ. de Paris, 1967. [Also in Annales de l’Institut Fourier, 18: 339- 456, 1968]. • R. J. Parikh, On context-free languages, JACM, 13, 570- 581, 1966. • D. J. Rosenkrantz, Matrix equations and normal forms for context-free grammars, JACM, 14:3, 501-507,1967. • J. Rhodes and E. Shamir, Complexity of grammars by group- theoretic methods, Journal of Combinatorial Theory, 222-239, 1968 • E. Shamir, A representation theorem for algebraic and context-free power series in noncommuting variables, Inf. and Control, 11, 239- 254, 1967. • M. P. Schutzenberger [Several articles: 1960-1965] • D. H. Younger, Recognition and parsing of context-free languages in time n , Inf. and Control, 10: 2, 189-208, 1967. 3

8. Chosen Books & Publications After 1970 • J. Autebert, J. Berstel and L. Boasson, Context-free language and pushdown automata. Chap. 3 In: handbook of formal languagesVol 1. G. Rozenberg and A. Salomaa (eds.), Springer-Verlag 1997. [Extensive reference list] • M. Droste, W. Kuich, H. Vogler (Eds.), Handbook of Weighted Automata, Springer 2009. • S. Greibach. The hardest context-free language. SIAM J. on computing 3 (1973), 304-310. • M. Harrison, Introduction to Formal Language Theory, Addison- Wesley, 1978. • L. Kallmeyer, Parsing Beyond Context Free Grammars, Springer, 2010. • E. Shamir, Some inherently ambiguous context-free languages. Inf. and Control 18 (1971). • J. Berstel, Transductions and context-free languages, TeubnerVerlag, 1979. • A. Salomaa, Formal Languages, Academic Press, 1973. • J. Sakarovitch, Pushdown automata with terminal languages, 421 in Publication RIMS, Kyoto University, 1981, pp. 15- 29. • S. Eilenberg, Automata, Languages and Machines, Vol. A & B, Academic Press, 1973. • G. Rozenbergand A. Salomaa. The mathematical theory of L systems, Springer 1976 . • P. Flojolet, Analytic models and ambiguity of context free languages Theor. Comp. Sci 49, 1987 283-309.

9. Hindsight of Central CF Results Chomsky- Schutzenberger Theorems:and their impact • Each CFL L= h (DykeᴖR) Dyke= {well bracketed strings}, R= regular language • A non-ambiguous L has an algebraic generating function • (Sh 1967): Each CFL maps into Non-deter. lifting of 1 sided Dyke hence it is Auniversal CFL thus a “hardest CFL”. mapa φ(a)= […+…+],φ(a1a2… an )= φ(a1)… φ(an )= =[…+…+] […+…+]… […+…+] (multinom product) wϵL(G) iffopening multinomproduct gives a term in DYKE. • (BGS 1960): Non-deter. lifting of CAT is also universal (hardest) CFL

10. (Hindsight (continued DYKE-j: All well-bracketed strings with j pairs. CAT: Well-cancelled categories-strings. a a/bb, a/b a/b/c c, a a/b/c b/c theyare determ. CFLs, their non-det. liftingsare “Hardest CFL. Algebraic path: Gauss elim-> Greib.NF->SH. Thm. & Pushdown Automat. Derivation path: triplets (p, A, q) [in BPS 1960] -> Pushdown Autom -> Greib. Normal Form and SH. Thm • Algorithm and Complexity : impact of the non-decidability results (BPS 1960). • Membership and parsing – tabular dynamic prog. algorithms (CYK, Earley ,…). • Time complexity reduced to multip. of Boolean matrices (L. Valiant, • L. Lee).

11. Ambiguity- Complex Issues • In (Linear)CFG, in Transductions, in Algeb Equations • Inherent ambiguity proofs using pumping in D - trees and by generating function method (Ph. Flajolet) • Effect of Transformations on ambiguity • Effects on Parsing of product ambiguity degree Inherently 1 or infinite? Open question • Eilenberg problem: decomposition of bounded degree language to union of 1 degree languages - open

12. Ambiguity in NLP: • Ambiguity in natural languages can be resolved (or created) by cyclic rotation of the sentence: • Bible Book of Job chapter 6 verse 14 (six Hebrew words). Translated : "a friend should extend # mercy to the sufferer$, even if he abandons God's fear." Anaphoric ambiguity: the pronoun "he" refers to the sufferer or to the friend? A poetic beautiful answer: to Both. • Cyclicrotated sentences, starting at the symbols # and $, resolve the ambiguity towards one way or the other. • Political loaded example: the policeman shot # the boy$ with the gun.

13. SRT: SPREAD - ROTATE TRANSFORMATION Of a grammar G, its trees and derived strings internal nodes labelled by prodacts of grammars: SRT TREE root label = #G, leaves labels = H(i) – linear grammars Thm ( invariance claim) 1-1 onto U{D – trees of H(i)} D - trees of #Gmapped Mod. Cyclic rotations (of trees and derives strings)But Works perfect for non – expansive CF grammars (quasi-rational) but also for mild context – sensitive with CF skeleton (E.G.LIG grammars) SRT: enhance parsing alg , property tests, and applications cosmetics of the CFG model to enhance its NLP adequacy: *Avoid expansive pumping B B B BUT ADD GENER. POWER BY LOCAL STACKS (AS IN INDEXED GRAMMARS)

14. Top Trunk Rotation of MN to (M*N^) N M* for trees: M M x1 180 y1 x2 x2 N^ y2 y2 x1 y1 EXIT N^ Cyclic rotation of derived strings: m x1x2 … n^ …y2y1 …y2y1 m x1x2 … n^

15. SRT For grammars: N grammar (top trunk) M* grammar BB’C B’CB BDB’ B’BD BB^, B^α B^= root(M) All productions not involving [B] carry over from N to M*; those of M unchanged. Note: Since M may contain symbols of [B] duplicate symbols [B] needed only for the new top trunk of M* The TTR rotation is invertible, one-one onto for the derivation trees, preserving weights andambiguity degree in ‘cyclic rotated’ sense.

16. Example (from [Sh., 1971]) R R • (M)(N)= (u$Ju ) (vJ$v), u, v ε {0.1}* = J u= reversal of u, • It has unbounded "direct (product) ambiguity" which increases time in CYK algorithm to n In one TTR step (see below)MN is rotated to • (M*)(N^)= (v u$ Juv ) (J $) , which has a linear grammar, with 3 pump classes. All (product ambiguity) trees are rotated to (union ambiguity)trees for M*N^. Each derived terminal string is CYC-rotated as well! R 3 R R

17. MILD Context-Sensitive Models & SRT Many models proposed incl. 4 equivalent ones: Linear-Index [LIG], Tree-Adjoint [TAG],…. Should satisfy some formal requirements: Proper extension of CFG, Poly-time parsing algor… We define NE-LIG as follows: Has NE-CFG skeleton aux. symbols A, B,… Each pump-class [B] maintains stack (pushdown) index, stack empty at enter & Exit of several consecutive pump blocks- THUS, it can, with skeleton -symbols as “states”, simulate any PDM, any CFG. The form of production rules is: B[index] C B’[index’] , Bˆ[ ] D[ ] E [ ] Push Pop

18. Glossary CFG/L- Context Free Grammars/Language LIG- Linear Indexed Grammar TAG- Tree Adjoining Grammar NLP- Natural Language Processing CYK- Cocke, Younger, Kasami CNF- Chomsky Normal Form GNF- Greibach Normal Form SRT- Spread Rotate Tree D-Tree- Derivation Tree EPOS- Epoch Semi-Order TTR- Top Trunk Rotation DP- Dynamic Programming NE- Non Expansive POS- Parts of Speech PDM- Pushdown Machine NT- Non terminals (symbols)