Lecture #7, Feb. 5, 2007

1. Lecture #7, Feb. 5, 2007 • Grammars • Top down parsing • Transition Diagrams • Ambiguity • Left recursion • Refactoring by adding levels • Recursive descent parsing • Predictive Parsers • First and Follow • Parsing tables

2. Assignments • Reading • Chapter 3 • Page 73-106 • Quiz on Wednesday • Mid Term exam • Monday. Feb 19, 2007. Time: in class. • Next Homework • On the web page, and the last page of this handout • Due date to be negotiated. • Recall Project #1 is due next Wednesday. • I promised no homework Today or Wednesday. • Project 1 • Recall Project #1, the scanner is due Feb. 14th

3. Grammars 1 • Grammar • A set of tokens (terminals): T • A set of non-terminals: N • A set of productions { lhs ::= rhs , ... } • lhs in N • rhs is a sequence of N U T • A Start symbol: S (in N) • Shorthands • Provide only the productions • All lhs symbols comprise N • All other sysmbols comprise T • lhs of first production is S

4. Grammars 2 • Rewriting rules • Pick a non-terminal to replace. Which order? • left-to-right • right-to-left • Derivations (a list if productions used to derive a string from a grammar). • A sentence of G: L(G) • Start with S • only terminal symbols • all strings derivable from G in 1 or more steps

5. Grammars 3 • Parse trees. • Graphical representations of derivations. • The leaves of a parse tree for fully filled out tree is a sentence. • Context Free Grammars • how do they compare to regular expressions? • Nesting (matched ()’s) requires CFG,’s RE's are not powerful enough. • Ambiguity • A string has two derivations • E ::= E + E | E * E | id • x + x * y • Left-recursion • E ::= E + E | E * E | id • Makes certain top-down parsers loop

6. Top Down Parsing • Begin with the start symbol and try and derive the parse tree from the root. • Consider the grammar Exp ::= id | Exp + Exp | Exp * Exp | ( Exp ) derives x, x+x, x+x+x, x * y x + y * z ...

7. Example Parse (top down) • stack input Exp x + y * z Exp x + y * z / | \ Exp + Exp Exp y * z / | \ Exp + Exp | id(x)

8. Top Down Parse (cont) Exp y * z / | \ Exp + Exp | / | \ id(x) Exp * Exp Exp z / | \ Exp + Exp | / | \ id(x) Exp * Exp | id(y)

9. Top Down Parse (cont.) Exp / | \ Exp + Exp | / | \ id(x) Exp * Exp | | id(y) id(z)

10. T E E’ T E’ E’ + F T T’ F T’ * T’ ( ) E F id Transition Diagrams • Transition diagrams for predictive parsers • One diagram for each Non-terminal • Shouldn't have left recursion ( left factored ) • Diagrams can (recursively) mention each other E -> T E' E' -> + T E' | <empty> T -> F T' T' -> * F T' | <empty> F -> ( E ) | id

11. Problems with Top Down Parsing • Backtracking may be necessary: • S ::= ee | bAc | bAe • A ::= d | cA try on string “bcde” • Infinite loops possible from (indirect) left recursive grammars. • E ::= E + id | id • Ambiguity is a problem when a unique parse is not possible. • These often require extensive grammar restructuring (grammar debugging).

12. Grammar Transformations • Removing ambiguity. • Removing Left Recursion • Backtracking and Factoring

13. Removing ambiguity. • Adding levels to a grammar • E := E + E | E * E | id | ( E ) • E ::= E + T | T • T ::= T * F | F • F ::= id | ( E ) • The dangling else grammar. • st ::= if exp then st else st | if exp then st | id := exp • Note that the following has two possible parses if x=2 then if x=3 then y:=2 else y := 4 if x=2 then (if x=3 then y:=2 ) else y := 4 if x=2 then (if x=3 then y:=2 else y := 4)

14. Adding levels (cont) • Original grammar st ::= if exp then st else st | if exp then st | id := exp • Assume that every st between then and else must be matched, i.e. it must have both a then and an else. • New Grammar with addtional levels st -> match | unmatch match -> if exp then match else match | id := exp unmatch -> if exp then st | if exp then match else unmatch

15. T T n T n n T a Removing Left Recursion • Top down recursive descent parsers require non-left recursive grammars • Technique: Left Factoring • E := E + E | E * E | id • E ::= id E’ • E’ ::= + E E’ | * E E’ | ε • General Technique to remove direct left recursion • Every Non terminal with productions • T ::= T n | T m(left recursive productions) | a | b(non-left recursive productions) • Make a new non-terminal T’ • Remove the old productions • Add the following productions • T ::= a T’ | b T’ • T’ ::= n T’ | m T’ | ε (a | b) (n | m) * “a” and “b” because they are the rhs of the non-left recurive productions.

16. Backtracking and Factoring • Backtracking may be necessary: • S ::= ee | bAc | bAe • A ::= d | cA • try on string “bcde” S -> bAc(by S -> bAc) -> bcAe(by A -> cA) -> bcde(by A -> d) • But this is the wrong answer! • Factoring a grammar • Factor common prefixes and make the different postfixes into a new non-terminal • S ::= ee | bAQ • Q ::= c | e • A ::= d | cA

17. Recursive Descent Parsing • One procedure (function) for each non-terminal. • Procedures are often (mutually) recursive. • They can return a bool (the input matches that non-terminal) or more often they return a data-structure (the input builds this parse tree) • Need to control the lexical analyzer (requiring it to “back-up” on occasion)

18. R.D. parser for R.E.’s • Build an instance of the datatype: datatype Re = empty of int | simple of string * int | concat of Re * Re | closure of Re | union of Re * Re; • The lexical Analyzer datatype datatype token = Done | Bar | Star | Hash | Leftparen | Rightparen | Single of string;

19. Ambiguous grammar • RE -> RE bar RE • RE -> RE RE • RE -> RE * • RE -> id • RE -> # • RE -> ( RE ) • Transform grammar by layering • Tightest binding operators (*) at the lowest layer • Layers are Alt, then Concat, then Closure, then Simple. Alt -> Alt bar Concat Alt -> Concat Concat -> Concat Closure Concat -> Closure Closure -> simple star Closure -> simple simple -> id | (Alt ) | #

20. Left Recursive Grammar Alt -> Alt bar Concat Alt -> Concat Concat -> Concat Closure Concat -> Closure Closure -> simple star Closure -> simple simple -> id | (Alt ) | # Alt ::= Concat moreAlt moreAlt ::= Bar Concat moreAlt | ε Concat ::= Closure moreConcat moreConcat ::= Closure moreConcat | ε Closure ::= Simple Star | Simple Simple ::= Id | ( Alt ) | # • For every Non terminal with productions • T ::= T n | T m (left recursive productions) • | a | b (non-left recursive productions) • Make a new non-terminal T’ • Remove the old productions • Add the following productions • T ::= a T’ | b T’ • T’ ::= n T’ | m T’ | ε

21. Lookahead and the Lexer val lookahead = ref Done; val input = ref [Done]; val location = ref 0; fun nextloc () = (location := (!location) + 1; !location) fun init s = (location := 0; input := lexan s; lookahead := hd(!input); input := tl(!input)) • Lex’s the whole input • Stores it in the variable input • Keeps track of next token (so that backup is possible)

22. Matching a single Terminal fun match t = if (!lookahead) = t then (if null(!input) then lookahead := Done else (lookahead := hd(!input) ; input := tl(!input))) else error ("looking for: "^(tok2str t)^ " found: "^(tok2str (!lookahead))); • Match one token • Advance the input • Handle the end of file correctly • Report errors in a sensible way • This function will be called a lot!!

23. moreConcat moreConcat closure closure moreConcat ... ... ε moreAlt and moreConcat When we removed left recursion, we added non-terminals that that might recognize ε. i.e. moreAlt and moreConcat Observe the shape of parse trees using those productions. moreConcat ::= Closure moreConcat | <empty> They always end in ε at the far right of the tree

24. 1 Function for each NT • A simple way to write a parser for a language is the technique called recursive descent parsing. • Each Non-terminal is represented by a function that returns a syntax item corresponding to the element that production parses. • If it can’t parse that element it raises an error. • When a production might return the empty string we need to handle that by using the SML ‘a option datatype. Alt : unit -> Re moreAlt : unit -> Re option Concat : unit -> Re Closure : unit -> Re moreConcat : unit -> Re option Simple : unit -> Re Alt ::= Concat moreAlt moreAlt ::= Bar Alt moreAlt | ε Concat ::= Closure moreConcat moreConcat ::= Closure moreConcat | ε Closure ::= Simple Star | Simple Simple ::= Id | ( Alt ) | #

25. Alt ::= Concat moreAlt fun Alt () = let val x = Concat () val y = moreAlt () in case y of NONE => x | SOME z => union(x,z) end

26. moreAlt ::= Bar Alt moreAlt | ε and moreAlt () = case (!lookahead) of Bar => let val _ = match Bar val x = Alt() val y = moreAlt () in case y of NONE => SOME x | (SOME z) => SOME(union(x,z)) end | _ => NONE “and” separates mutually recursive functions

27. Concat ::= Closure moreConcat and Concat () = let val x = Closure () val y = moreConcat () in case y of NONE => x | SOME z => concat(x,z) end

28. moreConcat ::= Closure moreConcat | ε and moreConcat () = case (!lookahead) of (Single _ | Leftparen | Hash) => let val x = Closure() val y = moreConcat() in case y of NONE => SOME x | SOME z => SOME(concat(x,z)) end | _ => NONE

29. Closure ::= Simple Star | Simple and Closure () = let val x = Simple() in case !lookahead of Star => (match Star; closure x) | other => x end A simple form of left-factoring is used here

30. Simple ::= Id | ( Alt ) | # and Simple () = case !lookahead of Single c => let val _ = match (Single c) val n = nextloc() in simple(c,n) end | Leftparen => let val _ = match Leftparen val x = Alt(); val _ = match Rightparen in x end | Hash => let val _ = match Hash val n = nextloc() in empty n end | x => error ("In Simple no match: "^(tok2str x));

31. Top Level Parser fun parse s = let val _ = init s val ans = Alt() val _ = match Done in ans end; parse "a(b*|c)#"; concat (simple ("a",1), concat (union (closure (simple ("b",2)), simple ("c",4)), empty 8))

32. Predictive Parsers • Using a stack to avoid recursion. Encoding the diagrams in a table • The Nullable, First, and Follow functions • Nullable: Can a symbol derive the empty string. False for every terminal symbol. • First: all the terminals that a non-terminal could possibly derive as its first symbol. • term or nonterm -> set( term ) • sequence(term + nonterm) -> set( term) • Follow: all the terminals that could immediately follow the string derived from a non-terminal. • non-term -> set( term )

33. Example First and Follow Sets E ::= T E' $ E' ::= + T E' E’ ::= ε T ::= F T' T' ::= * F T' T’ ::= ε F ::= ( E ) F ::= id First E = { "(", "id"} Follow E = {")","$"} First F = { "(", "id"} Follow F = {"+","*",”)”,"$"} First T = { "(", "id"} Follow T = {{"+",")","$"} First E' = { "+", ε} Follow E' = {")","$"} First T' = { "*", ε} Follow T' = {"+",")","$"} • First of a terminal is itself. • First can be extended to sequence of symbols.

34. Nullable • if ε is in First(symbol) then that symbol is nullable. • Sometime rather than let ε be a symbol we derive an additional function nullable. • Nullable (E’) = true • Nullable(T’) = true • Nullable for all other symbols is false E ::= T E' $ E' ::= + T E' E’ ::= ε T ::= F T' T' ::= * F T' T’ ::= ε F ::= ( E ) F ::= id

35. Computing First • Use the following rules until no more terminals can be added to any FIRST set. 1) if X is a term. FIRST(X) = {X} 2) if X ::= ε is a production then add ε to FIRST(X), (Or set nullable of X to true). 3) if X is a non-term and • X ::= Y1 Y2 ... Yk • add a to FIRST(X) • if a in FIRST(Yi) and • for all j<i ε in FIRST(Yj) • E.g.. if Y1 can derive ε then if a is in FIRST(Y2) it is surely in FIRST(X) as well.

36. Example First Computation • Terminals • First($) = {$} First(*) = {*} First(+) = {+} ... • Empty Productions • add ε to First(E’), add ε to First(T’) • Other NonTerminals • Computing from the lowest layer (F) up • First(F) = {id , ( } • First(T’) = { ε, * } • First(T) = First(F) = {id, ( } • First(E’) = { ε, + } • First(E) = First(T) = {id, ( } E ::= T E' $ E' ::= + T E' E’ ::= ε T ::= F T' T' ::= * F T' T’ ::= ε F ::= ( E ) F ::= id

37. Computing Follow • Use the following rules until nothing can be added to any follow set. 1) Place $ (the end of input marker) in FOLLOW(S) where S is the start symbol. 2) If A ::= a B b then everything in FIRST(b) except ε is in FOLLOW(B) 3) If there is a production A ::= a B or A ::- a B b where FIRST(b) contains ε (i.e. b can derive the empty string) then everything in FOLLOW(A) is in FOLLOW(B)

38. Ex. Follow Computation • Rule 1, Start symbol • Add $ to Follow(E) • Rule 2, Productions with embedded nonterms • Add First( ) ) = { ) } to follow(E) • Add First($) = { $ } to Follow(E’) • Add First(E’) = {+,ε } to Follow(T) • Add First(T’) = {*,ε} to Follow(F) • Rule 3, Nonterm in last position • Add follow(E’) to follow(E’) (doesn’t do much) • Add follow (T) to follow(T’) • Add follow(T) to follow(F) since T’ --> ε • Add follow(T’) to follow(F) since T’ --> ε • E ::= TE' $ • E' ::= + T E' • E’ ::= ε • T ::= F T ' • T' ::= * F T' • T’ ::= ε • F ::= ( E ) • F ::= id

39. M[A,t] terminals + * ) ( id $ E n o n t e r m s 1 1 2 3 3 E’ 4 4 T 6 5 6 6 T’ 7 8 F Table from First and Follow • For each production A -> alpha do 2 & 3 • For each a in First alpha do add A -> alpha to M[A,a] • if ε is in First alpha, add A -> alpha to M[A,b] for each terminal b in Follow A. If ε is in First alpha and $ is in Follow A add A -> alpha to M[A,$]. First E = {"(","id"} Follow E = {")","$"} First F = {"(","id"} Follow F = {"+","*",”)”,"$"} First T = {"(","id"} Follow T = {{"+",")","$"} First E' = {"+",ε} Follow E' = {")","$"} First T' = {"*",ε} Follow T' = {"+",")","$"} 1 E ::= T E' $ 2 E' ::= + T E' 3 | ε 4 T ::= F T' 5 T' ::= * F T' 6 | ε 7 F ::= ( E ) 8 | id

40. + * ( ) id $ T E’ T E’ E + T E’ ε E’ ε T F T’ F T’ T’ ε * F T’ ε ε F id ( E ) Predictive Parsing Table

41. Table Driven Algorithm push start symbol Repeat begin let X top of stack, A next input if terminal(X) then if X=A then pop X; remove A else error() else (* nonterminal(X) *) begin if M[X,A] = Y1 Y2 ... Yk then pop X; push Yk YK-1 ... Y1 else error() end until stack is empty, input = $

42. + * ( ) id $ T E’ T E’ E + T E’ ε E’ ε T F T’ F T’ T’ ε * F T’ ε ε F id ( E ) Example Parse Stack Input E x + y $ E’ T x + y $ E’ T’ F x + y $ E’ T’ id x + y $ E’ T’ + y $ E’ + y $ E’ T + + y $ E’ T y $ E’ T’ F y $ E’ T’ id y $ E’ T’ $ E’ $ $

43. CS321 Prog Lang & Compilers Assignment # 7 Assigned: Feb 5, 2007 Due: Wed. Feb 14, 2007 Cut and paste the following into your solution file. ================================================================== datatype RE = Empty | Union of RE * RE | Concat of RE * RE | Star of RE | C of char; ================================================================ The purpose of today's home work is to write functions analogous to "first" and "follow" for context free grammars from todays lecture. There are two differences, The functions you will write for homework are for regular expressions, not context free grammars, and since REs don't have non-terminal and terminal symbols, the functions are for a complete RE rather than a symbol. Write 3 ML functions 1) Write (nullable: RE -> boolean) Returns a boolean, true if the empty string is a member of the set of strings recognized by the RE, false otherwise. 2) Write (first: RE -> char list) Returns a (char list) which contains those characters which may appear as the first character in the strings recognized by that RE. 3) Write (last: RE -> char list). Returns a char list which contains those characters which may appear as the last character in the strings recognized by that RE. All these functions a simple functions defined with pattern matching. One clause for each constructor of RE.