Lecture #9, Feb. 12, 2007

1. Lecture #9, Feb. 12, 2007 • A data structure for grammars • Computing Nullable and First in SML

2. Assignments • Project 1 • Recall Project #1, the scanner is due Wednesday Feb. 14th • Midterm Exam. Recall the midterm is scheduled for 1 week from Today.

3. fun inputc fname = (fn n => TextIO.inputN(TextIO.openIn fname,n):string); val lex = Mlex.makeLexer (inputc "test.english"); fun inputc h = (fn n => TextIO.inputN(h,n):string); val lex = let val h = TextIO.openIn “test.english” in Mlex.makeLexer (inputc h) end;

4. Datatype for grammars val empty = "empty"; fun emptyP x = x=empty; datatype Grammar = Gram of (string list) * (* Non Terminals *) (string list) * (* Terminals *) (string * (string list)) list * (* Productions *) string; (* Start symbol *)

5. Example Grammar val g1 = Gram(["Expr","Term","Factor","Expr'","Term'"] ,["ident","+","-","*","div","(",")","num"] ,[("Expr",["Term","Expr'"]) ,("Expr'",["+","Term","Expr'"]) ,("Expr'",["-","Term","Expr'"]) ,("Expr'",[]) ,("Term",["Factor","Term'"]) ,("Term'",["*","Factor","Term'"]) ,("Term'",["div","Factor","Term'"]) ,("Term'",[]) ,("Factor",["(","Expr",")"]) ,("Factor",["num"]) ,("Factor",["ident"]) ] ,"Expr");

6. Operations on Grammars fun termP (Gram(nts,ts,ps,s)) x = List.exists (fn y => y=x) ts; fun nontermP (Gram(nts,ts,ps,s)) x = List.exists (fn y => y=x) nts; fun rhssFor (Gram(nts,ts,ps,start)) symbol = let fun test (lhs,rhs) = lhs=symbol fun rhs (l,r) = r in List.map rhs (List.filter test ps) end;

7. Tables • Nullable, First, and Follow compute a table by incrementally adding stuff, until no new stuff can be added. • A table is simply a list of (string * a ref) pairs (* Create a table with all slots initialized by the function “initf” *) fun init symbols initf = let fun f x = (x,ref (initf x)) in List.map f symbols end;

8. Updating • When we update a table we need to know if the update makes a change. Since we continue until no new changes are possible. (* Update the slot at "s" with the function "updatef" *) (* returns true if it makes a change, false otherwise *) fun update s updatef pairs = case List.find (fn (y,r) => s=y) pairs of SOME(_,r) => let val old = !r val new = updatef old in if new=old then false else (r := new; true) end | NONE => false

9. Basic Strategy let val table = init … symbols val changed = ref true in while (!changed) do (changed := false ; changed := onePass symbols); table end • Initialize table • Record changes • Apply rules until no changes are made in a single pass.

10. Code fun nullTable (gram as(Gram(nts,ts,ps,start))) = let val allsymbols = nts @ ts val table = init allsymbols (fn _ => false) val changed = ref true fun onePass [] = false | onePass (x::xs) = let val rhss = rhssFor gram x fun nullify b = List.exists (nullableRhs table) rhss val b1 = update x nullify table val b2 = onePass xs in b1 orelse b2 end in while (!changed) do (changed := false; changed := onePass allsymbols); table end

11. and nullable table s = case List.find (fn (y,r) => y=s) table of NONE => false | SOME(_,r) => !r and nullableRhs table [] = true | nullableRhs table (x::xs) = (nullable table x) andalso (nullableRhs table xs);

12. val NullT = nullTable g1; - NullT; val it = [("Expr",ref false), ("Term",ref false), ("Factor",ref false), ("Expr'",ref true), ("Term'",ref true), ("ident",ref false), ("+",ref false), ("-",ref false), ("*",ref false), ("div",ref false), ("(",ref false), (")",ref false), ("num",ref false)] : (string * bool ref) list

13. Onepass for “first” (* returns true if it makes a change *) fun onePass [] = false | onePass (x::xs) = let val rhss = rhssFor gram x fun first old = let val listOflists = map (firstRhs gram table) rhss val new = List.concat listOflists in norm(old @ new) end val b1 = update x first table val b2 = onePass xs in b1 orelse b2 end

14. First for a RHS and first table s = case List.find (fn (y,r) => y=s) table of NONE => [] | SOME(_,r) => !r and firstRhs gram table [] = [empty] | firstRhs gram table [x] = first table x | firstRhs gram table (x::xs) = let val temp = first table x in case List.find emptyP temp of NONE => temp | SOME _ => temp @ firstRhs gram table xs end;

15. The main loop fun firstTable (gram as(Gram(nts,ts,ps,start))) = let val termTable = init ts (fn x => [x]) val nontermTable = init nts (fn x => []) val table = termTable @ nontermTable val changed = ref true fun onePass [] = false | onePass (x::xs) = let val rhss = rhssFor gram x fun first old = let val listOflists = map (firstRhs gram table) rhss val new = List.concat listOflists in norm(old @ new) end val b1 = update x first table val b2 = onePass xs in b1 orelse b2 end in while (!changed) do (changed := false; changed := onePass nts); table end

16. val FirstT = firstTable g1; val it = [("ident",ref ["ident"]) ,("+",ref ["+"]) ,("-",ref ["-"]) ,("*",ref ["*"]) ,("div",ref ["div"]) ,("(",ref ["("]) ,(")",ref [")"]) ,("num",ref ["num"]) ,("Expr",ref ["(","ident","num"]) ,("Term",ref ["(","ident","num"]) ,("Factor",ref ["(","ident","num"]) ,("Expr'",ref ["+","-","empty"]) ,("Term'",ref ["*","div","empty"]) ] : (string * string list ref) list

17. Possible exam questions • Define a grammar. • Prove that a particular grammar is ambiguous. • i.e. find a single string accepted by the grammar with more than 1 parse. • Give a regular expression that describes comments (or some other given lexeme). • Translate a regular expression into a finite state automata • where the FSA has <empty> transitions • where the FSA has no <empty> transitions • Be able to construct an abstract (inductive) data definition for binary trees (or some other data type)

18. Possible exam questions (cont. 2) • Write a function over an abstract (inductive) data type by using patterns. • Produce the First, and Last sets from a regular expression. • Simulate a top-down parse of a string by hand for a particular grammar. • Remove left recursion from a grammar. • Layer an ambiguous grammar to produce non-ambigous grammar.

19. Possible exam questions (cont. 3) • Write a recursive descent parser for a simple grammar. • Construct the First and Follow sets from a context free grammar • Given the first and follow sets, construct a predictive parsing table for a given grammar. Tell if such a grammar is ambiguous. • Simulate a bottom up (shift-reduce) parse by hand using a stack and an input stream, for a given string and a given grammar. • Disambiguate a shift reduce parser by using precedence and associativity information.

20. Possible exam questions (cont. 4) • Write simple anonymous function in ML • Use the List library functions List.map, List.filter, List.exists, Lists.all, List.find to write simple functions.