Computer Science 112

1. Computer Science 112 Fundamentals of Programming II Recursive Processing of Languages

2. Languages and Grammars • A grammar specifies the rules for constructing well-formed sentences in a language • Every language, including a programming language, has a grammar

3. Applications • Grammar checkers in word processors • Programming language compilers • Natural language queries (Google, etc.)

4. Generate Sentences in English • Given a vocabulary and grammar rules, one can generate some random and perhaps rather silly sentences • Vocabulary - the set of words belonging to the parts of speech (nouns, verbs, articles, prepositions) • Grammar - the set of rules for building phrases in a sentence (noun phrase, verb phrase, prepositional phrase)

5. The Structure of a Sentence sentence noun phrase verb phrase A sentence is a noun phrase followed by a verb phrase

6. The Structure of a Sentence sentence noun phrase verb phrase article noun A noun phrase is an article followed by a noun

7. The Structure of a Sentence sentence noun phrase verb phrase article noun the girl Similar to the behavior of strings so far Pick actual words for those parts of speech at random

8. The Structure of a Sentence sentence noun phrase verb phrase article noun verb noun phrase prepositional phrase the girl Similar to the behavior of strings so far A verb phrase is a verb followed by a noun phrase and a prepositional phrase

9. The Structure of a Sentence sentence noun phrase verb phrase article noun verb noun phrase prepositional phrase the girl hit Similar to the behavior of strings so far Pick a verb at random

10. The Structure of a Sentence sentence noun phrase verb phrase article noun verb noun phrase prepositional phrase article noun the girl hit Similar to the behavior of strings so far Expand a noun phrase again

11. The Structure of a Sentence sentence noun phrase verb phrase article noun verb noun phrase prepositional phrase article noun the girl hit the boy Similar to the behavior of strings so far Pick an article and a noun at random

12. The Structure of a Sentence sentence noun phrase verb phrase article noun verb noun phrase prepositional phrase article noun preposition noun phrase the girl hit the boy Similar to the behavior of strings so far A prepositional phrase is a preposition followed by a noun phrase

13. The Structure of a Sentence sentence noun phrase verb phrase article noun verb noun phrase prepositional phrase article noun preposition noun phrase the girl hit the boy with Similar to the behavior of strings so far Pick a preposition at random

14. The Structure of a Sentence sentence noun phrase verb phrase article noun verb noun phrase prepositional phrase article noun preposition noun phrase article noun the girl hit the boy with Similar to the behavior of strings so far Expand another noun phrase

15. The Structure of a Sentence sentence noun phrase verb phrase article noun verb noun phrase prepositional phrase article noun preposition noun phrase article noun the girl hit the boy with a bat Similar to the behavior of strings so far More random words from the parts of speech

16. Representing the Vocabulary nouns = ['bat', 'boy', 'girl', 'dog', 'cat', 'chair', 'fence', 'table', 'computer', 'cake', 'field'] verbs = ['hit', 'threw', 'pushed', 'ate', 'dragged', 'jumped'] prepositions = ['with', 'to', 'from', 'on', 'below', 'above', 'beside'] articles = ['a', 'the'] Use a list of words for each part of speech (lexical category)

17. Picking a Word at Random nouns = ['bat', 'boy', 'girl', 'dog', 'cat', 'chair', 'fence', 'table', 'computer', 'cake', 'field'] verbs = ['hit', 'threw', 'pushed', 'ate', 'dragged', 'jumped'] prepositions = ['with', 'to', 'from', 'on', 'below', 'above', 'beside'] articles = ['a', 'the'] import random print(random.choice(verbs)) # Prints a randomly chosen verb The random module includes functions to select numbers, sequence elements, etc., at random

18. Grammar Rules sentence = nounphrase verbphrase nounphrase = article noun verbphrase = verb nounphrase prepositionalphrase prepositonalphrase = preposition nounphrase A sentence is a noun phrase followed by a verb phrase Etc., etc.

19. Define a Function for Each Rule # sentence = nounphrase verbphrase def sentence(): return nounphrase() + ' ' + verbphrase() Each function builds and returns a string that is an instance of the phrase Separate phrases and words with a space

20. Define a Function for Each Rule # sentence = nounphrase verbphrase def sentence(): return nounphrase() + ' ' + verbphrase() # nounphrase = article noun def nounphrase(): return random.choice(articles) + ' ' + random.choice(nouns) When a part of speech is reached, select an instance at random from the relevant list of words

21. Call sentence() to Try It Out # sentence = nounphrase verbphrase def sentence(): return nounphrase() + ' ' + verbphrase() # nounphrase = article noun def nounphrase(): return random.choice(articles) + ' ' + random.choice(nouns) … for x in range(10): print(sentence()) # Display 10 sentences You can also generate examples of the other phrases by calling their functions

22. Kinds of Symbols in a Grammar • Terminal symbols: words in the vocabulary of the language • Non-terminal symbols: words that describe phrases or portions of sentences • Metasymbols: used to construct rules

23. Metasymbols for a Grammar Metasymbols Use "" Enclose literal items = Means "is defined as" [ ] Enclose optional items { } Enclose zero or more items ( ) Group together required choices | Indicates a choice

24. A Grammar of Arithmetic Expressions expression = term { addingOperator term } term = factor { multiplyOperator factor } factor = primary ["^" primary ] primary = number | "(" expression ")" number = digit { digit } digit = "0" | "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9" addingOperator = "+" | "-" multiplyingOperator = "*" | "/" Example sentences:3, 4 + 5, 5 + 2 * 3, (5 + 2) * 3 ^ 4

25. * / ( ) number factor expression Alternative Notation: Train Track term = factor { multiplyingOperator factor } primary = number | "(" expression ")"

26. Parser Parsing • A parser analyzes a source program to determine whether or not it is syntactically correct Source language program OK or not OK Syntax error messages

27. Scanner Parser Scanning • A scanner picks out words in a source program and sends these to the parser Source language program Tokens Ok or not OK Lexical error messages Syntax error messages

28. The Scanner Interface Scanner(aString) Creates a scanner on a source string get() Returns the current token (at the cursor) next() Advances the cursor to the next token

29. Tokens • A Token object has two attributes: • type (indicating an operand or operator) • value (an int if it’s an operand, or the source string otherwise) • Token types are • Token.EOE • Token.PLUS, Token.MINUS • Token.MUL, Token.DIV • Token.INT • Token.UNKNOWN

30. The Token Interface Token(source) Creates a token from a source string str(aToken) String representation isOperator() True if an operator, false otherwise getType() Returns the type getValue() Returns the value

31. Recursive Descent Parsing • Each rule in the grammar translates to a Python parsing method expression = term { addingOperator term } defexpression(self): self.term() token = self.scanner.get() whiletoken.getType() in (Token.PLUS, Token.MINUS): self.scanner.next() self.term() token = self.scanner.get()

32. Recursive Descent Parsing • Each method is responsible for a phrase in an expression term = factor { multiplyingOperator factor } defterm(self): self.factor() token = self.scanner.get() whiletoken.getType() in (Token.MUL, Token.DIV): self.scanner.next() self.factor() token = self.scanner.get()

33. Recursive Descent Parsing primary = number | "(" expression ")" defprimary(self): token = self.scanner.get() iftoken.getType() == Token.INT: self.scanner.next() eliftoken.getType() == Token.L_PAR: self.scanner.next() self.expression() self.accept(self._scanner.get(), Token.R_PAR, "')' expected") self.scanner.next() else: self.fatalError(token, "bad primary")

34. For Monday Expression Trees