COGN1001: Introduction to Cognitive Science Topics in Computer Science Formal Languages and Models of Computation

COGN1001: Introduction to Cognitive ScienceTopics in Computer Science Formal Languages and Models of Computation Qiang HUO Department of Computer Science The University of Hong Kong (E-mail: qhuo@cs.hku.hk)

Outline • What is a Formal Language? • Phrase-Structure Grammars • Finite State Automata • Formal languages and Models of Computation

Natural Language vs. Formal Language • Natural language: written and/or spoken languages in the world, suchas Chinese, English, Japanese, German, French, Spanish, etc. • Syntax • Semantics • Formal language: a language specified by a well-defined set of rules ofsyntax. • A study of formal languages is important to computer science. • For example,we need to understand what kind of statements are acceptable in the Cprogramming language. This is the task of acompiler of a programminglanguage.

Formal Language • We will describe the sentences of a formal language using agrammar. • How can we determine whether a combination of words is a validsentence in a formal language? • How can we generate the valid sentences of a formal language? • We will only be interested in thesyntax, not thesemantics(meaning), of alanguage.

a sentence is made up of a noun-phrase followed by a verb-phrase; • a noun-phrase is made up of an article followed by an adjective followed by a noun, or • a noun-phrase is made up of an article followed by a noun; • a verb-phrase is made up of a verb followed by an adverb, or • a verb-phrase is made up of a verb; • an article isa, or • an article isthe; • an adjective islarge, • an adjective ishungry; • a noun israbbit, or • a noun ismathematician; • a verb iseats, or • a verb ishops; • an adverb isquickly, or • an adverb iswildly. If we define a subset of English using the list of rules shown here that describe how a validsentence can be produced, how the language looks like?

Example: a Subset of English • From the previous rules we can form valid sentences using a series ofreplacementsuntil no more rules can be used. • For instance, the valid sentencethe large rabbit hops quicklycan beobtained by the following sequence of replacements: • sentence • noun-phrase verb-phrase • article adjective noun verb-phrase • article adjective noun verb adverb • theadjective noun verb adverb • the largenoun verb adverb • the large rabbitverb adverb • the large rabbit hopsadverb • the large rabbit hops quickly • Some other valid sentences: • a hungry mathematician eats wildly • the rabbit eats quickly • An invalid sentence:the quickly eats mathematician

Some Terminologies • Avocabulary(oralphabet)Vis a finite, nonempty set ofelementscalledsymbols. • Aword(orsentence) overVis a string of finite length of elements ofV . • Theempty stringornull string, denoted by , is thestring containingno symbols. • The set of all words (orsentences) over V is denoted by V*. • AlanguageoverVis a subset of V* . • Example: In English, • ThealphabetV consists of English letters and other symbols. • Aword(orsentence) overVis a finite string of symbols. • The meaningful word(orsentence) of English is a subset ofV* .

How to specify a language? • to list all the words (or sentences) in the language; or • to give some criteria that a word (or a sentence) must satisfy to be in the language;or • to specify a language through the use of agrammar, such as the setof rules we gave in the previous example of English subset.

What is a Phrase-Structure Grammar? • Aphrase-structure grammaris G = (V,T,S,P), where • V is a vocabulary; • T is a subset of V consisting of terminal elements (i.e., the elementsof V which can not be replaced by othersymbols); • The elements of N = V–Tare callednonterminal symbols(i.e., the elements ofV which can be replaced by other symbols) • S is astart symbolfrom V (i.e., the element of the V that we alwaysbegin with; • P is a set ofproductions. • We denote by w0w1 the production that specifies that w0 canbe replaced by w1. • Every production in P must contain at least one nonterminal onits left side.

Example: a Phrase-Structure Grammar • G = (V,T,S,P), where • V = { a,the,large,hungry,rabbit,mathematician,eats,hops,quickly,wildly; sentence, noun-phrase, verb-phrase, article,adjective, noun, verb, adverb }; • T = { a,the, large,hungry,rabbit,mathematician,eats,hops,quickly, wildly }; • V–T= { sentence, noun-phrase, verb-phrase, article, adjective, noun, verb, adverb}; • S = sentence; • Production rules:P

P = { sentencenoun-phrase verb-phrase, noun-phrasearticle adjective noun, noun-phrasearticle noun, verb-phraseverb adverb, verb-phraseverb, articlea, articlethe, adjectivelarge, adjectivehungry, nounrabbit, nounmathematician, verb eats, verbhops, adverbquickly, adverbwildly}

Some Terminologies Let G = (V,T,S,P) be a phrase-structure grammar. Let w0 = lz0r andw1 = lz1r be strings over V . • If z0z1 is a production of G, we say that w1 isdirectly derivablefrom w0 and we write w0w1. • Example: theadjective noun verb adverb the largenoun verb adverbbecause adjective large • If w0,w1, … ,wn, n 0, are strings over V such that w0w1, w1w2, … ,wn-1wn, then • we say that wn isderivablefromw0, and • we write w0wn. • The sequence of steps used toobtain wn from w0 is called aderivation.

Example:sentencethe large rabbit hops quickly via the followingderivation: sentence  noun-phrase verb-phrase, noun-phrase verb-phrase article adjective noun verb-phrase, article adjective noun verb-phrase article adjective noun verb adverb, article adjective noun verb adverbtheadjective noun verb adverb, the adjective noun verb adverbthe largenoun verb adverb, the largenoun verb adverbthe large rabbitverb adverb, the large rabbitverb adverbthe large rabbit hopsadverb, the large rabbit hopsadverbthe large rabbit hops quickly.

What is the language generated by a Phrase-Structure Grammar? • Let G = (V,T,S,P) be a phrase-structure grammar. • Thelanguage generated byG(or thelanguageofG), denotedby L(G), is the set of all strings of terminals that are derivable fromthe starting symbolS. L(G) = { wT* | Sw }

Example:Suppose G = (V,T,S,P), where V = {a,b,A,B,S}, T = {a,b},S is the start symbol, and P = { SABa, ABB, Bab, ABb }. All the “sentences" (words) generated by this grammar are {abababa, ba}, since S  ABa  BBBa  abababa S  ABa  ba • Example:Let G be the grammar with V = {S,0,1},T = {0,1}, starting symbol S, and production rules P ={ S11S, S0 }. L(G) = {(11)n0 | n = 0,1,2, …}.

How to construct a grammar that generates a given language? • Example: Find a phrase-structure grammar to generate the set { 0n1n | n = 0,1,2, … } • Solution: G = (V,T,S,P), where V = { S, 0, 1 }, T = { 0,1 }, S isthe start symbol, and P = { S0S1,S }.

How to construct a grammar that generates a given language?? • Example: Find a phrase-structure grammar to generate the set { 0m1n | m,n = 0,1,2, … } • Solution 1:G1 = (V,T,S,P), where V = {S,0,1}, T = {0,1}, Sis the start symbol, and P = { S0S, SS1, S} • Solution 2:G2 = (V,T,S,P), where V = {S,A,0,1}, T = {0,1},S is the start symbol, and P = { S0S, S1A, S1, A1A, A1, S }  Two grammars can generate the same language!

How to construct a grammar that generates a given language??? • There are many techniques from thetheory ofcomputationwhichcan be used to systematically constructa grammar for a given formallanguage, but • This is beyond the scope of this course.

Types of Phrase-Structure Grammars (1) • Phrase-structure grammars can be classified according to thetypes of productionsthat are allowed. • Such a classification scheme introduced by NoamChomsky is as follows: • Type 0 grammar: has no restrictions on its production. • Type 1, or context-sensitive, grammar:can haveproductions only of theform • w1 w2, where l(w1)  l(w2), or of the form • w1. • Type 2, or context-free grammar:can haveproductions only of the form • A w2, where A is a nonterminal symbol.

Types of Phrase-Structure Grammars (2) • Type 3, or regular grammar:can have productions only of the form • AaB, • Aa, • S , where A and B are nonterminal symbols, S is the start symbol, and ais aterminal symbol. • Alanguagegenerated by a • type 1 grammaris called acontext-sensitive language; • type 2 grammaris called acontext-free language; • type 3 grammaris called aregular language.

Examples • { 0m1n | m,n = 0,1,2, … } is a regular language, since it can begenerated by a regular grammarGwith P: P = { S0S, S1A, S1, A1A, A1, S } • { 0n1n | n = 0,1,2, … }is a context-free language, since it can begenerated by a context-free grammarGwith P: P = { S0S1, S } • { 0n1n2n | n = 0,1,2, … }is a context-sensitive language, since itcan be generated by a type 1 grammar G = (V,T,S,P) withV = {0,1,2,S,A,B}, T = {0,1,2}, starting symbol S, and productions P = { S0SAB, S, BAAB, 0A01, 1A11, 1B12, 2B22 }; but not by any type 2 grammar.

Example: a Phrase-Structure Grammar • G = (V,T,S,P), where • V = { a,the,large,hungry,rabbit,mathematician,eats,hops,quickly,wildly; sentence, noun-phrase, verb-phrase, article,adjective, noun, verb, adverb }; • T = { a,the, large,hungry,rabbit,mathematician,eats,hops,quickly, wildly }; • V–T= { sentence, noun-phrase, verb-phrase, article, adjective, noun, verb, adverb}; • S = sentence; • Production rules:P

P = { sentencenoun-phrase verb-phrase, noun-phrasearticle adjective noun, noun-phrasearticle noun, verb-phraseverb adverb, verb-phraseverb, articlea, articlethe, adjectivelarge, adjectivehungry, nounrabbit, nounmathematician, verb eats, verbhops, adverbquickly, adverbwildly}

Example: Backus-Naur Form • What is the Backus-Naur Form of the grammar for a subset of English described before? <sentence> ::= <noun phrase><verb phrase> <noun phrase> ::= <article><adjective><noun>|<article><noun> <verb phrase> ::= <verb><adverb>|<verb> <article> ::= a | the <adjective> ::= large | hungry <noun> ::= rabbit | mathematician <verb> ::= eats | hops <adverb> ::= quickly | wildly

What is Backus-Naur Form (BNF)? • There is another notation that is used to specify a type 2 (context-free) grammar, called theBackus-Naur Form: • all productions having the same nonterminal as their left-hand sideare combined with the different right-hand sides of these productions, each separated by a bar ( | ), with • nonterminal symbols enclosed in angular brackets (<>), and • the symbol  replaced by ::= • Example:The Backus-Naur form for a grammar that produces signedintegers is as follows: <signed integer> ::= <sign><integer> <sign> ::= +|- <integer> ::= <digit>|<digit><integer> <digit> ::= 0|1|2|3|4|5|6|7|8|9

What is a Derivation (or Parse) Tree? • A derivation in the language generated by a context-free grammar can berepresented graphically using an ordered rooted tree, called aderivation (orparse) tree: • the root represents the starting symbol, • internal vertices represent nonterminals, • leaves represent terminals, and • the childrenof a vertex are the symbols on the right side of a production, in order from left to right, where the symbolrepresented bythe parent is on the left-hand side.

Example • Construct a derivation tree for the derivation of the sentence,the hungry rabbit eats quickly, discussed previously.

How to determine whether a string is in the language generated by a context-free grammar? • Top-down parsing: • begins with the starting symbol and proceedsby successively applying productions to see if the given string can bederived. • Bottom-up parsing: • work backwards.

Top-down parsing: S AB S AB CaB S  AB CaB cbaB S  AB  CaB  cbaB  cbab • Example:Determine whether the word cbab belongs to the L(G), where, G = (V,T,S,P) with V = {a,b,c,A,B,C,S}, T = {a,b,c}, S is the starting symbol, and the productions are S  AB A  Ca B  Ba B  Cb B  b C  cb C  b • Bottom-up parsing: Cab cbab Ab Cab  cbab AB Ab  Cab  cbab SAB Ab  Cab  cbab

Finite State Machines with No Output • Finite-state machines with no output are also calledfinite-state automata. • Finite-state automata do not generate output. But they have a set of specialstates, calledfinal states. • A finite-state automaton is often used for language recognition. • This application plays a fundamental role in the design and construction of compliers for programming languages.

What is a Deterministic Finite-State Automaton? • Afinite-state automatonM = (S,I,f,s0,F) consists of • a finite set S ofstates, • a finiteinput alphabetI, • atransition functionf that assigns a state to every pair of state andinput, • aninitial states0, and • a subset F of S consisting offinal states.

How to represent a Finite-State Automaton? • We can represent a finite-state automaton using either a state table or a state diagram. Final states are indicated in the state diagram by using double circles. • What is the state table of the above finite-state automaton?

What is the language recognized by a given Finite-State Automaton? • Aninput string is recognizedoracceptedby an automaton M if thestring takes the automaton to one of its final states. • The languagerecognized by an automaton M,denoted by L(M), is the set of all strings that are recognized by M. The language recognized by the above finite-state automaton M is L(M) = { 0n,0n10x | n=0,1,2, …, and x is any string }.

DeterministicvsNondeterministicFinite-State Automata • The finite-state automata discussed so far are deterministic, since for eachpair of state and input value there is a unique next state given by the transitionfunction. • There is another important type of finite-state automaton in which there maybe several possible next states for each pair of state and input value. • Suchmachines are callednondeterministic. • Nondeterministic finite-state automataare important in determiningwhich languages can be recognized by a finite-state automaton.

What is a Nondeterministic Finite-State Automaton? • Anondeterministic finite-state automaton M = (S,I,f,s0,F)consists of • a finite set S ofstates, • a finiteinput alphabet I, • atransition function f that assignsa set of statesto each pair ofstate and input, • aninitial state s0, and • a subset F of S consisting offinal states.

How to represent a NondeterministicFinite-State Automaton? • Using a state table:for each pair of state and input value we give a list ofpossible next states. • Using a state diagram: include an edge from each state to all possible nextstates, labelling edges with the input(s) that lead to this transition.

What is the language recognized by a given Nondeterministic Finite-State Automaton? • What does it mean for a nondeterministic finite-state automaton torecognizea string x = x1x2 … xk? • x1 takes the starting state s0 to a set S1of states; • x2 takes each of the states in S1 to a set of states. Let S2 be the union ofthese sets; • Continue this process, including at a stage all states that can be obtained using • a stateobtained at the previous stage and • the current input symbol; • The string x isrecognized oraccepted if there is a final state in the set ofall states that can be obtained from s0using x. • The language recognized by a nondeterministic finite-state automatonis the set of all strings recognized by this automaton.

Example • Determine the language recognized by the nondeterministic finite-state automaton M shown in the following figure. • Solution: L(M) = {0n, 0n01, 0n11 | n=0,1 ,2, … }.

An Important Fact • Theorem: If the languageL is recognized by a nondeterministic finite-state automaton M0, then L is also recognized by a deterministic finite-stateautomaton M1. • Two finite-state automata are calledequivalentif they recognize the samelanguage.

Build an FSA from a Regular Grammar • Suppose that G = (V,T,S,P) is a regular grammar generatingthe set L(G), where each production is of the form S ,Aa, or AaB, with a being a terminal symbol, A and Bare nonterminal symbols. • We can build a nondeterministic finite-state machine M = (S,I,f,s0,F) that recognizes L(G).

M = (S,I,f,s0,F) • S: contains a statesA for each nonterminal symbol A of G,and an additional final state sF ; • The start state s0 is the state formed from the start symbol S; • A transition from sA tosF on input of a is included if Aais a production; • A transition from sAto sB on input of a is included if AaBis a production; • s0 will also be a final state if S is a production. • It can be shown that L(M) = L(G).

Example • Construct a nondeterministic finite-state automaton that recognizes the language generated by the regular grammar G = (V,T,S,P)where • V = {0,1,A,S}, • T = {0,1}, and • the productions in P are S1A,S 0, S, A 0A, A 1A, and A 1.

Construct a Regular Grammar from an FSA • Suppose thatM = (S,I,f,s0,F) is a finite-state machine with the property that s0is never the next state for a transition. • A regular grammar G = (V,T,S,P) can be defined as follows: • V is formed by assigning a symbol to each state of S and eachinput symbol inI; • T is formed from the input symbols in I; • S is the symbol formed from the start state s0; • The set P of productions is formed from the transitions inM: • As a is included if the state s goes to a final state underinput a, where As is the nonterminal symbol formed from s; • As aAt is included if the state s goes to t under input a. • S is included if and only if L(M). • It can be shown that L(G) = L(M).

Example • Find a regular grammar that generates the language recognized by the finite-state automaton shown in the following figure: Soultion:G = (V,T,S,P) where • V = {S,A,B,0,1}, the symbols S,A, andB correspondto the states S0,S1, and S2, respectively; • T = {0,1}; • S is the start symbol; and • The productions are S  0A, S 1B, S 1, S , A 0A, A 1B, A 1, B 0A, B 1B, B 1.

More Powerful Types of Machines (1) • The main limitation of finite-state automata is their finite amount of memory.This prevents them from recognizing languages that are not regular, such as{0n1n|n = 0,1,2,…}. • A more powerful model of computation calledpushdown automatoncan beused to recognize the above language. • Theorem:A set is recognized by apushdown automatonif and only if itis the language generated by acontext-free grammar. • However, there are sets that cannot be expressed as the language generatedby a context-free grammar. One such set is { 0n1n2n|n = 0,1,2, … }.

More Powerful Types of Machines (2) • Actually, there exists an even more powerful machine than pushdown automata, calledlinear bounded automatawhich • can recognizecontext-sensitive languagessuch as the sets { 0n1n2n | n=0,1,2, …}; but they • cannot recognize all the languages generated by phrase-structure grammars. • The most general model of a computing machine is the so-calledTuring Machinewhich can • recognize all languagesgenerated by phrase-structure grammars; • model all the computationsthat can be performed on a computing machine.

Future: Scientists vs Engineers • Scientiststry tounderstand what is . • Engineerstry tocreate what has never been ! • The really great engineers have astrong background in science so that they thoroughly understand what is. • These special people also have to have theimaginationto create whathas never been, and this is what really sets them apart ! • The methodology of engineering research: • There exists some phenomenon of nature for which a model shouldbe found; • The mathematical analysis is just a tool that helps one to find this model; • The results of any analysis should be confirmed by experiments. • Future:What you make it to be !

COGN1001: Introduction to Cognitive Science Topics in Computer Science Formal Languages and Models of Computation