A Hierarchy of Formal Languages and Automata

1. CS 3240 – Chapter 11 A Hierarchy of Formal Languages and Automata

2. Interesting Fact About TMs • They may not halt on every possible input! • And not just because the creator of a specific TM was a doofus • This is related to the major mathematical/computational discovery of the 20th century! • There are propositions that cannot be decided (“proven”)

3. Important TermsDecidable • A question is decidable if there is a TM that always halts and answers “yes” or “no” for each possible input • The TM therefore constitutes an algorithm • A language is decidable if there is a TM that always halts and answers “accept” or “reject” whenever an input string is in the language or not • aka “Turing acceptable”

4. Important TermsComputable • A function is computable if there is a TM that always halts with the appropriate output for each possible input in the function domain • aka “Total function”

5. “Procedures” that may not Halt • Let g(x,y) be some computable function • Let f(x) = the smallest p where g(x, p) = 1, or0, if such a p does not exist • “Pseudo-algorithm” for f(x):m = 0;while (g(x, m) != 1) ++m;cout << m; • If there is no m for a given g and x, then we hang!

6. Halting and Hanging • 3 possibilities in general when a TM processes an input string: • Accepts (goes to an accepting halt state) • Rejects (e.g., crashes, or gives a “no” answer) • Hangs (infinite loop)

7. Important TermsRecognizable • A language for which there is a TM that always halts for and accepts strings in the language is recognizable • “It knows one when it sees one” :-) • But it may hang on strings not in the language

8. Important TermsPartial Functions • A function that is defined for only some of its domain elements is a partial function • It may hang (or “blow up”) on some inputs • e.g., divide by zero • This is the computational analogue to a recognizable language • you’ll always get an answer with a valid input • it may hang on invalid input

9. Where Are We? CS 3240 - Introduction

10. A language L over the alphabet is called recursively enumerable (aka “recognizable”) if there is a TM T that accepts every word in L and either rejects (crashes) or loops forever for every word in the language L', the complement of L: accept(T) = L reject(T) + loop(T) = L' recursively enumerable (r.e.) = recognizable

11. A Recursively Enumerable Language • We just saw one (slide 5) • Let g(x,y) be some computable function • Let f(x) = the smallest p where g(x, p) = 1, or0, if such a p does not exist • Let Lg be the set of functions, f, corresponding to all computable functions g, as explained above • Such functions can be encoded as strings, and are therefore countable • Lg can then be seen as a language (a set of strings)

12. A language L over the alphabet is called recursive (aka “decidable”) if there is a TM T that accepts every word in L and rejects every word in L'; that is, accept(T) = L reject(T) = L' A TM that accepts a recursive set is a model of an algorithm. It always halts. recursive = decidable

13. Important Facts • There exist languages that are r.e. but not recursive • We just saw one • There exist languages that aren't even r.e.! • (You'll see one soon) • All are “contrived” • Languages generated by grammars are r.e. or “better”

14. Closure Properties • All r.e. languages are closed under union, intersection, concatenation, and Kleene* • Everything but complement! • Recursive languages are also closed under complement • Also: If L and L' are r.e., then L is recursive • Here come the Proofs…

15. Complements of Recursive Languages • Let M be a machine that decides a recursive language, L • Form the machine M' by inverting the acceptability output of M • Goes to a reject state instead • Then M'decidesL' • So L' is recursive

16. Complements of r.e. Languages • Suppose L and L' are both r.e. • Let MrecognizeL, and M'recognizeL' • M may hang on elements of L', but M' doesn't • Form a new machine, M* that calls M and M' in parallel (non-deterministically) • If M accepts w, so does M* • If M' accepts w, rejectw • There are no other possibilities! (No hanging) • Therefore, L is decidable/recursive, by definition

17. Mid-stream Checkpoint • TMs can recognize/accept strings from certain languages and/or compute functions • If there is a TM, M, that accepts a language, L, and Malways halts, then L is recursive • If there is no such M for L, but there is instead a machine M that accepts every string in L, but M may hang on strings not in L, then L is recursively enumerable

18. Mid-stream Checkpoint(continued) • The complement of a recursive language is recursive • In fact, recursive languages are closed under all operations, like regular languages are • r.e. languages are closed under intersection • The complement of a r.e. language may not be r.e. • But if it is, then both languages are actually recursive!

19. Uncountable Sets A Timely Interlude

20. How Many Real Numbers are There? • The real numbers in (0,1) are uncountable • They cannot be mapped in a 1-to-1 fashion to the counting numbers • Proof: • Assume they can be: r0, r1, r2, … • Arrange their digits in a table a[ ][ ] • each row, a[i], contains the digits or ri • The diagonal sequence (a[n][n]+1) mod10, representing a valid real number, is not in the table! Contradiction!

21. P(N) (aka 2N) is Uncountable • The power set of the natural numbers, N • the set of all subsets (2N) • Suppose it is countable • The we can enumerate the sets: p0, p1, p2, … • Now consider the set T = {i | i ∉ f(i) = pi } • Certainly T is a set of integers, so T ∈ P(N) • Call it pk • Question: Is k in T = pk?

22. The Point? • The power set of any countably infinite set is uncountable • Much bigger than a countable set! • The number of countable sets is negligible compared to the number of uncountable sets

23. How Many Languages Are There?Countable vs. Uncountable? • Well?

24. How Many Languages Are There? • A language over an alphabet Σ is a subset of Σ* • the latter being an infinite set • The set of all languages over Σ is therefore the power set of Σ*(2∑*) • which we just showed is uncountable • So… the number of languages over any finite alphabet is uncountable

25. There are More Languages than TMs! • The # of TMs are countable, but the # of languages is not • Therefore, some languages cannot be recognized by a TM • There aren't enough TMs to go around! • Just like there are more reals than integers • So, non-r.e. languages must exist!

26. A non-r.e Language • Just take the complement of any r.e. language that is not recursive • Example (page 279): • Consider all TMs, Miwith alphabet Σ={a}. • Let X = {ai: ai ∈ L(Mi)} • This is r.e., but not recursive • Because we can construct a TM that carries out the computation, but it may not halt when ai ∉ L(Mi) • Then X’ must be non-r.e.!

27. A Slightly More Rigorous ProofPage 280 • Suppose X’is r.e. • Then there is a TM, Mk, that recognizes it • Now ask the question, is ak ∈ X’? • If it is, this means ak ∉ L(Mk) = X’, by definition • Contradiction! • If it isn’t, then this means that ak ∈ L(Mk) = X’ • Again, by definition • Again, a contradiction! • ⇒ X’ is not r.e.

28. The Membership Problem • What we have just described is the membership problem: • “Given a r.e. language, L, and a string, w, is w in L?” • We have just shown that the membership problem is, in general, undecidable

29. Where Are We? CS 3240 - Introduction

30. Unrestricted Grammars(Section 11.2) • Left-hand side of the rule is a concatenation of one or moresymbols • There must be at least one variable on the left • λ is not allowed on the left • Any string is allowed on the right, including λ • aka “Type 0” Grammar

31. A Grammar for anbncn, n ≥ 0 S → aAbc | abc | λ Ab→ bA Ac→ Bbcc bB→ Bb aB→ aa | aaA S ⇒aAbc ⇒abAc ⇒abBbcc ⇒aBbbcc ⇒aabbcc An A is created as the left-most variable. It travels to the right until it finds a 'c', then creates a new b and c, and becomes a B. The B moves back to create the extra needed a. The last rule allows the option to do it all over again (by introducing another A). This is similar to what the Turing machine for the same language does, except the TM marks instead of generates the letters.

32. A Grammar for na(w) = nb(w) • We have already seen CFGs for this • This unrestricted grammar: • Introduces X’s and Y’s together • Allows moving X’s and Y’s arbitrarily by swapping them • S → XYS | λXY → YXYX → XYX → aY → b

33. About Unrestricted Grammars • Cannot use the decidability algorithms for CFGs • e.g., CYK algorithm does not apply • No “Normal Form” • Non-null productions may create shorter strings • Terminals can disappear!

34. Equivalence of Grammars and TMs • For every r.e. language, there is an unrestricted grammar that generates it • For every unrestricted grammar, there is a TM that recognizes its language • It may or may not decide it • We will not prove this • but the book does

35. Context-Sensitive GrammarsSection 11.3 • Like Unrestricted Grammars except: • Right-hand side must be no shorter than left-hand side • So strings never shrink • Hence the name “non-contracting” or “monotonic” • Cannot contain λanywhere • Context-sensitive languages don't need unlimited memory • Since intermediate sentential forms never shrink, the largest memory requirement is proportional to |w| • Accepting machine: Linear Bounded Automaton

36. anbncn, n>0 is Context Sensitive S → aAbc | abc Ab→ bA Ac→ Bbcc bB→ Bb aB→ aa | aaA

37. Context-Sensitive Grammars and Recursive Languages • Every context-sensitive language is recursive • there is a TM that accepts it (i.e., always halts) • But not all recursive languages are context-sensitive • So Context Sensitive languages form a propersubset of Recursive Languages • Just like Recursive languages are a proper subset of Recursively enumerable languages

38. A Hierarchy of GrammarsSection 11.4 Type 0 grammar Type 1 grammar Type 2 grammar Type 3 grammar

39. Looking at the Language Subsets

40. The Chomsky Hierarchy