Cognitive Computing 2012-2013

1. Cognitive Computing 2012-2013 Consciousness and Computation: computing machinery and intelligence 1. TURING MACHINES Mark Bishop

2. Why is a theory of computation important? • Because the theory of computation impacts: • The fundamentals of computing • The formalisation of just what is meant be the notion of ‘computing’. • The fundamentals of mathematics • If provides insight into the failure of Hilbert’s program: the formalisation of all mathematics in axiomatic form and proof that this form is consistent. • The fundamentals of Artificial Intelligence and Cognitive Science • The ideology that the mind/psychology is fully explainable in computational terms. (c) Bishop: Consciousness and Computation

3. The algorithm (historical) • 9thC. Abū ʿAbdallāh Muḥammad ibn Mūsā al-Khwārizmī, (a Persian scholar based in the ‘House of Wisdom’ in Bagdad) • Amongst other things Al-Khwarizmi outlined a finite series of steps, (‘Al-jabr’, or transformations; algebra), which when followed produce the (real) roots of quadratic equations. • In so doing Al-Khwarizmi, (or Algorismus as he was known in Latin), had not only spoken of, but also given his own name to, the new art of computation; the ‘algorithm’. (c) Bishop: Consciousness and Computation

4. The algorithm (modern) • A modern definition states that a procedure to achieve a specified result is an algorithm iff: • Each step is ‘blind’, (each step needs no further insight). • Steps follow on ‘blindly’, (no insight needed to determine the next step). • A result is guaranteed after a finite number of steps, (cf. WINDOWS). (c) Bishop: Consciousness and Computation

5. On ‘computers’ … • Originally the term ‘computer’ did not refer to a machine, but a human being :- • A mathematical assistant who calculated ‘by rote’ in accordance with some ‘effective method’. • A method is ‘effective’ iff: • … it demands no insight or ingenuity from the human computer. • … produces a result in a finite number of steps. • It is clear that the notion of ‘effective method’ is closely linked with that of, ‘the algorithm’. (c) Bishop: Consciousness and Computation

6. On ‘computing machines’ … • A ‘computing machine’ was originally simply a term used to describe a machine, used by a human ‘computer’, which automated some aspect(s) of the calculations required when following an ‘effective method’ to solve some task. • Hence the use of ‘computing machines’ is analogous to that of modern [non-programmable] calculators. • However, in principle, the operation of a ‘computing machine’ could always be fully performed by a ‘human computer’ following an ‘effective method’. (c) Bishop: Consciousness and Computation

7. What is an ‘effective method’? • Because neither ‘insight’ or ‘ingenuity’ are defined, the mathematical notion of ‘effective method’ is not rigorously defined. • Turing set out to provide a formal model of what can be achieved by a ‘human computer’ following an ‘effective method’ by rote. • “To the question 'what is a "mechanical" process?' Turing returned the characteristic answer, 'Something that can be done by a machine' and he embarked on the highly congenial task of analysing the general notion of a computing machine.” • In the specification of the Turing Machine, (TM), Turing outlined his idealisation of the operation of a ‘Human Computer’ following an ‘effective method’. (c) Bishop: Consciousness and Computation

8. Finite State Automata • A finite-state automaton (plural: automata) or finite-state machine (FSM), or simply [discrete] state machine, is a simple mathematical abstraction, sometimes used to design digital logic or computer programs. • It is a behavior model composed of a finite number of states, transitions between those states, and actions. • It has finite internal memory, an input feature that reads symbols in a sequence, one at a time without going backward; and an output feature, (which may be in the form of a user interface, once the model is implemented). • The operation of an FSM begins from one of the states (called a start state), goes through transitions depending on input to different states and can end in any of those states available … • … Only a certain set of states mark a successful flow of operation (called ACCEPT states). (c) Bishop: Consciousness and Computation

9. Turing’s input-less discrete State Machine (1950) • At each time step the machine occupies one of a finite number of (3) possible physical positions. • In operation it cycles through a finite number of (3) possible computational states, {Q1, Q2, Q3}. • The next computational state is modally determined by the current state. • The output {LIGHT-ON} occurs when the machine is in computational state Q1. (c) Bishop: Consciousness and Computation

10. FSA specification • Three sets and two functions attached to state machines. • FSAs can be described by the five tuple <I, Z, Q, a, b> • I Finite set of inputs • Q Finite set of internal states • Z Finite set of outputs • a Input mapping function: fa (Q, I) → Q’ • b Moore output function: fb (Q’) → Z • b Mealy output function: fb (Q, I) → Z • The Moore and Mealy forms of FSA are ‘equivalent’. (c) Bishop: Consciousness and Computation

11. Classification of FSAs • There are two groups distinguished: Acceptors (or Recognizers) and Transducers. • Acceptors (recognizers) produce a binary output, indicating if the FSA input is ‘accepted’ by the machine or not. • All states of the FSM are said to be either accepting or not accepting. • If when all input is processed the current state is an accepting state, the input is accepted otherwise not. • Transducers generate output based on a given input and/or a state using actions and are typically used for control applications. (c) Bishop: Consciousness and Computation

12. What can FSA do? • Transducer FSA generate output based on a given input and/or state. • In engineering transducers are used to describe a large number of systems: electronic design automation, communication protocol design etc. • In biology and artificial intelligence research, state machines or hierarchies of state machines are sometimes used to describe, for example, cell operations and neurological systems • In linguistics, state machines are sometimes used to describe the grammars of natural languages. (c) Bishop: Consciousness and Computation

13. The Turing Machine, (1935/6) • A Turing Machine, (TM), consists of: • A control box - a finite state sequential machine, [m]. • A (potentially) infinite length tape, [T], divided into discrete frames with one symbol per frame. • A r/w head which can read or write discrete symbols from a finite alphabet. • The r/w head can move left or right unit frames along the tape. (c) Bishop: Consciousness and Computation

14. Turing Machine operation • The activity of a TM occurs in discrete steps defined by two conditions, (Q & I), to produce TM output (Z): • Current internal state: q • Current symbol on tape under machine head: s • Produce output, [Z], { z1 = symbol to write; z2 = tape head movement} • Given any current machine state Q, and an input symbol I, the state transition table specifies a 3-tuple, [z1 , z2 , Q’}, defining the Turing machines operation: • z1: Designates the symbol to leave on the tape. • z2: Move machine r/w head left or right. • Q’: Designates the next Turing machine state. (c) Bishop: Consciousness and Computation

15. An example • Design a simple TMp to add two unary numbers. The numbers are zero terminated and also separated by a zero. Assume TMp is initial state Q1 with its r/w head over the leftmost [1]. • Hence to add (2 + 3) the initial tape is [01101110] • Symbol on tape = ‘0’‘1’ • State: Q1 {0 Q1 R} {0 Q2 R} • State: Q2 {1 Q3 R} {1 Q2 R} • State: Q3 {HALT} {HALT} • In the Theory of Computation, the HALT state is sometimes refined into two sub-states: HALT-ACCEPT and HALT-REJECT. • It is trivial to see that this does not extend the single HALT state TM model as accept and reject values can be encoded as alternative terminating states in the state transition table, (in the above example: Q3 and Q4). (c) Bishop: Consciousness and Computation

16. The Universal Turing Machine • Turing showed that in principle the operation of any Turing Machine, TMm (T), can be simulated by a Universal Turing Machine, UTM (m, T). • Any TMm (T) can be encoded on a tape a series of ones and zero's, since every TM is described by its state transition table, (m), its ‘program’. • The table (m) must be finite in length since the number of machine states and the input alphabet are both finite. • The symbols on the tape of the UTM are divided into two groups: • The left is the encoded description of TMm with a pointer indicating which state TMm would be in at this stage of the computation. • The right is the sequence of symbols {X} that would be encountered by TMm on its input tape with a pointer indicating which square TMm would be currently scanning. • Turing proved that the net effect of the universal machine, UTM (m, T), is exactly what the effect of Turing Machine, TMm (T), would be. (c) Bishop: Consciousness and Computation

17. The Language of a Turing Machine • The set of input strings that result in the TMM, entering an ACCEPT state is called the Language of M, and is denoted L (M). • Turing Recognisable • Any Turing Machine will always either {ACCEPT, REJECT or LOOP} when given any input string. • LOOP occurs when a TM does not halt. • LOOPing may occur due to the TM being caught in an infinite loop (it repeats the same steps forever) or be due to a more complex process. • A language is Turing Recognisable if some Turing Machine ACCEPTs it. • A Turing Recognisable Language is also called 'recursively enumerable'. (c) Bishop: Consciousness and Computation

18. Deciders • A Turing Machine that never LOOPs is called a Decider. • ie. A Decider that is guaranteed to enter a HALT state on all possible input strings. • A Turing Machine that is a Decider is also called 'defined ‘ • Turing Decidable • A language is Turing Decidable, if there exists a Defined TM that ACCEPTs it. • A language that is Turing Decidable is also called 'recursive'. (c) Bishop: Consciousness and Computation

19. Variants of Turing Machines • A probabilistic Turing machine is one in which state transitions are defined by a probability distribution. • A non-deterministic Turing machine is one which has more than one next state for some combinations of contents of the current cell and current state. An input is accepted if any move sequence leads to acceptance. • Hence consider a nondeterministic Turing machine as a probabilistic Turing machine with all states leading to ACCEPT have transition probabilities ‘one’; • or simultaneously unfolding as a computation tree, each branch of which follows one such transition possibility. (c) Bishop: Consciousness and Computation

20. Parallel processing (Multi-Tape Turing Machines) • A multi-tape (parallel) Turing Machine (P) has k tapes and k read/write heads. • It is trivial to show that the set of languages ACCEPTable by P is exactly the same as the set ACCEPTable by D. • Similar to the UTM, encode each of the multi-tapes onto one tape and use pointers to indicate which tape to process next and where on the tape to process next. (c) Bishop: Consciousness and Computation

21. Equivalence of ndTM and TMs • A non-deterministic Turing Machine (ndTM) has an equivalent deterministic Turing Machine. • Informal outline proof: • We will show how to simulate any non-deterministic Turing Machine (N) by a deterministic Turing Machine (D). • Consider the operation of N on input L. This defines a tree of possible execution traces through states of N. • D must carefully evaluate all branches of N. • As soon as D finds an ACCEPT state of N then D will also ACCEPT L, otherwise D will not terminate. • It is clear that the language ACCEPTable by D is exactly the same as the language ACCEPTable by N. (c) Bishop: Consciousness and Computation

22. Deterministic and non-deterministic automata • Deterministic [and non-deterministic automata] recognise regular languages. • A regular language is a language (i.e. set of strings) that can be expressed using a only simple regular expression often called a pattern. • A pattern is simply a tool that specifies a set of strings via simple rules rather than by enumeration [via, say, a list]. • The basic rules of a regular expression are: • | vertical bar for OR • Grouping symbols into a composite element via brackets () • Quantification via ? means zero or one of the preceding elements; • * means zero or more of the preceding elements; • + means one or more of the preceding elements. • Hence a simple regular language might be {Hot; Hat} as defined by the regular expression H(a|o)t (c) Bishop: Consciousness and Computation

23. Push-down automata and Turing machines • Push-down automata recognise ‘context-free languages’. • A context free language is generated by a pushdown automata (i.e. a simple automata with additional memory implemented via a stack). • Hence a simple context-free language might be {ab aabb aaabbb aaaabbbb}. • I.e. a set of all non-empty even-length strings, the entire first halves of which are a's, and the entire second halves of which are b’s. • Turing Machine (and non-deterministic Turing Machines) recognise ‘recursively-enumerable languages’. • Hence the relative power of automata is: [non] deterministic automata; push-down automata and (the most powerful) [non-deterministic] Turing machines. (c) Bishop: Consciousness and Computation

24. Turing’s thesis … • Also known as the Church-Turing thesis: • A ‘Universal Turing Machine’ can perform whatever tasks a ‘human computer’ can achieve when following an ‘effective method’ by rote. • And, (with only one or two exceptions), this thesis remains, “agreed amongst logicians”. (c) Bishop: Consciousness and Computation

25. Other Universal Machines • McCulloch/Pitts: crude neuron simulations • Simulate OR, AND and NOT and hence build a computer. • Kleene: Recursive Functions • Foundation of functional programming. • Church: Lambda calculus • Foundation of LISP (McCarthy). • Post Systems & Cholmsky Grammars • (Character) String manipulation systems. • NORMA • The ultimate RISC machine ... (c) Bishop: Consciousness and Computation

26. And in the beginning … • The first computers were hard-wired to perform specific functions. • The British Colossus (1943) and the American ENIAC (1945) • The stored program machine, derived from the UTM model, stores both data and program in one homogeneous memory. • Re-programming the machine simply entails re-loading the memory. • The 1st Stored Program Machine, The Manchester baby, (1948) • Commercialised by Ferranti (1951). • The 2nd, The Cambridge University EDSAC, (1949) • LEO (1951) – sponsored by Lyons Coffee Houses… • In America the UNIVAC (1951); • The 1st mass produced computing machine being the IBM 701 (1953) • But no such [commercially available] stored program computer can ‘out compute’ the UTM … (c) Bishop: Consciousness and Computation