Penrose’s Mistake

1. Uri Sharir Computability Seminar – Prof. NachumDershowitz Tel Aviv University December 11, 2013 Penrose’s Mistake

2. Outline Brief history of AI Lucas and Gödel Penrose and Turing Summary

3. What is AI? artificial intelligencen1 :a branch of computer science dealing with the simulation of intelligent behavior in computers2 :the capability of a machine to imitate intelligent human behavior Merriam-Webster

4. Two Kinds of AI • “Strong AI” • An appropriately programmed computer is a mind • The human mind is in reality some kind of physical symbol-manipulation system • “Weak AI” • Physical symbol systems have the necessary and sufficient means for intelligent action • Allows for the possibility that the mind is not mechanical, but claims that it can (theoretically, at least) be simulated by a Turing machine

5. History: Myths and Antiquity • Throughout human history, people have used technology to emulate themselves • Greece: • Hephaestus’ golden robots, Talos, Pandora • Pygmalion’s Galatea • 384-322 BC: Aristotledescribes the syllogism • 1st century AD: Heron of Alexandria’s automata • China: • King Mu of Zhou and Yan Shi’s “artificial actor”

6. History: Alchemy, Fiction and Hoaxes • ~800: Jabir ibnHayyan’sTakwin • 1206: Al-Jazari’s programmable automata • ~1500: Paracelsus’ homunculus • ~1580: Rabbi Judah Loew ben Bezalel of Prague’s Golem • 1769: Wolfgang von Kempelen’sThe Turk • 1818: Mary Shelley’s Frankenstein • 1920: KarelČapek’sRossum’s Universal Robots (R.U.R.)

7. History: Formal Reasoning • 1275: Ramon Llull’sArs Magna • Early 17th century: René Descartes: bodies of animals are nothing more than complex machines (but not mind or soul) • 1641: Thomas Hobbes in Leviathan: “reason is nothing but reckoning” • 1672: Gottfried Leibniz: • Stepped Reckoner • Characteristicauniversalis

8. History: Formal Reasoning • 1854: George Boole’s Boolean algebra • “Investigate the fundamental laws of those operations of the mind by which reasoning is performed, to give expression to them in the symbolic language of a calculus” • 1913: Bertrand Russell and Alfred North Whitehead publish Principia Mathematica • 1920s and 1930s: • David Hilbert’s program • Alonzo Church’s lambda calculus • The Turing machine • Kurt Gödel’s incompleteness theorems • Within certain limits, any mathematicalreasoning could be mechanized

9. History: The Birth of AI • 1943: • Norbert Wiener's Cybernetics • Warren Sturgis McCulloch and Walter Pitts publish “A Logical Calculus of the Ideas Immanent in Nervous Activity” • 1950: • Alan Turing proposes the Turing Test • Isaac Asimov published hisThree Laws of Robotics • 1951: first working AI programs • 1952-1962: Arthur Samuel’s checkers programs

10. History: The Birth of AI • 1955: Allen Newell and Herbert A. Simoncreated the “Logic Theorist”, eventuallyproving 38 of the first 52 theorems inPrincipia Mathematica (and finding newand more elegant proofs for some) • Simon: they had “solved the venerable mind/body problem, explaining how a system composed of matter can have the properties of mind” • Later John Searle will call this the “Strong AI Hypothesis” • 1956: The first Dartmouth College summer AI conference is organized by John McCarthy, Marvin Minsky,Nathan Rochester (IBM) andClaude Shannon

11. History: The Golden Years (1956-1974) <P,O,I,G> • Several fields of work: • Reasoning as search • Stanford’s Shakey • Natural language • Joseph Weizenbaum’sELIZA • Micro-worlds • Terry Winograd’sSHRDLU • Optimism: • 1958: “Within ten years a digital computer will be the world's chess champion”; “within ten years a digital computer will discover and prove an important new mathematical theorem” (H.A. Simon and Allen Newell) • 1965: “Machines will be capable, within twenty years, of doing any work a man can do” (H.A. Simon) • 1967: “Within a generation ... the problem of creating ‘artificial intelligence’ will substantially be solved” (Marvin Minsky) • 1970: “In from three to eight years we will have a machine with the general intelligence of an average human being” (Marvin Minsky) • Funding: • 1963: MIT received a $2.2M grant from the newly created Advanced Research Projects Agency (later DARPA) A*

12. History: The 1stAI Winter (1974-1980) • Expectations set too high… • Limited computing power (AI requires computer power in the same way that aircraft require horsepower) • Intractability and combinatorial explosion (search or scaling toy micro-worlds) • Common sense and reasoning (general knowledge too vast) • Moravec’sparadox: proving theorems and solving geometry problems is comparatively easy for computers, but a supposedly simple task like recognizing a face or crossing a room without bumping into anything is extremely difficult • The end of funding…

13. Critiques from Across the Campus • 1961: John Lucas in “Minds, Machines and Gödel”: • Turing’s and (more so) Gödel’s theorems prove that “Mechanism is false”: minds cannot be explained as machines • 1965, 1972: Hubert Dreyfus in Alchemy and AI and What Computers Can't Do: • Challenges the four philosophical assumptionsAI researchers treated as axioms: • Biological (brains are composed of on/off switches) • Psychological (minds operate according to formal rules) • Epistemological (all knowledge can be formalized) • Ontological (the world consists of independent facts) • 1980: John Searle’s Chinese room argument

14. Outline Brief history of AI Lucas and Gödel Penrose and Turing Summary

15. John Randolph Lucas (1929-) • Follows Gödel’s proof: • Assume a consistent formal system F, which is strong enough to produce simple arithmetic (contains ℕ, + and ×) • Consider the Epimenides-inspired formula g = “This formula is unprovable in F” • If g is provable in F, we get a contradiction: • g is false in F (by its definition) • g is true in F (since F is sound) • Then g must be unprovable in F, and thus g is true • So all interesting consistent formal systems are incomplete: “contain unprovable, though perfectly meaningful, formulae … we, standing outside the system, can see to be true” • In particular, this “must apply to cybernetical machines” • “It follows that no machine can be a complete or adequate model of the mind, that minds are essentially different from machines”

16. Lucas’ Argument in Other Words • Assume that for any human Hthere exists at least one (deterministic) formal logical system F(H) which reliably predicts H’s actions in all circumstances • For any logical system F,a sufficiently skilled mathematical logician (equipped with a sufficiently powerful computer if necessary) can construct some statements G(F) which are true but unprovable in F (Gödel) • If a human Mis such a logician, then if Mis given F(M), he or she can construct G(F(M)) and determine that they are true (which F(M) can’t) • Hence F(M) does not reliably predict M’s actions in all circumstances • Lucas goes further to state the anti-deterministic view that this implies that M has free will • And it is implausible that the qualitative difference between mathematical logicians and the rest of the population is such that the former have free will and the latter do not

17. More from Lucas • “We are trying to produce a model of the mind which is mechanical – which is essentially ‘dead’ – but the mind, being in fact ‘alive’, can always go one better” • “Thanks to Gödel's theorem, the mind always has the last word” • Adding to the system additional axioms or rules to prove its Gödel formula wouldn’t help, because the “strengthened” system would have a Gödel formula of its own • It’s like a gamebetween the mechanist (M) and Lucas (L): • M goes first: he produces a definite mechanical model ofthe mind • L points to something that it cannot do, but the mind can • Mis free to modify his example, but each time he does so,L is entitled to look for defects in the revised model • If Mcan devise a model that L cannot find fault with,his thesis is established, otherwise it is not proven • And since Mnecessarily cannot, his thesis is refuted X

18. Objections to Lucas1. Consistency • Gödel’s theorem (and Lucas’ argument) applies only to consistent systems • If F is inconsistent, we can proveanything, including g and ¬g(inconsistent systems are not incomplete) • Two possible responses to Lucas: • We can’t establish our own consistency • This chimes with Gödel’s second incompleteness theorem: one cannot prove the consistency of a formal system from within the system itself • We might be able to prove our consistency drawing considerationsfrom outside the system, but who’s to say they are consistent? • We are in fact inconsistent Turing machines • Lucas responds that we are fallible but not systematicallyinconsistent • Otherwise we would believe anything, but we tend todiscard inconsistencies whenever we notice them

19. Objections to Lucas2. Complexity • Paul Benacerraf in God, the Devil, and Gödel (1967): • It is not easy to construct a Gödel formula • In order to do so for any given formal system one must have a solid understanding of the algorithm at work in the system • The formal system the human mind might implement is likely to be extremely complex (Hofstadter shares this view) • So Lucas’ argument is a disjunction: • Either no formal system encodes all human arithmetical capacity • Or any system which does has no axiomatic specificationwhich human beings can comprehend • Lucas’ response: • He could be helped in producing the Gödel formulafor any given formal system • It might be difficult, but we could at leastin principle determine what the Gödelformula is for any given system

20. Objections to Lucas3. The Whiteley Sentence • C.H. Whiteley (1962) claimed humans have similar limitations to the one that Lucas’ argument attributes to machines (so perhaps we are not different from machines after all) • The Whiteley sentence: w = “Lucas cannot consistently assert this formula” • If w is true, then if Lucas asserts it he is inconsistent • So one of two options: • Lucas is inconsistent (invalidating his own argument) • Lucas cannot utter w on pain of inconsistency, in which case w is true and Lucas is incomplete • Counter-response: • Lucas can recognize that w is true, as there’s a point of view from which he can understand how the sentence tricks him • From this point of view Lucas can appreciate that he can’t assert the sentence, and consequently he can recognize its truth

21. Objections to Lucas4. Idealizations • Lucas’ argument sets up a hypothetical scenario involving an idealized mind and an idealized machine • Some believe that once these idealizations are rejected, Lucas’ argument falters: • The output of any human mind is finite, and can thus be programmed (Lucas: but this is not modelling in principle; actuality instead of potentiality) • Add a Gödelizing operator, generating an infinite numberof Gödel formulae (Lucas: this new system will have atransfinite Gödel formula) • Most people do not understand Gödel’s theorem, so all Lucas has shown is that a handful of competent mathematical logicians are not machines (Lucas: it only means that the argument is more obvious regarding people who do understand Gödel’s theorem)

22. Objections to Lucas5. Lucas Arithmetic • David Lewis in “Lucas Against Mechanism” (1969): • “Peanoarithmetic” is the arithmetic that machines can produce and “Lucas arithmetic” is the arithmetic that humans can produce • Lucas arithmetic will contain Gödel formulae while Peano arithmetic will not, so humans are not machines • But Lucas has not shown that he (or anyone else) can produce Lucas arithmetic in its entirety, which he must do if his argument is to succeed, so his argument is incomplete • Lucas’ response: • All is required to disprove mechanism is to produce a single theorem that a human can see is true but amachine cannot, which Gödel’s theorem does

23. Lucas’ Final Thought • Lucas doesn't rule out the idea of intelligent computers: • “When we increase the complexity of our machines there may, perhaps, be surprises in store for us” • “Turing draws a parallel with a fission pile. Below a certain ‘critical’ size, nothing much happens: but above the critical size, the sparks begin to fly. So too, perhaps, with brains and machines…” • “Although it sounds implausible, it might turn out that above a certain level of complexity, a machine ceased to be predictable … and started doing things on its own account” • “It would begin to have a mind of its own when it was no longer entirely predictable … but was capable of doing things which we recognized as intelligent” • “But then it would cease to be a machine” • “There would then be two ways of bringing new minds into the world: • “The traditional way, by begetting children born of women… • “… and a new way by constructing very, very complicated systems of, say, valves and relays”

24. Outline Brief history of AI Lucas and Gödel Penrose and Turing Summary

25. Sir Roger Penrose (1931-) • English mathematical physicist, recreational mathematician and philosopher • Renowned for his work in mathematical physics, in particular general relativity and cosmology • Some notable contributions: • 1955: Moore-Penrose pseudoinverse • 1950s: Penrose triangle • 1959: Penrose stairs • 1965: Penrose-Hawkingsingularity theorems • 1974: Penrose tiling

26. Physics and Consciousness • Penrose has written several books on the connection between fundamental physics and (human) consciousness: • 1989: The Emperor’s New Mind • 1994: Shadows of the Mind • 1997: The Large, the Small and the Human Mind • The known laws of physics are inadequate to explain the phenomenon of consciousness • Consciousness arises from quantum mechanics, and its description requires a new kind of physics • Bases his arguments on: • The Emperor’s New Mind: Turing’s halting theorem (limitations of computation) • Shadows of the Mind: Gödel’s theorem(limitations of formal systems)

27. The Emperor’s New Mind (1989) Turing’s Theorem Will this program ever halt? The Halting Problem… Yes That’s the Question! No

28. Penrose’s Argument in a Nutshell • Consider all current sound human knowledge about non-termination • Suppose we could reduce this knowledge to a (finite) computer program • Then we could create a self-referential version of this program • A contradiction to its correct performance can be derived • Penrose’s resolution: the second step is invalid;no program can incorporate what (finitely many)humans know • So although humans can emulate Turing machines, machines cannot simulate all humans • The human mind possesses super-Turingabilities, using undiscovered physical processes

29. Formally • Let be the set of all Turing machines mapping partially to • The domain of Mi is • Let be the set of all r.e. sets( is computable if ) • Let

30. Formally • Penrose’s claimed theorem: • Proof: • Assume , then there is a fixed such that = Dk • So in particular (1) • And by definition of , (2) • The conjunction of (1) and (2) is of the form(¬AB)(BA), which implies A, i.e. • Note that we (human mathematicians) have convincingly demonstrated that, which means that • Combining this with (1), we get

31. Where’s the Beef? • The logical error is the step “Mk(k) = was convincingly demonstrated” • What actually was demonstrated is • But from assuming that holds, it cannot be concluded that human mathematicians can find a proof for

32. Turing’s Halting Problem in Brief • Suppose a programming language which: • Supports programs as data • Has some sort of conditional if ... then ... else ... • Includes at least one non-terminating program loop • Consider the decision problem of determining whether a program Xdoes not halt on itself: ? • Suppose A is a program that claimed to return Tfor (exactly) all such X • Then A would fail to answer correctly regarding the program C(Y ) := ifA(Y ) thenTelseloop() • Because C(C) = T ⇔ A(C) = T ⇔ C(C) =

33. Restating Penrose • Assume the following programs: • R • Can identify programs that diverge when fed themselves as input • But uses an undisclosed implementation (quantum mechanics?) • A • Incorporates all current, sound scientific knowledge on non-termination (equivalently, incorporates all R’s knowledge) • Known implementation • G • “God”, an oracle that always has the correct answer • C • Applies Cantoriandiagonalization to A to produce paradoxical behavior • K • Undisclosed program used by R (in its brain or logic circuitry) to inspect programs like A

34. Restating Penrose • Let denote the set of one-input partial predicates in any standard model of computation: • For any , one can construct CA as CA(Y) := ifA(Y ) thenTelseloop() • is the total predicate: G(X) := [X(X) = ]

35. Restating Penrose • Consider some partial predicate with the following rule (not necessarily its only rule): • return T if program X is of the formX(Y) := ifZ(Y) thenTelseloop(), whereZ …andK[Z(X) ≠ T] • R is a special case of what Penrose believes humans are capable of • It answers T regarding the divergence of X(X) if it believes (via K) that the test Z(X) does not succeed • Specifically, R(CA) = T if K[A(CA) ≠ T] = T • We don’t know what R will do if K returns F or does not respond within some reasonable time frame

36. Restating Penrose • The following facts are indisputable: A(CA) = T ⇔ CA(CA) = T A(CA) ≠T ⇔ CA (CA) = ⊥ A(CA) = T⇒ G(CA) = F A(CA) ≠T ⇒ G(CA) = T • Then no A can always be right when asked CA A(CA) ≠ G(CA) • Since we assumed A incorporates R’s knowledge, A(CA) = R(CA) ⇒ R(CA) ≠G(CA) • If A simulates Ron CA, then R does not respond properly • So there are two options: • A(CA) ≠ R(CA) A does not simulate R on CA • A(CA) = R(CA) = ⊥ R’s knowledge is incomplete:K[A(CA) ≠ T] = ⊥

37. Options for R A(CA) = R(CA) • So R has two options: • R has super-Turing abilities • R is incomplete R(CA) = ⊥ • What about other options, if R can err: • R internalizes an untruth K[A(CA) ≠T ] = T but in fact ¬[A(CA) ≠ T] • R“acts without thinking” K[A(CA) ≠ T ] ≠T and in reality R(CA) = ¬G(CA) • So if R gives the wrong answer on CA it’s either: • Because of the rule described earlier, and then K is unsound (III) • Based on some other rule which pre-empts our rule, and then K does not return T within some reasonable time frame (IV) F T I R(CA) = ⊥ F T II K = T F T IV III

38. Getting Back to Penrose • Penrose believes (I) is the only correct option, and proposes a non-Turing-equivalent model for R • Although R is deterministic, it cannot be captured algorithmically, it is more G-like CA(CA) = ?

39. What Does This Mean for AI? • If we disagree with Penrose’s option (I), and believe that there may exist a program A that simulates R, we must choose between: • Rreasons unsoundly with K • R feels under pressure and answers using a process other than K • R doesn’t answer at all

40. (II) R Reasons Unsoundly with K • Hilary Putnam: • Humans are inconsistent machines, believing falsehoods through unsound reasoning • Martin Davis: • “No human mathematician can claim infallibility. We all make mistakes!” • “There is nothing in Gödel’s theorem to preclude the mathematical powers of a human mind being equivalent to an algorithmic process that produces false as well as true statements”

41. (III) R Answers without K • John McCarthy: • “Much of Penrose’s reasoning is nonmonotonic, e.g. preferring the simplest explanation” • “But his methodology doesn’t allow for nonmonotonicreasoning by the program” • There is an A= R which answers incorrectly, but Penrose looks at A′which acts like R for which only monotonic K is consulted, and A’≠ R • J.R. Lucas: • “Our inconsistencies are mistakes rather than set policies” • “They correspond to the occasional malfunctioning of a machine, not its normal scheme of operations”

42. (IV) R Doesn’t Answer • R could be “inadequate”, not giving an answer about CA, and maybe that’s alright, cause we are also “inadequate” • David Chalmers: • “Perhaps we are sound, but we cannot know unassailably that we are sound” • Hilary Putnam: • “There is an obvious lacuna: the possibility of a program … which is not simple enough to appreciate in a perfectly conscious way is overlooked” • Geoffrey LaForte, Patrick J. Hayes, and Kenneth M. Ford: • “One can show … that Penrose’s notion of what it is to know oneself to be sound cannot itself be sound” • “Humans may be unable to know that they are consistent” • ArnonAvron: • “We cannot fully analyze a complicated learning machine, let alone the human mind” • “One cannot establish one’s own self-consistency”

43. Strange Loop • An interesting case: suppose A is designed to parrot R when asked whether CA(CA) = • Note that when anyone asks about CA, CA consults A, which turns the question over to R • So what does R answer? • R says “yes” to A A says “yes” to CACA(CA) = T • R says “no” to AA says “no” to CACA(CA) = • No matter what R says, it will be wrong • The only sound alternative for R is to“take the Fifth”: R(CA) =

44. Shadows of the Mind (1994) Gödel’s Theorem

45. Penrose’s Gödelian Error • Suppose: • A human Hhas a computational mind associated with the program P • His made aware of Pwhich we are using to “compute his brain” • We may use a formal mathematical system F to logically deduce the behavior of P, and hence H’s behavior, beliefs and thoughts • So F encapsulates all of H’s knowledge and beliefs • Penrose argues: • If H believes statement X, then since F encapsulates all of H’s beliefs, it must be possible to prove X in the system F • H will surely believe that the system F is sound • So H will believe (by Gödel's theorem) that g is true • But since g is not a theorem of F, it follows that F cannot after all encapsulate all of H’s beliefs • So H cannot have a computational mind

46. Penrose’s Gödelian Error • Penrose’s confusion: • If H believes X (on day 1) then it is necessary that F deduces thatHbelieves X on day 1, but it is not necessary that F deduces X! • If on day 2 H changes his mind and believes the opposite ¬X, then it is necessary that F deduces thatH believes ¬Xon day 2 (and not ¬X) • If on day 3 H goes crazy then it is necessary that F deduces that H is crazy on day 3; it is not necessary that F deduces all sorts of crazy formulae • Although F cannot deduce g, there is no reason why F cannot deduce that H believes g • Gödel's theorem doesn’t say anything about what can be proved concerning the state of H’s mind! • The essential point: • H’s beliefs do not form part of the deductive system F • If H believes two contradictory statements X and Y, these statements are not theorems of F and so it does not follow that F is inconsistent, as it would be if X and Y were theorems

47. Feferman’s Criticism • Solomon Feferman has a common cause with Penrose: • Opposes the dominant computational model of the mind • Human thought, and in particular mathematical thought, is not achieved by mechanical application of algorithms • Rather, by trial-and-error, insight and inspiration, in a process that machines will never share with humans • But has a lot of criticism: • Penrose extends his argument too far in areas such as mathematical soundness and consistency, providing ammunition for the computational-mind camp • Feferman finds several technical and logical errorsin Penrose’s reasoning • But then concludes that Penrose’s case wouldnot be altered by correcting these flaws

48. Feferman’s Criticism • The computational-mind argument stresses the equivalence between Turing machines and formal systems • But the model of mathematical thought using formal systems is closer to the nature of human (and particularly mathematical) thought • The Turing machine model: given a problem, human reason would plug away, applying the same algorithm indefinitely, in the hope of finding an answer • Butit’s ridiculous to think that mathematics is done this way • The basis of mathematical success is trial-and-error reasoning, insight and inspiration, based on prior experience – but not on general rules • A mechanical approach is only appropriate after an initial proof has been arrived at, for checking and validating it

49. Outline Brief history of AI Lucas and Gödel Penrose and Turing Summary

50. So What Does Gödel Have to Say? • “So the following disjunctive conclusion is inevitable: • “Either mathematics is incompletable in this sense, that its evident axioms can never be comprised in a finite rule, that is to say, the human mind (even within the realm of pure mathematics) infinitely surpasses the powers of any finite machine…” • “… or else there exist absolutely unsolvable diophantine problems of the type specified…” • Lucas: “It is clear that Gödel thought the second disjunct false”