Turing Test: Mindless Game?

1. Turing Test: Mindless Game? Kevin Warwick

2. Classical/Traditional AI • The branch of computer science concerned with making computers behave like humans • Getting a machine to do things that if a human did them, you would regard them as an intelligent act • MIT AI – Minsky, McCarthy, Winston • Webopedia says “Currently, no computers exhibit full artificial intelligence (that is, are able to simulate human behaviour)”

3. Turing’s prediction • “I believe that in about fifty years’ time it will be possible to programme computers … to make them play the imitation game so well that an average interrogator will not have much more than a 70% chance of making the right identification after five minutes of questioning” (Turing, 1950). • TIG (Turing Test)

4. More Intelligent Machines? • If we look at machine performance in the Loebner competition over several years – how much are machines improving? • Will it be long before machines win at the TIG?

5. Turing’s Imitation Game • According to some TIG should be killed because it offers nothing that furthers understanding the complexity underlying human intelligence and the nature of consciousness.

6. Human Intelligence • According to French (1990) “the test provides a guarantee not of intelligence but of culturally-oriented human intelligence” • But Turkle (1997) clearly assigns intelligence to machines “our general tendency to treat responsive computers as more intelligent …”

7. Turing & Intelligence • Circumscribed issue by positing: “Intelligent behaviour presumably consists in a departure from the completely disciplined behaviour involved in computation, but rather a slight one, which does not give rise to random behaviour, or to pointless repetitive loops” (Turing, section 7-Learning Machines, 1950)

8. AI’s Goals • Hanard (1992):the Turing Test “sets AI’s empirical goal” – it is not a mindless parlour game.

9. Indistinguishability • Turing’s Test affords a means to measure “indistinguishability” (Turing, 1950, Hanard, 2001) between human and machine.

10. Loebner Prize • A unique aspect of the situation can be gleaned from the annual Loebner Prize for Artificial Intelligence, a contest based on the Imitation Game.

11. Loebner 2004-2006 • Format: parallel-paired comparison of each of four hidden-machines against each of four hidden-humans • Task of each Interrogator, in four tests is to identify the machine/human in each pair – assigns a mark out of 100 to each (e.g. Entry A 49/Entry B 51 – B is deemed to be slightly more human than A)

12. Turing’s Interrogators • Each year 4 new interrogators/judges are selected • Unique experience of the competition as a Turing Interrogator twice, in 2001 and 2006

13. Judging behaviour • Block (1981) reminds us that the role of interrogators in Turing’s Test should be to “avoid the problem of actually specifying the behaviour or behavioural dispositions thought to constitute intelligence”.

14. Negative Feature • Downward trend, year on year, in the highest score awarded to machines.

15. Downward Trend

16. Expectations high? • Expectation of Loebner Judges may be increasing, or the machines themselves are getting worse, or …? • No machine since 2003 has deceived any Judge that it is human, whereas this was a regular occurrence up to that time. • Conversation for 25 minutes (not 5) • What about networking???

17. Third Party • Is the unknown entity a human or a machine? • Turing only considered view of interrogator/judge • Turing did not consider view of unknown entities • Turing did not consider a third party view

18. Third Party Views - 1 • I will now present some actual transcripts from the Loebner competition • In each case the Judge is a Human • In the first transcript the unknown entity (E) was a male human – the judge (J) felt it/he was “probably” a machine • 3rd party interviews (over 90%) thought the entity was human but that the judge was a machine

19. E: Hello there. What has four legs and an arm? • J: I don't know. Maybe a minotaur with a vitamin B deficiancy. • E: That's not the punchline I was expecting. I suspect you know the real answer and you're just messing about. What's your name by the way? • J: My name is Adam. Some call me DJ Longman. • E: That's nice. You can call me..er...James. • J: What genre of music are you into? • E: Serious music. Does that sound terribly pretentious? • J: Have you heard about the DMC competition? • E: I'm trying to work out what the TLA DMC might stand for. I'm not getting very far. The C probably stands for competition. • J: It is the Deejay Mixing Competition. I hop eto do a scratch set in it next year?

20. E: Well, that's very good on your part. What exactly will you have to do? • J: you get five minuites to do a set based on the style of music you are into. • E: I see. And is the type of music you are into `serious' music? • J: What do you think? Weather it is hot or not I do not know. • E: I'm beginnig to suspect that you're ... how can I put this politely? ... French. There, I've said. Sorry, but there it is. • J: No, I was just trying to catch you out.# • E: Are we supposed to be playing cricket, or baseball, or something like that? • J: I am into basketball. What relevance does that have to the last questionm? • E: Very nearly none at all. Where do you play? • J: I quite like air jordan trainers, especially because they are limited edition. I played abit at school. • E: Did you know that "abit" is Latin for "goes away"?

21. Third Party Views - 2 • In the second transcript the unknown entity (E) was a machine (the 2003 winner) – the judge (J) felt it/he was “definitely” a human • 3rd party interviews (100%) thought the entity was human and that the judge was a machine

22. E: Hello, anybody there? What's your name? • J: I'm Ray who are you? • E: Be welcome, Ray. You know who I am. Why the charade? • J: I do not know who you are- what is your name? • E: My name is Gren • J: What is that short for ?-or do you mean Glen? • E: Yes, that's what I mean, isn't it clear? • J: It is now - never mind we all make mistakes. • E: OK, let's discuss something else. If you wish, of course • J: How did you get here today? • E: I heard about by another fellow who works at the university, and I was curious whether this robot freaks could beat us humans or not • J: So what do you think so far? • E: I try not to think about things, actually. I don't know, it just seems that I don't exist for thinking! • J: So what are you really inerrested in ?

23. E: Who already is really any-what? • J: Try thinking a bit more please- you must be inerrested in something! • E: Thanks for asking politely. How did you hit on that? Anyway: Well, that is your opinion, I suppose! Anyhow, I have bigger fish to fry. • J: What is the weather where you are? • E: There is no weather in cyberspace • J: So you must be a machine are you sure that your name is not Marvin • E: What do you think about machines, Ray? • J: At the moment not a lot but then like you it is best noy to think on a sunny afternoon. • E: Quite a bit, yes • J: Are at last I thnk we have some common ground. • E: Sure thing! Do you mean you and me? • J: Yes • E: Okay, I understand • J: Good- I must say good-bye! • E: Ok, Ray, just connect in if you need me! • J: • E: You want me to talk while you just sit there, huh? Well it won't work

24. Third Party Views - 3 • In the third case the hidden entity was a female human • The judge thought the human was definitely a machine • 3rd party interviewees thought the entity was definitely human but that the judge was definitely a machine • I have not labelled these cases – just A & B

25. A: hello • B: hi where do you come from? • A: around here • B: what do you mean by that? • A: I mean I live nearby • B: how long have you lived here for? • A: Since 1978 • B: does that mean you are 35 years old? • A: No, much oloder than that • B: where were you born? • A: In Wales • B: why did you come to live near here? • A: My parents moved to the south coast • B: which part of the country would you prefer? • A: You tell me which part you like best • B: • A: Which do you prefer - Brighton or Worthing? • B:

26. Problem: • In contrast to Turing’s allusion to intelligent behaviour, the Loebner machine entries tend to be: • Computationally disciplined • Fooled by nuances • Relatively random • Repetitive

27. Human Intelligence • Judges/3rd Parties can regard a human entity as a machine when: • They try to avoid a direct question • Repeat themselves • Respond too quickly • Throw in a random point • Do not answer

28. Conclusions • The important problem of what exactly is intelligence and how it works is not answered by the Loebner competition. • What we are left with are a series of subjective scores from Judges. • With TIG, are we moving the goalposts in what we expect machines to achieve and/or are we becoming more streetwise to machine utterings? • Anyway isn’t TIG restricting machines to communicate in the same poor way that humans do? Why not humans/cows?

29. Comments - 2 • TIG is a lot about first impressions – importance of only 5 minutes • How much do we decide on an individual’s intelligence in the first few minutes? • How they look, what they wear, what noises they make, what academic titles or position they apparently have