Development of a Cognitive Agent

1. Development ofa Cognitive Agent Stuart C. Shapiro Department of Computer Science and Engineering and Center for Cognitive Science State University of New York at Buffalo shapiro@cse.buffalo.edu S.C. Shapiro

2. Outline • Introduction • Intensional Representation • Logic for NLU and Commonsense Reasoning • Discussing Propositions and Sentences • Symbol-Grounding by Perception and Action • Representation and Use of Indexicals • A Personal Sense of Time • Summary S.C. Shapiro

3. Goal • A computational cognitive agent that can: • Understand and communicate in English; • Discuss specific, generic, and “rule-like” information; • Reason; • Discuss acts and plans; • Sense; • Act; • Remember and report what it has sensed and done. S.C. Shapiro

4. Embodied Cassie • A computational cognitive agent • Embodied in hardware • or Software-Simulated • Based on SNePS and GLAIR. S.C. Shapiro

5. SNePS • Knowledge Representation and Reasoning • Propositions as Terms • SNIP: SNePS Inference Package • Bi-Directional Inference • SNeBR: SNePS Belief Revision • SNeRE: SNePS Rational Engine • Interface Languages • SNePSUL: Lisp-Like • SNePSLOG: Logic-Like • GATN for Fragments of English. S.C. Shapiro

6. GLAIR Architecture Grounded Layered Architecture with Integrated Reasoning Knowledge Level NL SNePS Perceptuo-Motor Level Sensory-Actuator Level Vision Sonar Motion Proprioception S.C. Shapiro

7. Interaction with Cassie (Current) Set of Beliefs [SNePS] English (Statement, Question, Command) Reasoning Clarification Dialogue Looking in World GATN Parser (Updated) Set of Beliefs [SNePS] (New Belief) [SNePS] Answer [SNIP] Actions [SNeRE] GATN Generator Reasoning English sentence expressing new belief answering question reporting actions S.C. Shapiro

8. Cassie, the FEVAHR S.C. Shapiro

9. Cassie in the Lab S.C. Shapiro

10. Garnet Simulation World S.C. Shapiro

11. Outline • Introduction • Intensional Representation • Logic for NLU and Commonsense Reasoning • Discussing Propositions and Sentences • Symbol-Grounding by Perception and Action • Representation and Use of Indexicals • A Personal Sense of Time • Summary S.C. Shapiro

12. Entities, Terms, Symbols, Objects • Cassie’s mental entity: a person named Bill • SNePS term: B5 • Object in world: S.C. Shapiro

13. Intensional Representation Intensional entities are distinct even if coreferential. “The morning star is the evening star.” “George IV wondered if Scott was the author of Waverly.” S.C. Shapiro

14. McCarthy’s Telephone Number Problem Mary's telephone number is Mike's telephone number. I understand that Mike's telephone number is Mary's telephone number. Pat knew Mike's telephone number. I understand that Pat knew Mike's telephone number. Pat dialed Mike's telephone number. I understand that Pat dialed Mike's telephone number. S.C. Shapiro

15. Answering the Telephone Number Problem Did Pat dial Mary's telephone number? Yes, Pat dialed Mary's telephone number. Did Pat know Mary's telephone number? I don't know. S.C. Shapiro

16. Outline • Introduction • Intensional Representation • Logic for NLU and Commonsense Reasoning • Discussing Propositions and Sentences • Symbol-Grounding by Perception and Action • Representation and Use of Indexicals • A Personal Sense of Time • Summary S.C. Shapiro

17. Logic for NLU &Commonsense Reasoning Either Pat is a man or Pat is a woman or Pat is a robot. I understand that Pat is a robot or Pat is a woman or Pat is a man. Pat is a woman. I understand that Pat is a woman. What is Pat? Pat is a woman and Pat is not a robot and Pat is not a man. S.C. Shapiro

18. + + Representation in FOPL? Man(Pat)  Woman(Pat)  Robot(Pat) but don’t want inclusive or Man(Pat) Woman(Pat) Robot(Pat) T T T F T So don’t want exclusive or either S.C. Shapiro

19. andor andor(i, j){Pi, ..., Pn} True iff at least i, and at most j of the Pi are True S.C. Shapiro

20. Outline • Introduction • Intensional Representation • Logic for NLU and Commonsense Reasoning • Discussing Propositions and Sentences • Symbol-Grounding by Perception and Action • Representation and Use of Indexicals • A Personal Sense of Time • Summary S.C. Shapiro

21. Discussing Propositions That Bill is sweet is Mary's favorite proposition. I understand that Mary's favorite proposition is that Bill is sweet. Mike believes Mary's favorite proposition. I understand that Mike believes that Bill is sweet. S.C. Shapiro

22. Discussing Sentences Hamlet said `To be or not to be'. I understand that Hamlet said `To be or not To be'. Polonius said `Neither a borrower nor a lender be'. I understand that Polonius said `Neither a borrower nor a lender be'. Who said `To be or not to be'? Hamlet said `To be or not To be'. S.C. Shapiro

23. Representing Propositionsand Sentences Propositions and sentences must be First-class entities of the domain Represented by terms S.C. Shapiro

24. Outline • Introduction • Intensional Representation • Logic for NLU and Commonsense Reasoning • Discussing Propositions and Sentences • Symbol-Grounding by Perception and Action • Representation and Use of Indexicals • A Personal Sense of Time • Summary S.C. Shapiro

25. Symbol Grounding byPerception and Action • Agent can demonstrate understanding • Of properties and descriptions • By locating correct objects in the world • Of verbs • By acting properly in the world S.C. Shapiro

26. Symbol-Grounding 1 Go to the green robot. S.C. Shapiro

27. Symbol-Grounding 2 Go to the green robot. I went to the green robot. I am near the green robot. S.C. Shapiro

28. Alignment • Terms at GLAIR KR level • Aligned with • Descriptions and procedures • At GLAIR PM level S.C. Shapiro

29. Outline • Introduction • Intensional Representation • Logic for NLU and Commonsense Reasoning • Discussing Propositions and Sentences • Symbol-Grounding by Perception and Action • Representation and Use of Indexicals • A Personal Sense of Time • Summary S.C. Shapiro

30. Representation and Use of Indexicals • Words whose meanings are determined by occasion of use • E.g. I, you, now, then, here, there • Deictic Center <*I, *YOU, *NOW> • *I: SNePS term representing Cassie • *YOU: person Cassie is talking with • *NOW: current time. S.C. Shapiro

31. Analysis of Indexicals(in input) • First person pronouns: *YOU • Second person pronouns: *I • “here”: location of *YOU. S.C. Shapiro

32. Generation of Indexicals • *I: First person pronouns • *YOU: Second person pronouns • *NOW: used to determine tense. S.C. Shapiro

33. Use of Indexicals 1 Come here. S.C. Shapiro

34. Use of Indexicals 2 Come here. I came to you, Stu. I am near you. S.C. Shapiro

35. Use of Indexicals 3 Who am I? Your name is ‘Stu’ and you are a person. Who have you talked to? I am talking to you. Talk to Bill. I am talking to you, Bill. Come here. S.C. Shapiro

36. Use of Indexicals 4 Come here. I found you. I am looking at you. S.C. Shapiro

37. Use of Indexicals 5 Come here. I found you. I am looking at you. I came to you. I am near you. S.C. Shapiro

38. Use of Indexicals 6 Who am I? Your name is ‘Bill’ and you are a person. Who are you? I am the FEVAHR and my name is ‘Cassie’. Who have you talked to? I talked to Stu and I am talking to you. S.C. Shapiro

39. Outline • Introduction • Intensional Representation • Logic for NLU and Commonsense Reasoning • Discussing Propositions and Sentences • Symbol-Grounding by Perception and Action • Representation and Use of Indexicals • A Personal Sense of Time • Summary S.C. Shapiro

40. A Personal Sense of Time • *NOW contains SNePS term representing current time. • *NOW moves when Cassie acts. S.C. Shapiro

41. find Representation of Time before after before after ! ! ! event ????????????? time agent act B1 action object B6 I lex NOW S.C. Shapiro

42. before after before after ! t2 ! t3 NOW NOW NOW Movement of Time t1 S.C. Shapiro

43. before after before after ! t2 ! t3 time ! event NOW NOW Performing a Punctual Act t1 S.C. Shapiro

44. before after ! t2 supint ! subint t3 time ! event NOW NOW Performing a Durative Act t1 S.C. Shapiro

45. Keeping Track of Time 1 Who are you looking at? I am looking at you, Stu. Come here. S.C. Shapiro

46. Keeping Track of Time 2 Who are you looking at? I am looking at you. Come here. I came to you. I am near you. Who are you looking at? I am looking at you. S.C. Shapiro

47. Keeping Track of Time 3 Find Bill. I found Bill. I am looking at Bill. Who are you looking at? I looked at you and I am looking at Bill. Who are you talking to? I am talking to you. S.C. Shapiro

48. Keeping Track of Time 4 Follow a red robot. I found a red robot. I am looking at a red robot. S.C. Shapiro

49. Keeping Track of Time 5 Follow a red robot. I found a red robot. I am looking at a red robot. I went to a red robot. I am near a red robot. I am following a red robot. S.C. Shapiro

50. Keeping Track of Time 6 Who are you talking to? I am talking to you. Who am I? Your name is ‘Stu’ and you are a person. Stop. I stopped. S.C. Shapiro