Social Interaction with Agents

1. Social Interaction with Agents Lewis Johnson Director, CARTE USC / Information Sciences Institute

2. Jeff Rickel 1963 - 2003

3. Rationale: Reeves & Nass’s Media Equation • People tend to relate to computers and other media as they do to people • Confirmed by multiple experimental studies • Synthetic agents exploit this tendency

4. Claims • Synthetic agents raise expectations of • Ability to understand user’s activities • Social interaction skills, i.e., social intelligence • The challenge: to meet these expectations • Primary focus: pedagogical agents • Educational context helps constrain the problems • Implications for other types of human-agent interaction

5. Social Intelligence Implies: • Ability to model people, other agents • May include goals, plans, emotions, motivations, personality, etc. • Ability to engage in social interaction • Sensitive to the model of the person or agent • Sensitive to the social context • In coordination with task activities • To motivate, influence, develop rapport

6. Adele Pedagogical Agent

7. Some Failures in Social Intelligence • Criticizing the same mistake over and over • Interrupting the learner after minor mistakes • Giving impression of negative emotional reactions to learner’s actions • Failing to show respect for the learner’s work • Failing to offer encouragement when the learner needs it • Failing to offer help when the learner is confused and frustrated

8. Teasing Apart Social Intelligence

9. One Aspect of SI: Team Coordination • Stems from an agent-oriented view of human-computer interaction • Learner, virtual tutor, other agents act together as a team • Roles and responsibilities dynamically allocated among team members • Learning activities, teamwork, and task work are flexibly integrated • Examples: Steve, MRE

10. Steve Team demo

11. Mission Rehearsal Exercise

12. Steve/MRE Team/Task Model • Team task represented as a hierarchical, nonlinear plan • Including possible alternative courses of action • Courses of action evaluated for expected utility • Step responsibilities (and authorities) assigned to team members • Possibility of dynamic sharing of responsibilities • Each team member has own (possibly partial) model of team plan & status Rickel & Johnson, IJCAI ‘01; Traum, Rickel, Gratch, & Marsella, AAMAS ‘03 See also: Scerri et al., AAMAS ’03; Davies et al., IUI ‘01

13. Another Aspect of SI: Face-to-Face Communication • Example: MRE • Dialog model tracks state of communication among team members • Components: • Which team members are in contact • Who and what is being attended to • Who are participants (or overhearers) in the conversation • Who has the conversational initiative • Common knowledge • Including social commitments to actions and to facts • Communicative acts and grounding acts • Negotiation acts • Eye gaze, nonverbal gestures signal attention, grounding Marsella, et al., 2003; Traum & Rickel, AAMAS ’02; Traum et al., AAMAS ‘03 Dialog demo

14. Another Aspect of SI: Interaction Tactics • How to help the learner • Respecting learner’s autonomy & sense of control • How to influence the learner • Motivating the learner as needed • When not to help the learner • Reinforcing autonomy, engagement • Assume appropriate social stance toward the learner • Interaction in the context of a social relationship • Bottom line: • Influence of social relationships on human-agent interaction • Rhetoric for human-agent interaction

15. Example: Carmen’s Bright IDEAS Marsella, Johnson, & LaBore: Agents 2000, AI-Ed ‘03

16. Praise Answer Question Reassure Gina’s Dialog Model • Gina’s main struggle: Get Carmen thru the I-D-E-A-S steps • At each step, suggest a joint strategy (e.g. “old 5Ws”) • Prompt/motivate Carmen thru that strategy • React to Carmen’s emotional and cognitive state • Employ interaction tactics based upon Carmen’s state; some focus on cognitive state, others on emotional state Suggest Strategy Prompt Next Step Summarize

17. Questions About the “Gina Model” • How appropriate is it for educational applications? • It is based on clinicians’ counseling-oriented view of training • It is a dramatization of instructional interaction • Built from a deconstructed script • Empirical studies of tutorial interaction were needed • To see how this model applies to other educational settings • To determine which learner characteristics are most relevant • To study social interaction processes in such settings

18. Experimental Study • Videotaped sessions of computer-based learning with human tutors, over multiple sessions • Students read tutorial on line and perform series of exercises with Virtual Factory Teaching System Johnson, Pain, Shaw, et al: IUI ’03, AIEd ‘03

19. Conclusions from Study • Wide variation in learners’ preferred interaction styles • Some prefer collaboration, some prefer working alone • Wide variation in confidence • Between subjects • Over time • Tutor generally able to assess learner confidence, ability, preferred interaction style

20. Conclusions from Study (Cntd.) • Information used by tutor: • Expectations from knowledge of task • Eye gaze, mouse location • Verbal feedback from student

21. Techniques for Promoting Learner Engagement • Tutor phrased comments in order to reinforce learner control and joint activity. E.g.: • “Why don’t you go ahead and read your tutorial factory” • “You want to save the factory” • “I’d skip this paragraph” • “So why don’t we do that?” • Tutor avoided giving direct instructions • Except for operating the interface

22. Theoretical Framework: Learner Motivation • Motivational factors • “Four Cs”: • Curiosity • Challenge • Confidence • Control • Learner goals and meta-goals: • Persistent goals • Attitudes toward goal achievement

23. Role of Motivational Factors • Curiosity • Employ tactics that promote inquiry • Challenge • Select tasks according to difficulty • Intervene in response to learner confusion, hard impasses • Confidence • Regulate amount of feedback • Control • Employ tactics that promote learner goal-setting • Learner goals • Employ tactics that promote learner goal identification

24. Theoretical Framework: Politeness (Brown & Levinson) • Social actors motivated by face wants • Negative face: freedom of action and freedom from imposition; autonomy • Positive face: consistent self-image, and desire that self-image is appreciated and approved of by others • Face-threatening acts pervasive in interaction • Warnings, offers, promises, challenges, emotional displays • Face threat depends upon power, distance, ranking of threats due to social context • Social actors employ politeness tactics to mitigate face threat

25. Role of Politeness Factors in Tutorial Interaction • Common tutorial actions (advice, hints) are face-threatening acts • Tactic failures impact agent’s positive face • Face threat depends upon distance • Distance depends on duration of interchange, established trust, learner’s negative face wants (preference for autonomy vs. collaboration) • Choose tactics to promote learner positive face, mitigate negative face threat • By promoting shared goals • By avoiding direct instructions • By reinforcing positive (self-)assessment of goal achievement • When dictated by social distance, learner motivational factors

26. Example Interaction Tactics • Rhetorical requests to give hints • “Can I give you a hint? Try this…” • Question reinforces learner negative face; failure to wait for answer avoids positive face threat • Hints phrased as questions • “Do you want to do x?” • Reinforces learner control (positive face), can influence learner goals (positive face) • Hints as suggestions • “You could do x.” • Similar face effects as questions

27. Interaction Tactics (Cntd.) • Hints as suggestions of joint goal • “Let’s do x.” • Suggestion mitigates negative face threat; reference to joint goal influences positive face wants; depends on learner autonomy preferences • Hints as references to tutorial authors • Deflect blame for face threat to authors • Imperative hints • Used only when blame is deflected (I.e., to interface), or possibly when distance is reduced

28. SI Text Generator • Generates text for interaction tactics • Input: type of intervention, object(s) of intervention, style of reference (e.g., as joint goal, user’s goal, etc.) • Parameters: social distance: importance of motivational influence • Common tutor wording styles captured and codified • Wording style chosen randomly if not specified

29. Architecture of SI System

30. Tracking Learner Attention & Confusion • Useful: • To detect proactive interaction opportunities • Avoids learner frustration • To avoid inappropriate interruptions • Avoids negative effects on learner affect, trust • To assess learner engagement • Helps determine objectives for interaction tactics

31. Assessing Learner Attention: Methods • Track learner’s interactions with tutorial and simulation interface • Track learner gaze • Fuse using Bayesian techniques to determine focus • Instrument tutorial with expected learner goals, time demands • Infer overall learner activity (e.g., scan vs. problem-solve), overall engagement, specific impasses

32. Student Interface

33. Student Interface with Motivational Self-Assessment

34. VFTS Plans

35. Inferring Learner Focus

36. Tracking Focus of Attention

37. A Final Comment: Social Actors or Dramatic Actors? • Social actor view: • Interaction with agent is a social interaction • Agent should act in a manner consistent with human social interaction • Inspiration: theories of social interaction • Dramatic actor view: • Interaction is part of an unfolding story • Agent should act so as to contribute to the story and its message • Make action clear, understandable, and engaging • Inspiration: theories of drama and narrative • Intersection: • Social theories of presentation of self – e.g. Goffman • Relationship between rhetoric and drama in effective communication