Displayed Bias as a Reflection of Both Speaker and Intended Hearer in Conversational Settings

1. Displayed Bias as a Reflection of Both Speaker and Intended Hearer in Conversational Settings Carolyn Penstein Rosé Language Technologies Institute and Human-Computer Interaction Institute With funding from the National Science Foundation and the Office of Naval Research

2. For more information about my group: http://www.cs.cmu.edu/~cprose Acknowledgements • Nguyen, D., Mayfield, E., & Rosé, C. P. (2010). An analysis of perspectives in interactive settings, in Proceedings of the KDD Workshop on Social Media Analytics. • Kumar, R. & Rosé, C. P. (2010). Engaging learning groups using Social Interaction Strategies, In Proceedings of the North American Chapter of the Association for Computational Linguistics. • Howley, I., Mayfield, E. & Rosé, C. P. (to appear). Linguistic Analysis Methods for Studying Small Groups, in Cindy Hmelo-Silver, Angela O’Donnell, Carol Chan, & Clark Chin (Eds.) International Handbook of Collaborative Learning, Taylor and Francis, Inc. • Ai, H., Kumar, R., Nguyen, D., Nagasunder, A., Rosé, C. P. (2010). Exploring the Effectiveness of Social Capabilities and Goal Alignment in Computer Supported Collaborative Learning, in Proceedings of Intelligent Tutoring Systems. Iris Howley Hua Ai Iris Howley Hua Ai Dong Nguyen Dong Nguyen Elijah Mayfield Rohit Kumar Elijah Mayfield Rohit Kumar

3. Outline • Motivation from Opinion Mining • Theoretical framework from Rhetoric and Discourse Analysis • Study one: Political bias in a political discussion forum • Study two: Goal orientation in chat based design discussions • Current Directions

4. Outline • Motivation from Opinion Mining • Theoretical framework from Rhetoric and Discourse Analysis • Study one: Political bias in a political discussion forum • Study two: Goal orientation in chat based design discussions • Current Directions

5. KEY ASSUMPTION: language is a reflection of the speaker’s perspective KEY ASSUMPTION: language is a reflection of the speaker’s perspective Are product reviews conversational? • Typical paradigm for sentiment analysis of product reviews: • Make a prediction based on text of single reviews taken out of context • Some evidence of group effects in product review blogs based on numerical ratings (Wu et al., 2008)

6. KEY ASSUMPTION: language is a reflection of the speaker’s perspective KEY ASSUMPTION: language is a reflection of the speaker’s perspective Are product reviews conversational? • Work towards weakening the assumption • Sometimes use syntactic cues to reverse polarity on some terms (Somasunderan & Wiebe, 2009; Wijaya & Bressan, 2008) • Factoring out the effect of context rather than modeling it • Aggregation over all reviews posted by the same individual • Taking opinions of similar individuals into account (e.g., collaborative filtering) • Falls short of modeling conversational aspects of product reviews

7. Are product reviews conversational? • “After many MANY weeks of research, gathering information from several sites, reviews etc I decided that the Britax Boulevard was definitely the safest bet available on the market. The things that sold me: All the safety gadgets that other seats don't have like the side impact wings, the HUGS system, the LATCH system and 5 point harness and also the fact that it lasts up to 29Kg. “

8. Are product reviews conversational? • “I did most of my research on the net, picking my top 3 choices I went and had a look at them in the shops. I looked at one the Graco Comfort Sport, the Britax Boulevard and the Decathlon and Marathon seats. By far it seems that Britax have the upper hand safely wise on the market, many professional reviews and crash tests agree on this so Britax was the clear choice for us. “

9. Are product reviews conversational? • “I have the seat front facing in my Camry (2007) I worried about the size of the chair from reading other reviews but that is NO problem in my car, my son has plenty of leg room and can see perfectly out the window.”

10. Are product reviews conversational? http://www9.georgetown.edu/faculty/irvinem/theory/Bakhtin-MainTheory.html

11. Outline • Motivation from Opinion Mining • Theoretical framework from Rhetoric and Discourse Analysis • Study one: Political bias in a political discussion forum • Study two: Goal orientation in chat based design discussions • Current Directions

12. Discourse and Identity • Identity is reflected in the way we present ourselves in conversational interactions • Reflects who we are, how we think, and where we belong • Also reflects how we think of our audience • Examples • Regional dialect: shows my identification with where I am from, but also shows I am comfortable letting you identify me that way • Jargon and technical terms: shows my identification with a work community, but also shows I expect you to be able to relate to that part of my life • Level of formality: shows where we stand in relation to one another • Explicitness in reference: shows whether I am treating you like an insider or an outsider

13. Discourse and Identity • Discourse is text above the clause level (Martin & Rose, 2007) • A Discourse is an ongoing conversation [type] • Socialization is the process of joining a Discourse (Lave & Wenger, 1991; Sfard, 2010) • We join Discourses that match our core identity (de Fina, Schiffrin, & Bamberg, 2006) • In moving from the periphery to the core of a Discourse community, we sound more and more like the community (Arguello et al., 2006) • A discourse is one instance of it [token] • All discourses contain echoes of previous discourses (Bakhtin, 1983) Lakoff & Johnson, 1980 Lave & Wenger, 1991

14. Metaphors Structure our Experience • We describe arguments using terms related to war • Using a typical war ‘script’ to structure a story about an argument • We orient towards arguments as though they were wars • Our conversational partner is our opponent • We may feel that we won or lost • We may feel wounded as a result

15. Discourses, Frames, and Metaphors • Frame: A portion of a discourse belonging to distinct Discourse • Metaphor : One linguistic device that can be used to define a set of discourse practices that constitute a frame • Topic models: a technical approach that makes sense for identifying frames within a discourse • A discourse could be drawn from a mixture of Discourses • Within the same conversation, we may wear a variety of “hats” • E.g., the same discourse with a co-worker may contain exchanges pertaining to our relationship as colleagues and others to our relationship as friends

16. Model of Communication from Rhetoric Author • Implied author: Communication style is a projection of identity • Impression management, not necessarily the ground truth • Implied reader: What we assume about who is listening • Real assumptions, possibly incorrect • What we want recipients or overhearers to think are our assumptions • Reader: may or may not understand the text the way it was intended Implied Author Implied Reader Text Effect Reader

17. Engagement: Social positioning in conversational style Author • The message: Most contributions express some content • Implied author: How I phrase it says something about my stance with respect to that content • Implied reader: Also says something about what I assume is your stance and my stance in relation to you • Reader: The hearer may respond either to the message or its positioning Implied Author Implied Reader Text Effect Reader

18. Engagement: Social positioning in conversational style Author • Speaker 1: I want chocolate for dessert. • Speaker 2: [you can’t have chocolate] • Options with different implications about author and reader • You can’t have chocolate. • You’re allergic to chocolate. • You’re allergic to chocolate, so eating it would be a bad idea. • Your mom said you’re allergic to chocolate • Having chocolate might be a poor choice for you. • Having chocolate might be a poor choice for you for a great number of reasons. Implied Author Implied Reader Text Effect Reader

19. Even Scientific Writing is Social and Conversational Author • Implied author: Rhetorical style in academic writing gives an impression of who we are as researchers • Implied reader: targeting writing to community standards • Abstracts and literature reviews position us in research community • Reader: research papers teach us both about the content of our field and its politics Implied Author Implied Reader Text Effect Reader

20. Outline • Motivation from Opinion Mining • Theoretical framework from Rhetoric and Discourse Analysis • Study one: Political bias in a political discussion forum • Study two: Goal orientation in chat based design discussions • Current Directions

21. Bias Estimation • Start with LDA model of politics dataset with 15 topics • Then separate the texts into two collections, one left affiliated, and one right affiliated • We then have a Left model and a Right model • We can then compute a rank for each word w in each topic t in each model • Intuition: a word is more distinguishing for a particular point of view if it has a high probability within the associated model and a low probability in the opposite model • Bias(w,t) = log(rankright(w,t) + 1) – log(rankleft(w,t) + 1) • The bias of a text is the average bias over the terms within the text • Left scores positive, right scores negative

22. Qualitative Analysis • Terror Language (Right): evokes emotional response to thread of attack. Define target as evil and as a threat. Provokes a defensive posture. • Imperialist rhetoric (Right): racial prejudice, attitude of superiority. • Web of concern (Left): focus on opposition as individuals with a culture and history, concern for wellbeing of all people, focus on potential negative effects of war

23. Right Bias Left Bias Quantitative Analysis • Score of poster • Score of quoted message • Score of full post • Score of words that appear in both messages • Score of words that appear only in quoted message • Score of words that appear only in the post

24. Investigation of Quoting behavior • Negative correlation between words only in quoted message and words only in post (r=-0.1, p < 0.05) • Positive correlation between score quoted words and score of the whole post (r=0.18, p < 0.02) • Score of words only in post are significantly more reflective of the affiliation of the poster than that of the author of the quoted message • Similar result with score of words only in quote with affiliation of author of quoted message

25. Investigation of Quoting behavior

26. Investigation of Quoting behavior by pointing out the inflationof Saddam’s body count by neoconsin an effort to further vilify him and thus further justify our invasion we are not DEFENDING saddam....just pointing out how neoconsrarely let facts get in the way of a good war. So wait, how many do you think Saddam killed or oppressed? You’re trying to make him look better than he actually was. You’re the one inflatingthe casualties we’ve caused! Seriously, what estimates (with a link) are there that we’ve killed over 100,000 civilians. Not some crack pot geocities page either. Which words are quoted?

27. Thread level analysis • Effect of initial post • Correlation between score thread (without first post) and first post = 0.210 (p<0.01) • Effect of Prior posts • Aggregate score of previous posts • Difference in score of current post and average score of user • Small correlation (r=0.133, p < 0.01) indicating that users talk more left than they usually do when previous posts are left and similarly for right.

28. Overview of Findings • Quotes from opposite point of view include the words that are less strongly associated with the opposite perspective • Because of quotes, displayed bias shifts towards the bias of the person to whom the message is directed • Personal bias of the speaker is most strongly represented by non-quoted portions of text • The effect of a post extends past just the immediate response

29. Outline • Motivation from Opinion Mining • Theoretical framework from Rhetoric and Discourse Analysis • Study one: Political bias in a political discussion forum • Study two: Goal orientation in chat based design discussions • Current Directions

30. Collaborative Design Task • Goal: Design a power plant • Competing Student Goals: • Power:Design a power plant that achieves maximum power output • Green: Design a power plant that has the minimum impact on the environment • Competing goals encourages deeper discussion • Exploration of design space • Explicit articulation of reasoning > Experimental Design >Task 30

31. Classroom Study Chat room style interaction ConcertChat 106 students CMU undergrads in ME Classroom session During the semester Instruction Pretest  Session  Posttest  Questionnaire 31 > Experimental Design >Study Procedure 31

32. Experimental Design • Social Behavior • Frequent, Infrequent, None • Goal Alignment • Green, Power, Neutral • Hypotheses • Students will be more engaged with agents displaying social behaviors • Students will be sensitive to tutor goal orientation • Interaction effect Frequent Green Infrequent Green None Green Frequent Power Infrequent Power None Power Frequent Neutral Infrequent Neutral None Neutral > Experimental Design >Manipulations 32

33. An Example of Displaying Bias Green Bias • Green: What is bad about increasing heat input to the cycle is that more waste heat is rejected to the environment. • Neutral and Power: Increasing heat input to the cycle increases waste heat rejected to the environment. Power Bias: • Power: What is good about increasing heat input to the cycle is that more power output is produced. • Neutral and Green: Increasing heat input to the cycle increases power output produced. > Experimental Design >Agent Design 33

34. Example of Social Behaviors Adapted from Bales’ IPA (Bales, 195) Tutor: Let’s Introduce ourselves. My name is Avis. Tutor: Be nice to your teammates! Tutor: I’m happy to work with our team :-) Tutor: m-hmm (showing attention) 34 > Experimental Design >Agent Design 34

35. Experimental Design • Social Behavior • Frequent, Infrequent, None • Goal Alignment • Green, Power, Neutral • Hypotheses • Students will be more engaged with agents displaying social behaviors • Students will be sensitive to tutor goal orientation • Interaction effect Frequent Green Infrequent Green None Green Frequent Power Infrequent Power None Power Frequent Neutral Infrequent Neutral None Neutral > Experimental Design >Manipulations 35

36. Measuring Student Bias Using a topic modeling tool – ccLDA [Paul and Girju, 2009] Corpus Collection1 Collection2 Topic1 Topic 2 Topic 3 36 > Experimental Design >Displaying Bias 36

37. Example of Extracted Topics TOPIC 1 TOPIC 2 Background Green Power Background Green Power > Experimental Design >Displaying Bias 37

38. Bias Measurement Metrics • Max Topic-word bias: count the number of words in the list of the N most strongly associated words, and take the maximum across topics • Average Topic-word bias: count the number of words in the list of the N most strongly associated words, and take the average across topics • Weighted Topic-Word Average bias: Same but weight each word by its association within the background model first • All three measures highly correlated both for Green and for Power perspectives • Students in the Green condition got higher Green scores on average than Power scores and vice versa in the Power condition • Only statistically significant for the first two metrics

39. Measuring Influence • Within pairs, the Green score of the Green student and the Green score of the Power student were significantly correlated • Same story for Power scores • Result consistent with analysis of Politics dataset > Experimental Design >Displaying Bias 39

40. Operationalization of Authoritativeness • Negotiation coding scheme (Martin & Rose, 2007, Chapter 7) • Agreement on K1/K2/Other • .72 Kappa • Coded all transcripts from Infrequent Social condition • Authoritativeness score = K1/[K1 + K2] • Within pairs, one Authoritative student and one NonAuthoritative student

41. Balance Effect • Alignment • Align: Authoritative partner shares affiliation with agent • Neutral:Agent is neutral • NoAlign: Non-Authoritative partner shares affiliation with agent • Affiliated agent has polarizing effect on displayed bias • Difference in bias scores was significantly higher in conditions with affiliated agents • Direction of polarization depends on alignment • Balance Effect • Authoritative student shows less of his own bias when he’s in the minority • NonAuthoritative student is less non-authoritative when he’s in the minority

42. Overview of Findings • Topic models can display differences in goal orientation in chat data • Confirmation of influence of partner speech on displayed bias • Complex relationship between personal orientation, authoritativeness, and ingroup/outgroup effects

43. Outline • Motivation from Opinion Mining • Theoretical framework from Rhetoric and Discourse Analysis • Study one: Political bias in a political discussion forum • Study two: Goal orientation in chat based design discussions • Current Directions

44. Current Directions • Full circle for opinion mining • Continuing to operationalize multiple dimensions of relational codes • Howley, I., Mayfield, E., & Rosé (to appear). Linguistic Analysis Methods for Studying Small Groups, in Hmelo-Silver, O’Donnel, & Chan (Eds.) International Handbook of Collaborative Learning, Taylor & Francis, Inc. • Collaboration with Bob Kraut: investigating how exchange of social support is reflected in relational codes • Collaboration with Bhiksha Raj: investigating evidence of relational codes like Negotiation in speech • CHI submission in progress: Analysis of effects of bullying behavior on distribution of relational codes and learning • New grant!: studying the emergence of leadership in ad-hoc teams

45. Computational Models of Discourse Analysis • Focus on literature from the field of Discourse Analysis • Investigating issues such as Conversational Structure, Attitude, Perspective, Persuasion and Positioning • Critical reflection on the state-of-the art in language technologies • Hands-on programming assignments, fun contests

46. Questions? Carolyn Penstein Rosé http://www.cs.cmu.edu/~cprose cprose@cs.cmu.edu Gates-Hillman Center 5415