Does History Help?

1. Does History Help? An Experiment on How Context Affects Crowdsourcing Dialogue Annotation Elnaz Nouri Computer Science Department University of Southern California Natural Dialogue Group, Institute for Creative Technologies

2. Crowdsourcing Annotation Faster (?) Cheaper (?) Quality (?) Snow(2008)

3. In Crowdsourcing Dialogue Annotation Tasks • Dialogue data is sequential by nature. • Does providing context from previous parts of the dialogue (e.g. turns) affect the annotation of the target part? Example: Judge the sentiment on the following turn of the dialogue: Person 1: Come on out, honey! I'm telling you look good! Tell her she looks good, tell her she looks good. Person 2: Oh my God, you look so good!

5. From Seinfeld… WAITRESS : Tuna on toast, coleslaw, cup of coffee. GEORGE: Yeah. No, no, no, wait a minute, I always have tuna on toast. Nothing's ever worked out for me with tuna on toast. I want the complete opposite of on toast. Chicken salad, on rye, un-toasted with a side of potato salad ... and a cup of tea. JERRY: You know chicken salad is not the opposite of tuna, salmon is the opposite of tuna, 'cuz salmon swim against the current, and the tuna swim with it. GEORGE: Good for the tuna! Link to Video

6. Interesting Questions • General aspect: • Do annotators need context to do each instance of the annotation? • Can we present them with only the needed previous context? • How does context affect stability of the annotation? • Crowdsourcing aspect: • Should we present the whole dialogue to annotator if the compensation rate is low? • Can we consider each annotation task as a stand alone micro task? • Do annotators on Amazon Mechanical Turk read the instructions or the context provided?

7. So we ran an experiment…

8. How is it going? I am Bronson from the Hill restaurant. The Idea: A Variable Context Window Size I am Milton. I am from the Vally restaurant. Alright, cool. So looks like we got some good resources on the table. And, uh, we want to find a way that works for both of us. Uh, yeah I agree. I just want to, we want to maximize both of our profits. So what do we have right here?

9. The Data Set • The “Farmers Market” negotiation dataset • 41 dyadic sessions of negotiation based on instructions • Two restaurant owners are trying to divide some items among themselves

10. 3 dialogues used: D1 (31 turns), D2 (16 turns), D3 (30 turns) = 77 turns • 5 annotators for each instance: A1, A2, A3, A4, A5 • annotators recruited on Amazon Turk • $0.02 for annotating each instance • “Sentiment Annotation Task”on the turns of the dialogue The Task: Sentiment Analysis Task

11. Previous Context Window Size =3 Example Stimuli

12. Example Annotation Result Gold annotation: the whole dialogue was presented to the annotator

13. Evaluation Method 1: Distance to the Gold Annotation (* shows the minimum distance from Gold annotation)

14. Evaluation Method 2: Inter-annotator Agreement Hypothesis: higher inter-annotator reliability implies more stability indicator of the optimal context window size The differences between window sizes were not significant according to t-test except for that of 0 window size. * shows the maximum agreement

15. Conclusions (and considerations) Our results imply that: • the number of previous turns doesn’t really affect the annotation of the target  not necessary to show a big number of previous turns or the whole dialogue. • A context window size of 3 is perhaps enough to do the job. Considerations: • sample size is very small • the nature of the dialogues and the negotiation task might have affect the results • our dataset wasn’t too emotional!  • these are not real negotiation or conversations • the annotation task can also affect the outcome

16. Future Work Further investigation is needed: • Different datasets • Different annotation tasks • Appropriate metrics for measuring • Suitable baseline annotation for comparison Questions? Please tell me what you think! Your feedback and ideas are sincerely appreciated!