PIER Research Methods Protocol Analysis Module

1. PIER Research Methods Protocol Analysis Module Hua Ai Language Technologies Institute/ PSLC

2. Questions? • "There was a significant negative correlation between the first and third metrics used to compute a Power score for the partner's conversational contributions and this question's numeric value, and a marginal negative correlation in the case of the second metric.“ • What are we supposed to interpret/learn from this statement?

3. Rose at el. • Automatic Analysis • By Machine Learning

4. Machine learning is about automatically finding meaningful patterns in data Example for medical data: Rule predicts who is more likely to have problems with their teeth as they get older. What is machine learning?

5. Why Machine Learning? • We use machine-learning products every day • Weather forecast • Spelling checker • Automated voice response system • … • It has been successfully applied to many research areas • Natural language processing • Market analysis • Bioinformatics • … Note: Search engines use machine learning to personalize search results and suggest related sites or queries.

6. Outlook: Sunny -> No Overcast -> Yes Rainy-> Yes A slightly more sophisticated rule learner will find the feature that gives the most information about the result class. What do you think that would be in this case? The simplest rule learner will learn to predict whatever is the most frequent result class. This is called the majority Class. <Feature Name>: <value> -> <prediction> <value> -> <prediction> … What will the rule be in this case? It will always predict yes. How does machine learning work?

7. Classification Engine Learning Algorithm Data Model Prediction New Data What is machine learning? • Automatically or semi-automatically • Inducing concepts (i.e., rules) from data • Finding patterns in data • Explaining data • Making predictions

8. Outlook: Sunny -> No Overcast -> Yes Rainy-> Yes Yes What will be the prediction? Model New Data

9. Terminology • Concept: the rule you want to learn • Instance: one data point from your training or testing data (row in table) • Attribute: one of the features that an instance is composed of (column in table) bad * Compute the predicted value.

10. Task • Assign labels to a collaborative learning corpus using the Weinberger and Fischer’s coding scheme • Text classification task

11. Two approach categories • The Feature Based Approach • Basic feature • Thread Structure feature – depth • FSM features • LIWC features (Linguistic Inquiry Word Count) • The Algorithm Based Approach • Cascaded Binary Classification • Confidence Restricted Cascaded Binary Classification • Supervised/Unsupervised

12. Methodology issues related to automatic corpus analysis • Validity • Whether the automatic coding accomplished by the computer captures human analysts’ intention • Reliability • How faithfully the automatic codes match those of human experts’ • Efficiency • Are we saving time by using the automatic classifier?

13. Ai et al. • Manual Analysis • By statistical tests • Interaction between tutor’s social behaviors and tutor’s bias towards one student’s stance or the other • Topic modeling: sub-dialog level instead of utterance level

14. Study design • 3*3 • Social: how, low, none • Goal Match: yes, no, neutral

15. The ccLDA model Green Collection Power Collection Topic 3 Topic 2 Topic 1 Topic 2 Topic 3 Topic 1 A topic is a distribution of words. We computed a score for each utterance based on the words contained in the utterance. This score stands for to what extend this utterance is biased towards this topic.

16. Results • Learning Gain • Student learned most in Low-social + Yes-matched • Perception (Questionnaire & Topic detection) • Effect on Goal-match manipulation • Conversational Data • More social turns in social conditions • More off-task turns in non-social condition • More Jokes on tutor in the high social condition

17. Important message • Understand the data is important • Design good features • Both in automatic and manual analysis