1 / 39

deLearyous

deLearyous. Frederik Vaassen, Walter Daelemans Jeroen Wauters, Frederik Van Broeckhoven, Maarten Van Overveldt , Karen Vanderloock , Koen Eneman. Training Interpersonal Communication Skills Using Unconstrained Text Input. Overview. Introduction Leary’s Rose Test Case

sanne
Download Presentation

deLearyous

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. deLearyous Frederik Vaassen, Walter Daelemans Jeroen Wauters, Frederik Van Broeckhoven, Maarten Van Overveldt, Karen Vanderloock, Koen Eneman Training Interpersonal Communication Skills Using Unconstrained Text Input

  2. Overview • Introduction • Leary’s Rose • Test Case • System Architecture • Scenario • Natural Language Processing • Scenario Engine • Audio • Visualization • FutureWork

  3. Introduction • deLearyous - Training Interpersonal Communication Skills through Natural Language InteractionwithAutonomous Virtual Characters • Serious game • Virtual conversation partner • Train the user’scommunication skills • Writtennaturallanguage input • “Realistic” reactions • (In Dutch)

  4. Leary’S Rose Framework forInterpersonal Communication

  5. Leary’S Rose • deLearyous • Timothy Leary (1957) –Interpersonal Diagnosis of Personality: Functional Theory and Methodology for Personality Evaluation • Leary’s Rose /Interpersonal Circumplex: • Position of speaker vs. hearer in dialogue • Two axes: • Above – below • Opposed – together • 8 octants / 4 quadrants

  6. Leary’S Rose • Examples: • My name is John, how can I be of assistance? helping • What do you suggest we do about this? dependent • That’s not my fault, administration’s not my responsibility! defiant • If you’re going to be rude there’s no use in continuing this conversation!  aggressive

  7. Leary’S Rose • Interpersonal dynamics: • ABOVE triggers BELOW • BELOW triggers ABOVE • OPPOSED triggers OPPOSED • TOGETHER triggers TOGETHER

  8. Leary’s Rose • Teachplayersto: • Control instinctivereactions • Consciouslychoosetheirposition on the Rose • Achieve the intended effect on the listener

  9. Test Case • Proof-of-concept on a well-defined scenario: • Conversationbetweenemployer (player) and employee (Virtual Agent / VA) • “parking spaceswill no longerbe free”

  10. Test Case • Three difficulty levels

  11. Test Case • Whicharguments are commonlyused? • Any special conversationmechanisms we needto take into account? • How do argumentsrelatetopositions on Leary’s Rose? • Facial expressions, body language, tone of voice? • … •  “Wizard of Oz” tests

  12. Test Case • “Wizard of Oz” tests • Results: • State chart of possibleargumentsandtheir links • Training set of sentencesannotatedwiththeirposition on Leary’s Rose audio video • Player • VA • Actor text

  13. System Architecture • Scenario • Text Input • NLP • ScenarioEngine • Audio • Visualization

  14. System Architecturescenario • Scenario • Text Input • NLP • ScenarioEngine • Audio • Visualization

  15. SCENARIO • Twolists of statements • VA (virtual actor) statements • PL (player) statements • Stored as XML-file • Pluggable! • New scenarioscanbeaddedby non-programmers

  16. SCENARIOVA statementS • Id • Title • Onephrasing per octant • Oneneutralphrasing

  17. SCENARIOPL statementS • Id • Title • List of keywords • List of possible follow-up VA statements

  18. System Architecturenaturallanguage processing • Scenario • Text Input • NLP • ScenarioEngine • Audio • Visualization

  19. Natural Language Processing • Two sub-modules: • LearyClassifierdetermine the position of the player on Leary’s Rose • Topic Detectionidentify the state correspondingto the user input

  20. Natural Language ProcessingLearyClassifier • Training Vectors(withposition) • Feature Extraction • Parser • Training Sentences • Machine Learner • Model Training • Test Vector(no position) • Feature Extraction • Parser • Test Sentence PredictedPosition Testing

  21. Natural Language ProcessingLearyClassifier Classifier Performance • Relatively low performance • Still well above random •  Whatabouthumans?

  22. Natural Language ProcessingLearyClassifier • Low agreement between human annotators • Difficulttask even forhumans • Wizard of Ozconversationsworked well despite low agreement Agreement between 4 annotators on a random conversation

  23. Natural Language ProcessingTopic Detection • Keyword matching • Every state has its keywords • Definedmanually! Limit amount of manual work • Potentialproblems: • Synonymsmeasure != decision • Word formsmeasures != measure • Negationfinal != !final • Decision is final • decision, final • “We’re only considering these measures, they’re not yet final.” • Decision is NOT final • decision, !final

  24. Natural Language ProcessingTopic Detection • Synonyms: • Automatic expansion of wordswithsynonymsandrelatedterms (Cornetto)measure > decision • Word forms • Lemmatizationmeasures > measure • Negation: • Automatic negationdetection(based on parser output)final> !final •  Broadcoveragewith a minimum of manual work • Decision is final • decision, final • “We’re only considering these measures, they’re not yet final.” • Decision is NOT final • decision, !final

  25. Natural Language ProcessingTopic Detection • FurtherEnhancements • Keyword weightsvery common, lessinformativewords lowerweight • Linked keywordsset+in+stone>final • Result: broadcoveragewith a minimum of manual work

  26. System Architecturescenario engine Scenario Text Input NLP Audio Visualization ScenarioEngine

  27. Scenario Engine • Twomain jobs • update current VA Leary-positionbased on player octant • Based on axisdynamics • Select VA response • Based on links in scenario • Special statements mayoverride links!

  28. Scenario Engine • Special statements • Statement withspecific trigger, specific action, or both • Examples • Reactionto insult • Trigger: at leastoneoffensive word in player input • Storm out of the room • Trigger: VA is in negative octant, player has pushed in wrong direction multiple times • Action: play “storming out” animationand end conversation

  29. System Architectureaudio Scenario Text Input NLP Audio Visualization ScenarioEngine

  30. Audio P1 P2 • Professional actor usedto record all statements • Extra work, but also extra benefit • The reply of the VA is played in his currentLeary octant • Problem: sudden changes of position • Solution: useneutralphrasingwhen close to center of Rose

  31. System Architecturevisualization Scenario Text Input NLP Audio Visualization ScenarioEngine

  32. VISUALIZATION

  33. VISUALIZATION Body Background Head

  34. VISUALIZATIONbackground • Visual styleadjustableto in-house company style • Colorschemeconfigurablethrough scripts • Logo and paper texturescanadapted Result:

  35. VISUALIZATIONbackground Result

  36. VISUALIZATIONbody • Map body positionstoCircumplex • Examples: • Lean forward • Arms crossed • Lean backward • Avoideye contact

  37. VISUALIZATIONFACE • Map facial expressions (6 basic emotions) toCircumplex • Lip Synchronization 1 Neutral 2 Disgust 3 Happiness 4 Sadness 5 Fear 6 Surprise 7 Anger

  38. FutureWork • Furtherimprovements in NLP and Scenario Manager • Focus testing • Technical • How naturaldid the conversation feel? • Influence of erroneousLearyclassifications? • Influence of wrong topic matches? • Learning goals • Can the playerlearnanythingfromplaying the game? • How much human supervision is required?

  39. Contact: frederik.vaassen@ua.ac.bejeroen.wauters@groept.be

More Related