High Resolution Statistical Natural Language Understanding: Tools, Processes, and Issues.

1. High Resolution Statistical Natural Language Understanding:Tools, Processes, and Issues. Roberto Pieraccini SpeechCycle roberto@speechcycle.com

2. Please choose one of the following: account balance, fund transfer, payments, mortgage rates. Account balance Please tell me what you are calling about. I want to buy a house and I would like to know how much it would cost to borrow money from the bank. Context-Free Grammars Statistical Spoken Language Understanding Directed Dialog vs.Open Prompt DIRECTED DIALOG OPEN PROMPT

3. balance account fund transfer payments mortgage ANYTHING ELSE rates NO-MATCH • <rule id=“bank" > • <one-of> • <item>”account balance”</item> • <item>“fund transfer</item> • <item>payments</item> • <item>”mortgage rates”</item> • </one-of> • </rule> handcrafted SRGS Out of grammar Context-free grammars N-best semantic categories In grammar utterances BALANCE account balance fund transfer payments mortgage rates TRANSFER PAYMENT MORTGAGE CFG

4. N-best word strings Statistical Language Model (SLM) Statistical Semantic Model (SSM) OTHER Anything Else Classifies a word string into a number of predefined categories Provides a probabilistic constraint to the speech recognition engine Bi-gram language model Statistical classifier Statistical Spoken Language Understanding (SSLU) N-best semantic categories All possible natural language expressions BALANCE TRANSFER PAYMENT MORTGAGE SSLU

5. Annotated Transcriptions I need to know how much money I have  BALANCE I need to move funds from checking to savings  TRANSFER How much would it cost to borrow money to by a house  MORTGAGE I need to pay my utility bills  PAYMENT I dialed the wrong number  OTHER … … Building SSLUs Statistical Language Model (SLM) SSLU Training Statistical Semantic Model (SSM)

6. expected responses unexpected responses More training data Increase grammar coverage, Tighten prompt The accuracy point of view DIRECTED DIALOG OPEN PROMPTS WITH SSLU High accuracy obtained by limiting unexpected responses and by controlling vocabulary and word confusability Unlimited input and uncontrolled vocabulary results in lower accuracy than directed dialog. expected responses unexpected responses When unexpected responses and user’s vocabulary can be controlled, directed dialog typically provides higher accuracy.

7. Why SSLU? • Number of options too large for directed dialog. • Please choose one of the following: clothing, automotive, hardware, appliances, …, gardening, …, bedding, … • Options make little sense to users • Do you have a hardware, software, or configuration problem? • User may chose the wrong option • Hmm…hardware? • Unexpected responses and user’s vocabulary hard to control • I need to buy a car CD player that plays MP3s In all these situations, open prompts with SSLU can outperform directed dialog.

8. Low and High Resolution SLUs • APPLICATION: Call routing • 10s of broad semantic categories • APPLICATION: Technical Support • 100s of semantic categories • Different degrees of specificity • Detailed confirmation • User model differs from underlying model • User don’t know the underlying model Low resolution High resolution

9. I have a problem with my TV service I could not order a show My movie on demand does not work I ordered a pay-per-view event but all I see is an error code on the display. Hierarchical SSLU TV Symptoms Ordering On Demand Error PIN Other Pay-per-view Error No Picture

10. I have a problem with my TV service My movie on demand does not work I ordered a pay-per-view event but all I see is an error code on the display. Hierarchical SSLU TV Symptoms Ordering I could not order a show On Demand I understand you have a problem with ordering. Is it on demand or pay-per-view? Error PIN Other Pay-per-view Error No Picture

11. In order to build good SSLU it is important to establish a repeatable process • Transcription management • Creation of annotation guide • Measure annotation consistency • Revise annotation guide • Create VUI

12. Development Cycle Tens of thousands of utterances are needed for creating high performance SLUs Transcription Remove artifacts, acronyms, misspellings Annotate according to what the user says Normalization Symptom and Annotation Guide Develop disambiguation VUI Linguistic Annotation SSLU Training Measure annotation consistency Merge or split categories for better SLU performance SSLU Test Review Annotation Guide

13. Confusion Matrix SLU RESULTS SEMANTIC TRUTH

14. Performance Analysis SSLU accuracy analysis

15. Confirmation Analysis • As a result of the confirmation prompt, users can • Accept a correct hypothesis • Accept a wrong hypothesis • Deny a correct hypothesis • Deny a wrong hypothesis • Do not confirm at all

16. Experimental VUIThe effect of the prompt

17. Conclusions • Understanding the choice between SSLU and directed dialog • High-resolution SSLU for applications with hundreds of semantic categories. • SSLU development process • Data, data, data – assessment of performance is key to success.