Supporting Natural Language with Finite State Grammars

1. Supporting Natural Language with Finite State Grammars David Claiborn Senior VUI Designer

2. State of the Art Meets State of Design • Clearly, natural language dialogs are the present and future of VUI design. • Caller input increases in diversity, when asked open ended questions. • Effective natural language interfaces must be supported by comprehensive grammars. • Creating clever and comprehensive grammars normally takes extensive research into how callers interact with that particular state.

3. Focus on “Can” vs. “Should” • Many consulting practices almost insist that in order to support a natural language dialog state, a statistical language model must be employed, without considering other options. • In many cases an SLM is not necessary to support natural language, even in larger scale IVR deployments

4. Why NotUse an SLM? • High front end cost • Infrastructure barriers Processor Load Port Classing • Difficult to transfer • Often requires specific “one trick” tools • Often difficult to tune and update • Severe amount of training data • Rarely practical in multiple dialog states • Seldom deployable as a Just in Time grammar • Many ASR hosts will not host

5. Why Use an SLM? • Extremely powerful • State of the art • Increased number of targets decreases the amount of customer misroutes and CSR transfers • Able to support a vast amount of targets and reliably delineate between highly confusable utterance groups

6. Natural Language Application: SLM Sample Call Interaction • Traditionally an SLM is the front door into the IVR. • In the diagram below we can see the menu offers unique treatment to seven different “claim” centered requests. “have you paid my claim” “claims” “I need to speak with a claims agent” “how do I file a claim” “I need to file a claim” “I need to cancel a claim” “claim status please”

7. Natural Language Application: Finite Sample Call Interaction • In the diagram below we can see the menu offers unique treatment to seven different “claim” centered requests. “have you paid my claim” “claims” “I need to speak with a claims agent” “how do I file a claim” “I need to file a claim” “I need to cancel a claim” “claim status please”

8. Advanced Finite State Grammar Techniques • Usability findings can build better bootstrap grammars • Using transcriptions to build grammar content • Using transcriptions to weight slots and utterances Example: ;----------------------------------------------------------------- ; 01/17/07 dclaiborn Order Updates Version 4.7 ;----------------------------------------------------------------- .Billingdisambig ( ?(Filler )~0.0918005147 [ Order_Status~0.2687530884 {<choice order_status>} Billing_General~0.1272143539 {<choice billing_general>} Generalsupport~0.5079047434 {<choice general_support>} ] ?( Postfiller )~0.0878072589 ) Order_Status [ ( order )~0.0981125936 ( ?( LIB_I_WANT )~0.0834713481 ?( [ help~0.0628999155 status~0.8687673548 sent~0.0683327297 ] )~0.8453130176 ?( [ or~0.0957498779 and~0.9042501221 ] )~0.64822789 ?( receiving )~0.883405101 [ taking~0.1099701888 update~0.8900298112 ] )~0.901 ]

9. Advanced Finite State Grammar Technique Options • By Hand… 1. Inexpensive 2. Time Consuming; writing code by hand and creating custom tools and utilities 3. Inaccurate • Using a Tool… 1. Higher upfront cost 2. Very fast 3. Automatically generate grammars from transcription file 4. Comparatively very accurate 5. Context free rule generation

10. Comparison • SLM Main Menu for Telco *101 targets, training set of 56,000 utterances, highly confusable training data CA-in: 87.1 FA-in: 2.71 FR-in: 10.18 CR-out: 61.75 FA-out: 38.25 • GSL Main Menu for Same Telco *42 slots, around 10,000 possible utterances, distinct utterances CA-in: 85.22 FA-in: 6.77 FR-in: 8.01 CR-out: 64.29 FA-out: 35.71

11. Is an SLM a Match for My Company’s Natural Language Initiative? • How many targets does my application support? • What does my data say? How many different core requests are callers making? Are those requests confusable with other requests? • What applications are in the IVR now and are there plans for additions? • What is the IVR’s baseline performance today? • What are the performance increases expected from this initiative? • How many callers enter the IVR in a given year, what are the high and low months and are there certain months or times of each month where certain requests increase? • How rapidly will this system need to be taking calls? • What are your goals; increased CSAT and Call Completions, decreased agent to agent transfers?

12. Recommendation from 10,000 Feet • Usability study with at least 20 callers, rating their interaction with Legacy Dialog States and Proposed NL State *NL Main Menu can be WOZ’d, so you don’t have to code before know the quantifiable merit. • Analyze results and produce specific ROI for transition • Create Design Documentation • Code Beta grammar and make application changes to allow for increased grammar slots and sidedoors. • Create a Beta Environment and direct a small percentage of calls to this application. • Tune NL grammar using randomized transcriptions • Roll tuned grammar and analyze results. *Soak period 2-4 weeks • If all goes well, deploy combined application

13. Questions?