Deploying Directory Assistance Automation Solutions February 2007

1. Deploying Directory Assistance Automation Solutions February 2007 Krishnan Srinivasan, Senior Manager, Professional Services, Menlo Park, CA

2. Motivation for Automation • Service draw large volumes (hundreds of millions of calls / year) • Automation helps to • Save costs associated with operator work time and positions • Improve operator productivity • Provide a consistent user interface and encourages callers to be more specific about their requests • Challenges posed by Automation • Technical (high accuracy with low false accept rate) • User acceptance (getting used to automated service) • ROI to the provider (get reasonably quick return on investment) • Operator acceptance (provide seamless service along with automation)

3. Full Automation Partial Automation Types of Automation – Role of Speech Recognition Voice Store and Forward

4. DA Automation – High Level Call Flow Locality Listing Type Res Biz/Gov Listing Name Last Name First Name Name/Address Disambig Caption/Location/Mixed Disambig Multi-Tier Confirm Number/Information Release

5. All Calls Factors that Influence Automation Rates System Losses Service Scope City, Listing Losses Caption NavigationLosses VUI Gain Automated Calls

6. Automation Requests – FRL Strategy Automate top N listings (typically 2-10 K) • Advantages • Smaller grammar resulting in good in-grammar recognition accuracy • Simple dialog that blends with partial automation • Disadvantages • FRL churns (weekly, monthly, seasonal) • High out of domain rate from non-FRLs resulting in higher false accepts • High maintenance cost to keep FRLs updated • Limited coverage • Limits on how much automation can be achieved

7. Automating Requests – Full Coverage Strategy All listings are potentially automatable • Advantages • Good coverage • Lower churn than FRL • Automated builds • Increased potential for automation • Challenges • Bigger footprint • Need good recognition/search technology • Sophisticated dialog to navigate through complex listing captions • Need for sophisticated confirmation and error recovery

8. What makes for a good DA Application?

9. Design Driven by Caller Expectations • Maintain User Confidence • Confident that they can complete the task and get accurate information • User Satisfaction is directly correlated with the user confidence • Provide the most accurate number/information • Don’t just release *any* number, but rather the *right* number • Minimize the number of repeat calls due to releasing the incorrect number • Maximize efficiency • Only ask questions that are absolutely necessary to resolve to final listing • Decide to “partially automate”, when likelihood of a successful interaction is low • Truly efficient DA interactions are the ones that are completely automated

10. Key Elements of Directory Automation • Listing Normalization • Directory databases are designed for printed form and for lookup by human operators. The listings are usually hierarchically indented for presentation on print. • Need to adapt/normalize listings for automated search • Listing Recognition and Search • Recognize and understand callers “naturally spoken” query • Search for the requested listing in the database • Listing Disambiguation • Progressively narrow down the search till a single “releasable” listing is found • Update Process • Database changes everyday

11. Listing Normalization: Examples

12. Listing Normalization: Processing Steps DB Clean & Normalization MetaDB Distortion Analysis Process for TTS TTS Prompts Grammars

13. Listing Recognition and Understanding (from Nuance DA) What it means What was said? “Um, is there a Burger King in Atherton?” (650)564-2348 “I want Burger King” Identify Salient Segments that contribute to meaning “Burger King Restaurant” “Uman Enterprises. U M A N Enterprises.” (321)432-6232 “Safeway in Crescent Park.” Identify noise words (602)932-8427 “Safeway near the corner of Alice” SLM, SSM

14. Statistical Semantic Models: unsupervised learning Learn Statistical unsupervised learning, based on actual customer calls PLUS Listing Search & Routing (from Nuance DA)SSM combined with Unsupervised Learning Finite State: Grammar Rules (I'm looking for) (I'd just like) (I'd like) (I'd really like) (I'll have) (I'm after) (I'm searching in) (sears) (sears roebuck) (sears retail stores) (sports department at sears) (movies at the regal theater) . . . Thousands of grammar rules manually created and maintained

15. Listing Recognition Data Driven Intelligent Disambiguation Listing Disambiguation (from Nuance DA) What Name ? “Residential listing in San Jose” State Disambig Error Retry Confirm “Safeway in San Jose” “Safeway” “San Jose” “UnitedAirlines in San Francisco” Location? (City) “San Jose” “San Jose” Res Listing ? Biz/Gov Listing ? “Safeway” “Safeway” “John Smith” Caption-1 ? “Bakery” “Pharmacy” Location ? (Street) “McArthur drive” Caption-2 ? “Prescription refill”

16. Formatter Normalization Trainer Compiler Update Processor (from Nuance DA)Components Raw (Listing) Data • Formatter • Raw data reformatted to follow standard structures • Normalization • Spelling correction, acronym/abbreviation expansion, context-sensitive modification Statistically learned rules (Nuance Blue origination) • Manual rules (from DA deployments) • Applicable to any domain – DA, EDA • Trainer • SLM training from normalized data - for recognition of wide-ranging input • SSM training from normalized listing data and field data – for semantic routing • Generic capability usable by Call Steering, EDA • Compiler • Simply generation of the grammars in a fast search format Update Processor Searchable (Listing) Data SLM SSM Rule Pack

17. Architecture (from Nuance DA) Component V-Builder Core Technology DA Technology EDA app Nuance DA Vxml (Reference Implementation) DB Interface Third-Party OSD Disambig Engine Searchable Listing Data Update Processor NVP reference Vxml Platform DB Feed SWMS SSM ASR, TTS DA, EDA reports Service Control Alarms Logging Operator Action Server Operator Workstation OAM-Mgt (Mgt Stn) OSI Admin Data Log Data OSW

18. Performance tuning during pilot deployment Performance Tuning • Dialog Tuning • Grammar coverage (SLM) • Search Optimization (SSM) • Automated Pronunciation Tuning • Model and Language Adaptation • Parameter Tuning • Grammar weights • Recognition parameters • Text to Speech Tuning • Pronunciation • Stress and intonation

19. Usability • Iterative Usability • Based on simulated dialog (Wizard of Oz simulation) • Application not required • Refines design • Evaluative Usability • Based on pilot system • Select users from representative population • Call Monitoring • Interviews • Post Deployment Health Checks • Based on deployed system • Call Monitoring • Interviews

20. Implement & Rollout (from Nuance DA) Release 2:Dev/Int/Tst Captions Navigation Location Disambig TTS Confirmation Number Release Release 1:Dev/Int/Tst Locality Auto Straight Line Listings VSF + Screen Pop Requirements Business User Application System Data Normalization Implement Tagging with Operator Actions Platform Activation Release 2 - Pilot (Live Caller Launch) Tune Locality Listing Caption Location Release 1 – Pilot (Live Caller Launch) Tune Locality Listing • Data Collection • Live Traffic • Transcription Data Collection App Basic Call Flow Biz/Gov/Res Locality Biz/Gov Listing Last/First Name

21. Thank You! Q & A