1 / 13

Building Robust Grammars to Handle Unexpected Responses

Building Robust Grammars to Handle Unexpected Responses. Sunil Issar Director Convergys Corporation. Convergys at a Glance. Enabling organizations to enhance the value of their relationships with customers and employees. Solutions and Service Offerings.

livana
Download Presentation

Building Robust Grammars to Handle Unexpected Responses

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Building Robust Grammars to Handle Unexpected Responses Sunil IssarDirectorConvergys Corporation

  2. Convergys at a Glance Enabling organizations to enhance the value of their relationships with customers and employees Solutions and Service Offerings • Customer Care, Human Resources, Billing Services • Solutions, Software, Outsourcing, Consulting Worldwide capabilities • Industry Strengths Include: Financial, Communication, Consumer Package Goods, Technology, Health Care, Retail, Pharmaceutical • 73 contact, service and data centers worldwide • Over 575 clients in 70+ countries Leading public company • Listed on NYSE, S&P 500, Fortune 1000 • A Fortune Most Admired Company for seven consecutive years • $2.8 billion in revenues

  3. Agenda • Designing Speech Grammars for High Recognition Accuracy • Missing Science • Achieving High Recognition Accuracy with Robust Speech Grammars • Summary

  4. Designing Grammars • Grammars define rules describing callers responses • Airport Grammar • <city> • Boston • <city> <state> • Boston Massachusetts • <airport name> • Logan International Airport • <city> <airport> • Boston Logan • <airport code> • B O S • Define rules for expected responses • Use menu options listed in VUI • Use synonyms, prefixes and suffixes • Use external data sources to ensure completeness • Postal database for city/state

  5. Measuring Grammar Performance • Use Speech Recognition Accuracy to measure grammar performance • Correct if recognition result matches callers response • Table below shows recognition results for a digits grammar • Accepts 2-16 digits • Expects single digits • In grammar accuracy measures performance on responses listed in grammar • Perceived speech recognition performance is based on total accuracy

  6. Identifying Speech Recognition Issues • High out of grammar rate leads to low overall accuracy • Noise (Peru 800) • Hello (help 900) • Conversational: Give me a agent please (city 600) • Background talk (international from city 600) • Correction: March 3rd 19 or 2007 (March 19th 2007 750) • Restart: December December 14th 2006 (Sat Dec 14th 2006, 700) • Large grammars with uniform probabilities lead to recognition errors • Causes substitutions • Noise is easily misrecognized as “help” • Causes insertions • Recognizer incorrectly inserts short words, digits or letters • Easy to insert “O” for zero or “a” for eight

  7. Missing Science • Caller does not know the grammar rules • Follows conversational behaviors and uses unexpected events • Restarts, Corrections, Side conversations • Around 20% of the responses include unexpected events • It is difficult even for people to understand unfamiliar words

  8. Building Robust Grammars • Focus on improving recognition accuracy metrics • Conduct recognition experiments • Classify recognition errors • Out of Grammar • Substitution • Insertion • Deletion • Confidence scores

  9. Use Appropriate Grammars Based on Caller Responses • Review caller responses • Finite State grammars are appropriate for most speech applications • Short well-formed responses • Use Finite State Grammars (FSG) • Use optional prefixes • I want to find an ATM in Reston, Virginia. • I want <um> I am looking for an ATM in Reston, Virginia. • Use optional suffixes • Reston please • Spontaneous Speech, Large vocabulary • Use Statistical Language Models (SLM) • SLM models n-grams • Example bigram (2-gram): Pr (Virginia | Reston) • SLM are automatically trained from callers responses • Need to assign semantic interpretation before training

  10. Use Weights to Improve Accuracy for Frequently Used Phrases • Table below shows accuracy improvements based on limiting digit grammar to frequent digit lengths • Need to look at the data source or actual responses to identify frequent phrases • Yes, No are the most frequent responses for a Yes/No question • Adding synonyms to reduce out of grammar rate causes substitution errors • Talk by Vaibhav Goel from IBM at SpeechTek West 2007 describes accuracy improvements using weights

  11. Reduce Out of Grammar (OOG) Rate • Lower Out of Grammar rate will improve overall accuracy • May not reduce number of turns • Need to reprompt even if we recognize side conversations • Will improve caller perception of speech application • Classify Out of Grammar Responses • Noise • Background talk • Data • Format • Restart • Correction • Multiple slots • Model most common OOG responses • Garbage model for noise and background talk • Include rules for missing data elements • Ignore less obvious formats • Date: the second two thousand six

  12. Use Completion Techniques to Reduce Out of Grammar Rate • Completion techniques better model callers responses • Responses may not be presented as options • Responses follow conversational behaviors • Menus • Frequent responses include “none of these” or synonyms • “None of these” is not an option typically presented to the caller • Grammar should include “none of these”

  13. Summary • Grammars need to model conversational behaviors to improve callers perception of speech applications • In-grammar accuracy is high • Clients and callers perceptions don’t match in-grammar accuracy • Reviewed several techniques for building robust grammars • Use appropriate grammar type • Use weights • Reduce out of grammar rate • Use garbage models

More Related