A Scientific Approach to Improving VUI Design

2. Peter U. Leppik, CEORick Rappe, VP of Business DevelopmentVocal Laboratories Inc. A Scientific Approach to Improving VUI Design

3. Agenda • Principles of Gourmet Customer Service • Data Collection Toolbox • The Project Life-Cycle • Conclusion/Q&A

4. Principles of Gourmet Customer Service • Nearly every effort to improve service makes at least one of these mistakes: • No data is collected from actual callers. • The data collected does not relate to business goals. • There is not enough data collected to be statistically significant. • There is no context or understanding of the data (for example: what is a “good” satisfactions score?). • No action is taken based on the data collected.

5. The Scientific Approach • Set Goals • Use relevant business goals, not technical metrics • Gather data • Multiple kinds of tests may be needed to get a complete picture • Take action • Validate • Repeat the tests from Step 2. • Commit: Go back to Step 1.

6. Data Collection Toolbox • There is no one way to collect data about a VUI or call center. • A VUI designer who only uses one test method is like a carpenter who only owns a hammer.

8. The Toolbox: General Tools • Test Goals • Statistical Considerations • Margin of Error • Probability of Finding Problems • Prototypes • Functional Prototype (aka “Rapid Prototype”) • Wizard-of-Oz Prototype

9. The Toolbox: Idea Generating Tools • Heuristics • “Best Practices” • Expert Opinions, “Audits,” “Assessments” • Agent Feedback • Call Recording and Monitoring

11. The Toolbox: Data Generating Tools • Controlled Testing • Lab Tests • Rapid Assessments • Large Controlled Tests • Follow-Up Surveys • End-of-Call Surveys • Not Recommended due to extreme sample bias

13. The Toolbox: More Data Generating Tools • Friends & Family & Employee Tests • Not Recommended due to unrepresentative sample • Automated Load Testing • Traversal Testing • Call Logs and Stats

14. The Project Life-Cycle • Every project goes through certain phases. • Analysis • Design • Code • Test • Release • Operational • Every phase has different testing requirements

15. Analysis Phase • Lay out the project scope, goals, and budget • Ask the big questions • Who are these callers, and how do they live? • Why are they using the phone and not (e.g.) the web? • What is the competition doing? • Look at the big picture and big ideas: • Generate ideas from consultants, vendors, agents, and customers • Develop baselines & benchmarks

16. Analysis Phase Tools • Controlled Testing • Benchmarking, competitive analysis, baseline measurements • Need to understand your callers before you start • Agent Feedback • Follow-up Surveys • Call Recordings • Heuristics • “Best practices,” consultants’ opinions, etc.

17. Design Phase • This is when you plan the details of the VUI • Use a Prototype  Test  Prototype  Test iterative design cycle • The keys to success are speed and flexibility • Test several designs, sometimes in the same day • Don’t blow the entire testing budget on The Usability Test

18. Design Phase Tools • Prototypes • WOZ or Functional Prototype • Controlled testing • Rapid Assessment or Lab Test • Lets you use an iterative design process • Nothing ends arguments faster than solid data.

19. Code Phase • Implement the design • Most testing is for debugging • Unit testing and some functional testing • Relatively little VUI testing takes place here • Implementation may require some VUI changes; these should be tested

20. Test Phase • Last stage before rolling out the application to live callers • This is where the most thorough testing takes place

21. Test Phase Tools • Traversal test • Ensures that the implementation matches the design • Large controlled tests • Designed to catch even small usability glitches • Automated load testing • Done last because changes in the first two test phases can affect load handling

23. The Test Phase can also be the Acceptance Phase • Business goals are what are important, not the exact design • They are what should be in the contract. • Tests which measure business goals should be the acceptance criteria. • Many of these tests can be used in the design phase.

24. Release Phase • This is the process of rolling out the application • By this time, you should be confident that the system works • Confident doesn’t mean cocky • This is the time for a phased roll-out • Some call it a “pilot test” • This is a deployment strategy, not a testing strategy

26. Operational Phase • The work doesn’t end when the system is live! • Application Drift • External changes • Company • Caller Lifestyle • Competition • This looks a lot like the analysis phase for the next project….

27. Operational Phase Tools • Benchmarking (Controlled Testing) • Competitive analysis (Controlled Testing) • Follow-up surveys • Agent feedback • Monitoring call stats and logs • Listening to call recordings

28. Lessons Learned • Good service does not have to be more expensive than bad service • Customers are smarter than machines • Every system can stand some improvement • The best VUI designs come from paying attention to the right things at the right time

29. Q & A