Usability & Satisfaction Testing Pitfalls

Usability & Satisfaction Testing Pitfalls Rick Rappe’ Vocal Laboratories Inc.

Pitfall #1Not Testing At All -Big Egos The Hallmark of a Good VUI Designer is Humility -Too Much Vendor Trust -Budget & Time Constraints Testing needs to be an integral part of the cost/design process

Pitfall #2 Inadequate Testing Methodologies The Focus Group is a design aid, not a method to vet the new application for ease of use and caller satisfaction. A usability problem affecting 5% of calls requires 100 test calls to reliably hit the problem just once. A 1% problem takes 500. If a problem is hit but once, will it be recognized?

Inadequate Testing Methodologies, cont. • The WOZ (Wizard of Oz) test does not exercise speech recognizer accuracy. • The rapid prototype test is a good design aid, but does not address technical performance (load/stress tests) and will not uncover all grammar/ recognizer issues. • Friends and Family (insider) testing is flawed. You are not like your customers. You understand jargon and how the system is supposed to work. • Testing is an ongoing requirement. Things change.

Pitfall #3 Things Change • A well done (satisfied caller) application will degrade over time. • Products, procedures and marketing “changes” result in Application Drift, and • As consumers adopt self service, their expectations rise.

Pitfall #4 Bad Survey Technique • End-Of-Call surveys systematically exclude anyone who hangs up early. (Participation bias) • Follow up surveys reveal few useful specifics on improvement • Too small a sample, too high a margin of error • Insiders and Focus Groups do not emulate a customer base (Sample bias)

Pitfall #5 Poor Survey Usage • Were you happy, or would you like to be run over by a truck? • Over 1/2 of all surveys are said to be written to justify a currently held belief rather than objectively seek truth. • How satisfied were you? • Biased upward question. Should be: “How satisfied or dissatisfied were you?” • Do you like Cookies? Will get different answers if previous Q is: “Do you like milk?”

The Solution (New Apps.) • Consider a usability/satisfaction exam of existing service as a base line from which to measure acceptance of the new service. • Run “mini” usability tests throughout the development process when changes are easy to accomplish. • Do not rely on a pre-cut Pilot test. Why? • You WILL uncover problems when the fix will cause cutover and budget delays; and a tendency to declare the application “good enough” is strong. • Using actual customers as test call guinea pigs is a risk to customer retention and brand loyalty.

The Solution (Existing Apps.) • An IVR/Speech recognition application is NOT a build it and forget it exercise. • Because “things” change, consider an ongoing survey large enough to detect opinion shifts.

Conducting a Quality Survey • Is poor survey technique, delivering slanted results/interpretation, a cause of mediocre customer service? • Yes, if “false positives” lull into a why change attitude.

Five Survey Rules • 1. Sample must reflect a fair cross section of typical users. • Sample bias, participation bias & insider jargon must be avoided • 2. Sample must be large enough to reliably exercise the application. • The trade off between sample size and accuracy must be understood. Situations differ, but beyond a study size yielding a 4-5% margin of error, more surveys yield decreasing levels of accuracy for the cost/effort

Five Survey Rules cont. • 3. Ask meaningful and objective questions. • Specific queries yield actionable results • 4. Understand the responses. • The human desire to please must be taken into account. “Satisfied” in the mind of the consumer is not a positive/loyalty rating. • 5. Do something with the information. • Feedback suggests that results from as many as 2/3 of satisfaction surveys are never acted upon.

Rick Rappe’ rappe@vocalabs.com952-941-6580 X205

Usability & Satisfaction Testing Pitfalls