Holistic Usability Measure VUI Design: Philosophy and Practice

1. Holistic Usability MeasureVUI Design: Philosophy and Practice

2. VUI Design: Science vs. Art Our session was described as “... dives deep into some of the science-versus-art issues in dialog design.”

3. Art & Science in VUI Design • VUI Design is both an Art and Science

4. Art & Science in VUI Design • Art in VUI Design • Creative uses of text/speech • More than 1 way to say something • Creative uses of audio • “Color” in the interface • Voice talent • Mood of the interface

5. Art & Science in VUI Design • Science in VUI Design • Usability • Measurable metrics • Human factors • Short term memory • Software engineering • Reuse • Development

6. Art & Science in VUI Design • What really matters?

7. Usability! • “If you could build a system that resulted in world peace, but no one could use it ... it would be useless. Usability matters.” – Gilbert • However, “there has been no generally agreed-upon method of measuring usability, which makes it difficult to compare among different findings.” - Polkosky

8. Holistic Usability Measure

9. VUI Goals • What is the goal of your VUI? • Who decides the primary goal of the VUI? • Designer • Client • Manager

10. VUI Goals • Did you build the right VUI? • Who decides? • How do you know if your VUI is better than another that does the same tasks? • You must test it!

11. Background • During requirement analysis, before design • Information design; what is to be built • What is the VUI suppose to accomplish? • Who are the users? • Etc. • These are typical, but most people miss • Usability metrics • These are typically an afterthought • As a result, usability gets cut

12. Background • Quantitative Metrics • Speech recognition accuracy • Task completion time • Subjective Metrics • User friendliness • Ease of use

13. Background • Which metrics are most important? • It depends on your VUI • Decide how you will test your VUI before you design it • Given several different metrics • How do you measure the overall effectiveness of the VUI?

14. Holistic Usability Measure • HUM • Assign each metric a weight (1 – 99%) based on its importance • Design and implement the VUI • Evaluate the VUI • Convert all metrics to a similar numerical scale

15. Holistic Usability Measure • HUM = W1*Metric1 + W2*Metric2 + ... + Wn*Metric n • where W1 + W2 + ... + Wn = 1 • and 0 ≤ W1, W2, ..., Wn ≤ 1 • HUM yields a percentage measure rating the VUI’s usability.

16. Conclusions • VUI Design is both an Art & Science • However, what really matters is usability • The goodness of your VUI can only be determined through usability testing • Otherwise, you are guessing • Testing is essential!

17. Conclusions • HUM is a tool for testing • It reports usability with respect to the VUI goals • Multiple ways to design same VUI. Which one is the best? • HUM is ideal for comparing more than one implementation of the same VUI • HUM is easy to use and it makes since • Recall, “there has been no generally agreed-upon method of measuring usability, which makes it difficult to compare among different findings.” – Polkosky (SpeechTECH Magazine November/December 2005)

18. Thank You http://www.HumanCenteredComputing.org