Natural Speech Interfaces for Environment and Device Control

1. Natural Speech Interfaces for Ana Ramírez Chang PhD Forum Grace Hopper Environment and Device Control

2. Environment Control Example Devices (Objects) Commands Configurations • kitchen television • kitchen lights • kitchen speakers • dining room lights • dining room speakers • blinds • stereo system cooking dinner • bright lights in kitchen • kitchen TV on • loud speaker volume eating dinner • blinds closed • dimly lit dining room • dinner music • soft speaker volume breakfast • blinds open • brightly lit • news on MOTIVATION

3. What kind of interface? • allow users’ own natural forms of expression • calm interface • almost invisible • except during direct (focal) interaction[Weiser et al, 1996] natural speech interface MOTIVATION

4. Why Speech? • Speech is an ambient medium rather than an attentional one • Speech is descriptive rather than referencial • Speech requires modest physical resources • [Rosenfeld, Olsen, Rudnicky, 2001]

5. More Example Domains • Designing natural speech interfaces for environment and device control • Mobile • i.e. configuring settings on your cell phone, • Home media control • i.e. music selection, program recording • Environmental • i.e. controlling tv, speakers, lights and blinds together for particular tasks MOTIVATION

6. Main Sink Public Machine Screen w o d n i W Public Machines 11'-11 11/16" Test Application

7. Sink Screen 11'-11 11/16" Customized Lighting Control • “more light in the toolshop” • “my cube on” • “kitchen lights on” • “can you turn the lights over the couch off” • “can you turn the lights in my cubicle on”

8. Sink Screen 11'-11 11/16" Training the system • Record command • Type name (in web interface) • Demonstrate lighting scene

9. phrases phrases objects objects 79 dimmable lights ? light 00 2 from window, 3 from wall kitchen area soft space wizard-of-Oz formative data collection Traditional Speech Interface Design wizard-of-Oz formative data collection or

10. Challenge in designing speech interfaces for configuration tasks • Discovering how users give commands and • Discovering the configurations named by those commands That is, • it is not just a problem of building a language model or grammar for user input, • but in discovering the semantics of those inputs SNAC - Simultaneous Naming And Configuration MOTIVATION

11. objects 79 dimmable down-lights configurations phrases kitchen area soft space SNAC formative data collection SNAC Speech Interface Design

12. Data Collection • 16 regular occupants of the lab • 2 weeks • each time a user wanted to change the lighting scene • recorded their command • typed their name • demonstrated lighting scene

13. Algorithms • Factorization algorithms • Initial results using non-negative matrix factorization • 0.90 accuracy • 0.07 loss in accuracy when comparing manual transcripts and automatic speech recognition transcripts • Natural Language Processing algorithms • Compare these results with those from factorization algorithms

14. Evaluation • User study with regular occupants • Each user • train system with 3-4 demonstrations of each personalized command • Record command & demonstrate lighting scene • test system on personalized command • Record command • Give feedback on system’s response in web GUI

15. Conclusion Propose and evaluate an approach to the simultaneous naming and configuration problem in the design of natural language interfaces for environment and device control. • Wizard-of-Oz data collection • Collect system state data • In addition to user utterances • Train a pattern-recognizer on • System state data • User utterances CONCLUSION