690 likes | 850 Views
Multimodal Interaction in Smart Environments A Model-based Runtime System for Ubiquitous User Interfaces. Dipl.-Inf. Marco Blumendorf Fakultät IV – Elektrotechnik und Informatik Technische Universität Berlin. Agenda. Motivation Current Trends Fundamentals User Interface Models Approach
E N D
Multimodal Interaction in Smart EnvironmentsA Model-based Runtime System for Ubiquitous User Interfaces Dipl.-Inf. Marco BlumendorfFakultät IV – Elektrotechnik und InformatikTechnische Universität Berlin
Agenda • Motivation • Current Trends • Fundamentals • User Interface Models • Approach • Runtime Concepts • UI Description Language Extensions • Architecture • Implementation • Does it work? • Conclusion • Some issues remain A Model-based Runtime System for Ubiquitous User Interfaces
“Recent” Trends World Wide Web & Ubiquitous Computing A Model-based Runtime System for Ubiquitous User Interfaces
World Wide Web A Model-based Runtime System for Ubiquitous User Interfaces
World Wide Web - Basics • Websites hosted on servers • Accessible from everywhere • HTML language • Browser as HTML interpreters A Model-based Runtime System for Ubiquitous User Interfaces
Web Page Development in 4 Steps • Write HTML code • Add some CSS (style) • Add some JavaScript • Deploy it on a webserver Model, interpreted by the browser A Model-based Runtime System for Ubiquitous User Interfaces
Ubiquitous Computing A Model-based Runtime System for Ubiquitous User Interfaces
Computers are everywhere! A Model-based Runtime System for Ubiquitous User Interfaces
“Ubiquitous Computing” (Mark Weiser, 1991) “In the 21st century the technology revolution will move into the everyday, the small and the invisible.” A Model-based Runtime System for Ubiquitous User Interfaces
Smart Objects Smart Devices Smart Networks Smart Environments A Model-based Runtime System for Ubiquitous User Interfaces
Interaction happens everywhere! A Model-based Runtime System for Ubiquitous User Interfaces
Ubiquitous User Interfaces (UUIs) A Model-based Runtime System for Ubiquitous User Interfaces
Utilizing Ubiquitous User Interfaces in Smart Environments used interaction modalities A Model-based Runtime System for Ubiquitous User Interfaces
Compared to a web page … How would you develop a Ubiquitous User Interface? A Model-based Runtime System for Ubiquitous User Interfaces
Language and Infrastructure for Ubiquitous User Interfaces A Model-based Runtime System for Ubiquitous User Interfaces
Agenda • Motivation • We need a language and infrastructure for UUIs • Fundamentals • User Interface Models • Approach • Runtime Concepts • UI Description Language Extensions • Architecture • Implementation • Does it work? • Conclusion • Some issues remain. A Model-based Runtime System for Ubiquitous User Interfaces
User Interface Models A Model-based Runtime System for Ubiquitous User Interfaces
HTML as a Model User Interface HTML HTML A Model-based Runtime System for Ubiquitous User Interfaces
Models to Maximize Common Code User Interface Model A Model-based Runtime System for Ubiquitous User Interfaces
UIDLs use Multiple Levels of Abstraction Concepts and Task Model Abstract Interface Concrete Interface Final User Interface See also Szekely 1996, Cameleon Reference Framework (Calvary et al. 2003) Similar concepts can be found e.g. in UsiXML (Limbourg et al. 2004) and TERESA (Berti et al. 2004) A Model-based Runtime System for Ubiquitous User Interfaces
UIDLs use Multiple Levels of Abstraction Concepts and Task Model Abstract Interface Concrete Interface Final User Interface See also Szekely 1996, Cameleon Reference Framework (Calvary et al. 2003) Similar concepts can be found e.g. in UsiXML (Limbourg et al. 2004) and TERESA (Berti et al. 2004) A Model-based Runtime System for Ubiquitous User Interfaces
What if you want to change anything at runtime? A Model-based Runtime System for Ubiquitous User Interfaces
You need the models! A Model-based Runtime System for Ubiquitous User Interfaces
Problems • Current approaches have a design time focus • At runtime, knowledge is hidden within code • Design intentions are not explicit anymore • Design decisions can not be revised • This makes UIs static and limits adaptivity! => No Ubiquitous User Interfaces A Model-based Runtime System for Ubiquitous User Interfaces
Keep the UI models at runtime! A Model-based Runtime System for Ubiquitous User Interfaces
Model Interpretation A Model-based Runtime System for Ubiquitous User Interfaces
Model Interpretation A Model-based Runtime System for Ubiquitous User Interfaces
Model Interpretation A Model-based Runtime System for Ubiquitous User Interfaces
Model Interpretation A Model-based Runtime System for Ubiquitous User Interfaces
Agenda • Motivation • We need a language and infrastructure for UUIs • Fundamentals • We need models interpretation at runtime • Approach • Runtime Concepts • UI Description Language Extensions • Architecture • Implementation • Does it work? • Conclusion • Some issues remain. A Model-based Runtime System for Ubiquitous User Interfaces
Framework Building Blocks • Executable Models • UI Description Language • Server Architecture A Model-based Runtime System for Ubiquitous User Interfaces
How to make Models Executable? • Preserve design information at runtime • Add state information • Add execution logic A Model-based Runtime System for Ubiquitous User Interfaces
Extending Current Design Models Dynamic Executable Model See e.g. Breton & Bézivin 2001 for details A Model-based Runtime System for Ubiquitous User Interfaces
A Task Tree Example Abstract Task Interaction Task >> [] Temp. Operators A Model-based Runtime System for Ubiquitous User Interfaces
Metamodel of Executable Task Models A Model-based Runtime System for Ubiquitous User Interfaces
Metamodel of Executable Task Models A Model-based Runtime System for Ubiquitous User Interfaces
Metamodel of Executable Task Models A Model-based Runtime System for Ubiquitous User Interfaces
Metamodel of Executable Task Models A Model-based Runtime System for Ubiquitous User Interfaces
Framework Building Blocks • Executable Models • UI Description Language • Server Architecture A Model-based Runtime System for Ubiquitous User Interfaces
How to build a language for such models? • Extend existing models with runtime concepts • Add state & execution logic • Synchronization of models through mappings • Interaction with external entities • Additional concepts for Ubiquitous User Interfaces • Multimodal and distributed interaction • Context-sensitivity • Adaptation A Model-based Runtime System for Ubiquitous User Interfaces
Mediating between human and computer 10011010101111100100 “turn the light on” A Model-based Runtime System for Ubiquitous User Interfaces
Mediating between human and computer ? 10011010101111100100 “turn the light on” A Model-based Runtime System for Ubiquitous User Interfaces
Mediating between human and computer ? 10011010101111100100 “turn the light on” A Model-based Runtime System for Ubiquitous User Interfaces
Mediating between human and computer 10011010101111100100 “turn the light on” A Model-based Runtime System for Ubiquitous User Interfaces
Net of UI Models A Model-based Runtime System for Ubiquitous User Interfaces
Mediation Process A Model-based Runtime System for Ubiquitous User Interfaces
Adding State and Execution Logic A Model-based Runtime System for Ubiquitous User Interfaces
Synchronizing Models at Runtime Task Model Service Model Mapping Link Target Source A Model-based Runtime System for Ubiquitous User Interfaces
A UUI is defined by a set of linked & stateful executable models. Ubiquitous User Interface A Model-based Runtime System for Ubiquitous User Interfaces
Framework Building Blocks • Executable Models • UI Description Language • Server Architecture A Model-based Runtime System for Ubiquitous User Interfaces