1 / 48

Multimodality and context-aware adaptation

Multimodality and context-aware adaptation. Quentin Limbourg, Jean Vanderdonckt. Context-awareness. What is in the context of use? User Platform Environment Organisation Socio-psychological factors Two examples of adaptation due to context-awareness Adaptability = adaptation by the user

chandler-ewing
Download Presentation

Multimodality and context-aware adaptation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Multimodality and context-aware adaptation Quentin Limbourg, Jean Vanderdonckt WCC'2004 - Topical day on Multimodal interaction

  2. Context-awareness • What is in the context of use? • User • Platform • Environment • Organisation • Socio-psychological factors • Two examples of adaptation due to context-awareness • Adaptability = adaptation by the user • Adaptivity = adaptation by the system WCC'2004 - Topical day on Multimodal interaction

  3. Example of adaptability • Personalisation • Requires many editing • Is hard to achieve • Does not necessarily factor out common parts • Does not necessarily separate fixed parts from variable parts • Goal • To have a system for automated generation of UIs • With support of personalisation • That can generate multiple variants of the same UI • For this purpose, we use • A feature model • Generative Programming WCC'2004 - Topical day on Multimodal interaction

  4. What‘s a Feature Model? Exclusive Optional C C F1 F1 F2 Alternate Dependencies 1. C 2. F1 F2 WCC'2004 - Topical day on Multimodal interaction

  5. Example of a Feature Model WCC'2004 - Topical day on Multimodal interaction

  6. Feature model Model Specifications C C 0 1 1 F1 F2 F3 F1 F2 F3 1 0 0 1 F2a F2b F3a F3b F2a F2b F3a F3b <C> <F1>0</F1> <F2>1 <F2a>1</F2a> <F2b>0</F2b> </F2> <F3>1 <F3a>0</F3a> <F3b>1</F3b> </F3> </C> USIXML Specifications WCC'2004 - Topical day on Multimodal interaction

  7. Final code Components void areaOperation(const int&, const int&, const int&, const int&, ImgProcessT<T>& (ImgProcessT<T>::*pimgpr)(const int&, Pl*), const int& nAngle, Pl* ppl ); void areaOperation(const int&, const int&, const int&, const int&, ImgProcessT<T>& (ImgProcessT<T>::*pimgpr)(Pl*), Pl* ppl ); void areaOperation(const int&, const int&, const int&, const int&, ImgProcessT<T>& (ImgProcessT<T>::*pimgpr)(const int&, const int&, const int&, const int& ), const int&, const int&, const int&, const int& ); void channelShowHistogram( int*, const int&, const int&, const int&, const int&, const int&, const int&, const int& n = 0 ) const; void channelFft ( T*, T*, T*, const unsigned int&, const unsigned int&, Pl* ppl = NULL ); void channelIfft ( T*, T*, T*, const unsigned int&, const unsigned int&, Pl* ppl = NULL ); void channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL); void channelRange (T*, Pl* ppl = NULL ); void nlinFilterCommon (const int& nID, Mask*, Pl*); int pot (const unsigned int& ) const; ImgProcessT<T>& channelXor (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL); public: ImgProcessT(const char* const psz = "", Pl* ppl = NULL); ImgProcessT(const int&, const int& nWidth = 20, const int& nHeight = 20, const int& nFillColor = 0, ProgressLine* ppl = NULL ); ImgProcessT(const ImgProcessT<T>&); virtual ~ImgProcessT(); virtual char* getClassName() const; virtual char getClassID () const; ImgProcessT<T>& operator+=(const ImgProcessT<T>&); I ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL); public: ImgProcessT(const char* const psz = "", Pl* ppl = NULL); ImgProcessT(const int&, const int& nWidth = 20, const int& nHeight = 20, const int& nFillColor = 0, ProgressLine* ppl = NULL ); void channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL); void channelRange (T*, Pl* ppl = NULL ); void nlinFilterCommon (const int& nID, Mask*, Pl*); int pot (const unsigned int& ) const; ImgProcessT<T>& channelXor (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL); ImgProcessT<T>& chann elMix ImgProcessT<T> & channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL) ImgProcessT<T>& channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL) void channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL); void channelRange (T*, Pl* ppl = NULL ); USIXML-Specification V oid channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL); void channelRange (T*, Pl* ppl = NULL ); void nlinFilterCommon (const int& nID, Mask*, Pl*); int pot (const unsigned int& ) const; ImgProcessT<T>& channelXor (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL); public: ImgProcessT(const char* const psz = "", Pl* ppl = NULL); ImgProcessT(const int&, const int& nWidth = 20, const int& nHeight = 20, const int& nFillColor = 0, ProgressLine* ppl = NULL ); ImgProcessT(const ImgProcessT<T>&); virtual ~ImgProcessT(); virtual char* getClassName() const; virtual char getClassID () const; ImgProcessT<T>& operator+=(const ImgProcessT<T>&); ImgProcessT<T>& operator-=(const ImgProcessT<T>&); ImgProcessT<T>& operator^=(const ImgProcessT<T>&); ImgProcessT<T> operator+ (const ImgProcessT<T>&); ImgProcessT<T> operator- (const ImgProcessT<T>&); ImgProcessT<T> operator^ (const ImgProcessT<T>&); ImgProcessT<T>& channelNLinFilter(const int&, Mask*, T*, Pl* ppl=NULL); mgProcessT<T>& operator+=(const ImgProcessT<T>&); ImgProcessT<T>& operator-=(const ImgProcessT<T>&); ImgProcessT<T>& operator^=(const ImgProcessT<T>&); ImgProcessT<T> operator+ (const ImgProcessT<T>&); ImgProcessT<T> operator- (const ImgProcessT<T>&); ImgProcessT<T> operator^ (const ImgProcessT<T>&); ImgProcessT<T>& channelNLinFilter(const int&, Mask*, T*, Pl* ppl=NULL); void channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL); void channelRange (T*, Pl* ppl = NULL ); void nlinFilterCommon (const int& nID, Mask*, Pl*); int pot (const unsigned int& ) const; ImgProcessT<T>& channelXor (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL); void channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL); void channelRange (T*, Pl* ppl = NULL ); void channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL); void channelRange (T*, Pl* ppl = NULL ); void channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL); void channelRange (T*, Pl* ppl = NULL ); Generative Programming WCC'2004 - Topical day on Multimodal interaction

  8. Example of adaptivity • Computer-aided surgery by mixed reality WCC'2004 - Topical day on Multimodal interaction

  9. Platform 1: Web terminal Task & Concepts Logical User Interface Physical User Interface Final User Interface Multi-platform • Extending approaches for multiple contexts of use • Context is now restrictedto one computing platformat a time WCC'2004 - Topical day on Multimodal interaction

  10. Platform 1: Web terminal Platform 2: Personal computer Platform 3: Pocket PC Final User Interface Final User Interface Final User Interface Multi-platform • Traditional approach: last level only WCC'2004 - Topical day on Multimodal interaction

  11. Platform 1: Web terminal Platform 2: Personal computer Platform 3: Pocket PC HTML/Java UI code Visual C++ UI code Mobile Visual Basic UI Multi-platform • Traditional approach: Atomica WCC'2004 - Topical day on Multimodal interaction

  12. Platform 1: Web terminal Platform 2: PC Platform 3: Pocket PC Java UI #2 Java UI #3 Java UI #1 C++ UI #1 C++ UI #2 C++ UI #3 Mobile UI #1 Mobile UI #3 Mobile UI #2 Multi-platform [UIML,1999] • Shortcomings of the traditional approach (ct’d) • Repetition of development and maintenance Application #1 Application #2 Application #3 WCC'2004 - Topical day on Multimodal interaction

  13. Platform 1: Platform 2: Platform 3: Physical User Interface Physical User Interface Physical User Interface Final User Interface Final User Interface Final User Interface Multi-platform = Translation relationship = Transformation at the same level of abstraction • Physical level deployment WCC'2004 - Topical day on Multimodal interaction

  14. Multi-platform • Physical level deployment: Galaxy Platform 1: Personal Computer Platform 2: Macintosh Mac OS Platform 3: Workstation OSF/Motif Physical User Interface Physical User Interface Physical User Interface Windows UI code Macintosh UI code OSF/Motif UI code WCC'2004 - Topical day on Multimodal interaction

  15. Platform 1: Personal Computer Platform 2: Macintosh Mac OS Platform 3: Workstation OSF/Motif Windows UI code Macintosh UI code OSF/Motif UI code Multi-platform • Physical level deployment: Galaxy WCC'2004 - Topical day on Multimodal interaction

  16. Platform 1: Personal Computer Platform 2: Macintosh Mac OS Platform 2: Workstation OSF/Motif Logical User Interface Logical User Interface Logical User Interface Physical User Interface Physical User Interface Physical User Interface Final User Interface Final User Interface Final User Interface Multi-platform • Logical level deployment WCC'2004 - Topical day on Multimodal interaction

  17. Platform 1: Personal Computer with Web browser Platform 2: Personal Computer With XML browser Platform 3: Mobile Phone UIML Specification HTML Renderer XML Renderer WML Rendered HTML UI code XML UI code WML UI code Multi-platform • Logical level deployment: UIML WCC'2004 - Topical day on Multimodal interaction

  18. <interface> <structure> <part class=“Button”> … </part> ... </structure> </interface> Multi-platform:UIML Skeleton – Part 1 [Abrams et al., 1999] What parts comprise the UI &what’s their relationship? <?xml version="1.0" ... ?> <uiml version="2.0"> </uiml> WCC'2004 - Topical day on Multimodal interaction

  19. Platform 1 Task & Concepts Platform 2 Logical User Interface Platform 3 Physical User Interface Task & Concepts Task & Concepts Final User Interface Logical User Interface Logical User Interface Physical User Interface Physical User Interface Final User Interface Final User Interface Multi-platform • Task & Concepts level deployment WCC'2004 - Topical day on Multimodal interaction

  20. Platform 1 Task & Concepts Platform 2 Logical User Interface Platform 3 Physical User Interface Task & Concepts Task & Concepts Final User Interface Logical User Interface Logical User Interface Physical User Interface Physical User Interface Final User Interface Final User Interface Multi-platform [Thevenin,2001] • Multiple levels of translation/reification WCC'2004 - Topical day on Multimodal interaction

  21. Platform 1 Task & Concepts Logical User Interface Physical User Interface Final User Interface Multi-platform • Translation at Task & Concept level Platform 2 Task & Concepts Logical User Interface Physical User Interface Final User Interface WCC'2004 - Topical day on Multimodal interaction

  22. Multi-platform • Translation at Task & Concept level • Task model primitives • Sub-tasks (nodes) • Remove sub-task: for different reasons • Add sub-task: unlikely to appear • Modify sub-task: switch, reshuffle, relax constraints • Hierarchical & temporal operators (edges) • Modify temporal operator: reordering • Cutting sub-trees • Concepts model primitives WCC'2004 - Topical day on Multimodal interaction

  23. Pay registration >> Register to CHI’2001 Conference Payment mode [ ] Specify all registration needs By credit card By fax Record | | | | | | Enter personal information Specify special needs Specify attendee type Multi-platform • Task translation | | | >> WCC'2004 - Topical day on Multimodal interaction

  24. Multi-platform [Märtin,1990] • Concepts removal: CT-UIMS WCC'2004 - Topical day on Multimodal interaction

  25. Platform 1 Task & Concepts Logical User Interface Physical User Interface Final User Interface Multi-platform • Translations T&C + logical levels Platform 2 Task & Concepts Logical User Interface Physical User Interface Final User Interface WCC'2004 - Topical day on Multimodal interaction

  26. Specify all registration needs | | | | | | Enter personal information Specify special needs Specify attendee type Multi-platform • Translations T&C + logical levels WCC'2004 - Topical day on Multimodal interaction

  27. Specify all registration needs >> | | | Enter personal information Specify special needs Specify attendee type Multi-platform • Translations T&C + logical levels WCC'2004 - Topical day on Multimodal interaction

  28. Multi-platform • Translations T&C +logical levels WCC'2004 - Topical day on Multimodal interaction

  29. Platform 1 Task & Concepts Logical User Interface Physical User Interface Final User Interface Multi-platform • Translations T&C + logical + physical levels Platform 2 Task & Concepts Logical User Interface Physical User Interface Final User Interface WCC'2004 - Topical day on Multimodal interaction

  30. Multi-platform • Translations T&C + logical + physical levels • T&C + logical remain constant over time or coordinated with changes • Graceful degradation of presentation • Widgets: replacement, reduction, resizing • Text: text summarization, sentence elision, automatic summary • Images: color reduction, cropping, resizing,… • Animation: reduction to still images,… • Video: reduction of frame, refreshing,… WCC'2004 - Topical day on Multimodal interaction

  31. Multi-platform • Graceful degradation of presentation • Widgets WCC'2004 - Topical day on Multimodal interaction

  32. Multi-platform • Graceful degradation of presen-tation • Images WCC'2004 - Topical day on Multimodal interaction

  33. Multi-platform • Graceful degradation of presentation • Images WCC'2004 - Topical day on Multimodal interaction

  34. Platform 1 Task & Concepts Logical User Interface Physical User Interface Final User Interface Multi-platform • Translations combined at all levels Platform 2 Task & Concepts Logical User Interface Physical User Interface Final User Interface WCC'2004 - Topical day on Multimodal interaction

  35. Platform 1 Task & Concepts Logical User Interface Physical User Interface Task & Concepts Final User Interface Logical User Interface Physical User Interface Multi-platform • Multiple transformations possible Platform 2 Final User Interface WCC'2004 - Topical day on Multimodal interaction

  36. Platform 1 Task & Concepts Logical User Interface Physical User Interface Final User Interface Multi-platform [Calvary, Coutaz & Thevenin, 2001] • Plastic UI = UI that can accomodate multiple changes of context (here, platforms) while preserving some usability properties Platform 2 Platform 3 Final User Interface Final User Interface WCC'2004 - Topical day on Multimodal interaction

  37. Multi-platform [Grolaux, 2001] • Plastic UI: Virtual keyboard WCC'2004 - Topical day on Multimodal interaction

  38. Multi-platform [Grolaux, Van Roy, Vanderdonckt, 2002] • Plastic UI: FlexClock WCC'2004 - Topical day on Multimodal interaction

  39. Multi-platform • Plastic UI: adaptable bounded value WCC'2004 - Topical day on Multimodal interaction

  40. Multi-platform = Abstraction relationship = Transformation into amore abstract level ofabstraction • Reverse engineering Platform 1 Platform 2 Task & Concepts WebRevenge ReversiXML Logical User Interface Logical User Interface ReversiXML Physical User Interface Physical User Interface Final User Interface Final User Interface WCC'2004 - Topical day on Multimodal interaction

  41. XIML2WML generator WCC'2004 - Topical day on Multimodal interaction

  42. Multi-platform = Crossing relationship = Transformation betweendifferent levels ofabstraction • Reverse engineering [Bouillon & Vanderdonckt,2002] Platform 1 Platform 2 Retargeting Logical User Interface Physical User Interface Final User Interface Final User Interface WCC'2004 - Topical day on Multimodal interaction

  43. Multi-directional User Interfaces • A UI is represented by a suite of models that are • Analysable • Editable • Transformable • Transformational approach • Any UI is subject to any transformation at any time • From one language to another • From one computing platform to another • From reverse engineering to forward engineering • … WCC'2004 - Topical day on Multimodal interaction

  44. Multi-directional User Interfaces • Model engineering • Forward engineering (top-down) • Reverse engineering (bottom-up) • Bidirectional engineering (both ways) • Model transformations • Reification, abstraction • Translation • Crossing (retargeting, revamping,…) WCC'2004 - Topical day on Multimodal interaction

  45. Contexts of use Devices Platforms Modalities Channels Multiple levels of consideration • Devices plus : • Software characteristics • Hardware characteristics • Languages: programming, markup • Tools: renderer, interpreter, compiler • Platforms plus : • Modalities of interaction • Mono vs multimodality • Modalities plus : • Market segments • Time of usage • Place of usage • Network, Wi-fi • Various tasks • Channels plus: • User behaviour • User expectations • User experience • User groups • Physical constraints • Environment WCC'2004 - Topical day on Multimodal interaction

  46. Contexts of use Devices Platforms Modalities Channels USIXML WCC'2004 - Topical day on Multimodal interaction

  47. An overview of USIXML tools • Running prototypes • GrafiXML: CUI Hi-Fi editor + Code Generator (Java Swing, XHTML) • FormiXML: CUI Hi-Fi editor for Forms (XForms, HTML) • SketchiXML: CUI Lo-Fi Sketching • VisiXML: CUI Mid-Fi (MS Visio Plug-in) • FlashiXML: CUI renderer for SVF • ReversiXML: reverse engineering from HTML to CUI, AUI • TransformiXML API: transformation tool • Teresa (F. Paternò, CUI level to multi FUI generation) Demo Demo Demo WCC'2004 - Topical day on Multimodal interaction

  48. www.plasticity.org www.usixml.org www.similar.cc Thank you very much for your attention! Credits WCC'2004 - Topical day on Multimodal interaction

More Related