1 / 16

Measuring Human Multimodal Behavior and Developing Multimodal Interfaces

Explore the research on identifying a minimal set of generic cooperation types in human multimodal behavior that can be used in computer input/output interface development. The framework, measuring behavior techniques, and future directions are discussed, offering insights into analyzing, specifying, and annotating multimodal systems. The content covers diverse applications like natural interaction, teaching, speech therapy, and home automation. The study, led by Jean-Claude Martin from LIMSI-CNRS, France, provides valuable guidance for developing effective multimodal interfaces.

tharold
Download Presentation

Measuring Human Multimodal Behavior and Developing Multimodal Interfaces

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Measuring Human Multimodal BehaviorandDeveloping Multimodal Interfaces Jean-Claude MARTIN LIMSI-CNRS, FRANCE LINC, University Paris 8, FRANCE martin@limsi.fr www.limsi.fr/Individu/martin

  2. Is it possible to identify a “minimal set” of “generic” types of cooperation between communication modalities that • are independent of the modalities • are observed in human behavior • should be used in computer I/O behavior 2

  3. CONTENT • Framework • Measuring Behavior • Futures applications 3

  4. 1. FRAMEWORKTYCOON(Martin 93) 4

  5. 1. FRAMEWORKDefinitions 5

  6. 2. MEASURING BEHAVIOR SRI WOZ Toronto Map (Cheyer et al. 98) • How to analyze the multimodal behavior ? • How to specify a multimodal system ? • Find a common grid ? • 10 annotated subjects 6

  7. 2. MEASURING BEHAVIORSRI WOZ Toronto Map (Cheyer et al. 98) 7

  8. 2. MEASURING BEHAVIOR SRI WOZ Toronto Map • [Kehker et al. 98] 8

  9. 2. MEASURING BEHAVIORSRI WOZ Toronto Map (Cheyer et al. 98) • Annotating and managing ambiguity • Annotate cooperation at a finer level • Nb modalities, nb of attributes 9

  10. 2. MEASURING BEHAVIOR IST-ISLE Project http://isle.nis.sdu.dk/ • 2000-2002 • Natural Interaction and MultiModality • NIS, URome, IMS, DFKI, MPI, ELRA, LIMSI • Survey on Data Resources • 22+5 on gestures + speech • Survey on Coding Schemes • 10+3 on gestures + speech 10

  11. 2. MEASURING BEHAVIOR XML DTD + Java 11

  12. 2. MEASURING BEHAVIORXML DTD + Java For each object o For each reference r If the target of r is o Then sum += salienceOf o in r Ratered-compl = sum / nbReferences 12

  13. 3. FUTURE DIRECTIONSApplication to teaching • How do teachers modify their mm communication behavior depending on student level, time slots … • Towards mm pedagogical agents • BQR Université Paris 8 13

  14. 3. FUTURE DIRECTIONSSpeech Therapist Corpus • Multiple tasks • Goals • Who types ? • Improvement • Multimodal • Mediated • Collective gesture 14

  15. 3. FUTURE DIRECTIONSApplication to home automation • Goals • Usability of mm for in-home application (interactive TV) • Applying / Adapting the TYCOON methodology • RNRT Project • 2 years • Partners : • MediaLive (simulation room of in home environment) • Laboratoire Conception de Produits Nouveaux et Innovation(Ecole Nationale Supérieure d’Arts et Métiers) 15

  16. Measuring Human Multimodal BehaviorandDeveloping Multimodal Interfaces Jean-Claude MARTIN LIMSI-CNRS, FRANCE LINC, University Paris 8, FRANCE martin@limsi.fr www.limsi.fr/Individu/martin

More Related