ITI-CERTH Augmented and Virtual Reality Laboratory (AVRLab)

1. ITI-CERTH Augmented and Virtual Reality Laboratory (AVRLab) The AVRL of the Informatics and Telematics Institute exhibits substantial research activity, both basic and industry-oriented, as well as technology transfer actions, in the area of virtual, augmented and mixed reality.This area includes modeling and simulation of complex virtual environments, haptic force feedback applied research, 3D modeling and compression, deformable objects and cloth modeling, collaborative virtual environments, new sensor development for head/hands/body tracking, multimedia analysis/synthesis, gesture recognition, head motion modelling and tracking, advanced applications of virtual/augmented/mixed reality in education and training, e-health, assistive technologies, culture, industry and e-business.

2. The ITI-CERTH AVRLab operates a fully operational and well-equipped Virtual Reality Laboratory thatserves as a reference for augmented and virtual reality applications and demonstrations in Greece. It employs a high quality scientific group of personnel for the design, development and demonstration of advanced augmented and virtual reality applications and services.It participates in research networks with assorted institutes and industrial partners in Greece and Europe and has established a spin-off company, to commercially exploit the research results of the Laboratory. Some significant applied research fields where ITI is activated are: • Haptics • Computer Graphics • 3D Semantics • Mixed and Augmented Reality • Virtual, Augmented and Mixed Reality Applications

3. Haptics Extensive study and development of very efficient approaches for haptic rendering, real time force feedback and real time collision detection for complex virtual environments. Applications include virtual environments for the visually impaired, geometry education in 3D environments, virtual assembly planning, real time interaction with complex cultural objects, etc • Computer Graphics Research and development of efficient algorithms for modeling, representation animation and monitoring of deformable and complex virtual objects. It includes new approaches for cloth modeling, collision and self-collision detection, new interaction paradigms, real time avatar simulations, smart objects and complex behavioral modeling

4. 3D Semantics Research and development of advanced algorithms for search and retrieval of 3D content from large databases. Research includes new modeling approaches, descriptor extraction using rotation/translation/scaling invariant features and identification of appropriate matching and data fusion strategies for optimal descriptor selection • Mixed and Augmented Reality Extensive study and development of new approaches in real time head/hands/body tracking for mixed reality applications. Research directions include real time 3D acquisition, stereo and multiview image sequence processing, feature detection and tracking, coding and compression, advanced prediction and detection approaches, etc

5. Virtual, Augmented and Mixed Reality Applications Application oriented research in areas such as education and training, e-health, assistive technologies, culture, industry, e-business, phobia treatment, etc. Includes various approaches utilizing and integrating appropriate virtual/augmented/mixed reality strategies for enhanced application realism

7. The goals of the SIMILAR European network • Integrating a task force on multimodal interfaces that respond efficiently to speech, gestures, vision, haptics and direct brain connections • Merging 32 excellent European laboratories in Human-Computer Interaction (HCI) and in Signal Processing in a single research network. • Address challenges in medical, edutainment and disabled areas

8. The SIMILAR partners SIMILAR institutions are spread all over Europe (and one in Africa) Red : Research Blue : Industry

9. Secondary Objectives • SIMILAR will develop a common theoretical framework for fusion and fission of multimodal information using Signal Processing tools constrained by Human Computer Interaction rules. • SIMILAR will develop a network of usability test facilities. • SIMILAR will manage an International Journal, Special Sessions in conferences, summer schools, interact with industrial partners and promote new research activities. • SIMILAR will address a series of grand challengesin edutainment, disabled people and medical applications and will develop new interfaces for environments where the user is unable to use his hands, like in surgical operation rooms, or in cars.

10. The 7 SIGs (Special Interest Groups ) • Information Fusion and Fission • Multimodal Analysis and Synthesis • Context Aware Adaptation • Usability • Medical Applications • Edutainment Applications • Disabled People Applications

11. car SIMILAR Synopsis Multimodal Signal Proc. Fusion & Fission (IT) Context aware. Platforms(OpenInterface) Usability Medical Edutainment Disabled And Cars

12. What is OpenInterface ? • A general specification for multimodal interfaces geared to solve interconnection problems between hardware and software from heterogeneous origins. • OpenInterface is mainly driven by :FPMs with Multitel (Belgium), ITI/CERTH (Greece), UCL-Tele (Belgium), CLIPs-IMAG(France). • OpenInterface will be accessible on the web :http://www.openinterface.org .

13. Interconnection problems (1) Collaborative research between a large amount of partners leads to interconnection problems…

14. ? Interconnection problems (2) • John is working on a voice recognition algorithm and would like to test it into a real application… • Brenda has a 3D visualisation system that would be interesting to drive with voice… … but programming interfaces are not the same !

15. ? Interconnection problems (3) • Brenda’s 3D visualisation system looks very promising and Wiliam would like to use it to inspect his 3D segmented medical structures… … but programming interfaces are not the same !

16. Interconnection problems (4) • Solution : SIMILAR Platforms (OpenInterface) Creation of a common language, using XML, that will be understood by anyone and used to describe module’s interfaces. SIMILARPlatforms

17. Zoom 1,5 Zoom 1,5 Interconnection problems (5) • Once a common language as been created, everybody will be able to drive partner’s plugin easily and in a way that could be re-used to drive other partner’s module of the same type.

18. OpenInterface Main Goals • Easy development of multimodal applications • Avoid redundances of common tasks • data formats support, usual audio/image/video processing,… • Easy integration between partners’ modules using SIMILARPlatforms specifications • Once a module is finished, he could then be integrated in the OpenInterface library to facilitate his deployment.

19. OpenInterface Architecture

20. OpenInterface : Component based architecture (1) • Each functionality is a component • Components are grouped into plugins (shared libraries) • The content of each plugin is described using the SimilarPlatforms language

21. Video plugins speech Image plugins KERNEL gesture recognition Fusion/fission visualization Conception: Component based architecture (2)

22. OpenInterface Editor • The OpenInterface Editor will • Facilitate the generation of new multimodal applications by simple drag and drop of partner’s modules in a intuitive graphical user interface, • Use the OpenInterface library and the SIMILARPlatforms language to automaticaly add new modules, • Allow rapid creation of protopytes to test new multimodal interfaces.

23. Similar eNTERFACE workshops • Similar organizes annual workshops on multimodal signal processing and interaction • Graduate, master, PhD, postdoctoral students as well as Seniors meet for four weeks and work together in common or complementary research areas and develop novel applications! • eNTERFACE 2005 in Mons, Belgium was a complete success • eNTERFACE 2006 in Dubrovnik was also successful • Hopefully eNTERFACE projects will continue far beyond SIMILAR

24. eNTERFACE 2005 projects • Project 1:Combined Gesture-Speech Analysis and Synthesis • Project 2: Multimodal Caricatural Mirror • Project 3: Biologically-driven musical instrument • Project 4: Multimodal Focus Attention Detection in an Augmented Driver Simulator • Project 5: Multilingual Multimodal Biometric Identification/Verification • Project 6: Speech Conductor • Project 7: A Multimodal (Gesture+Speech) Interface for 3D Model Search and Retrieval Integrated in a Virtual Assembly Application

25. eNTERFACE 2006 projects • Project 1:An Agent Based Multicultural User Interface in a Customer Service Application • Project 2: Multimodal tools and interfaces for the intercommunication between visually impaired and “deaf and mute” people • Project 3: Sign Language Tutoring Tool • Project 4: Multimodal Character Morphing • Project 5: Introducing Network-Awareness for Networked Multimedia and Multi-modal Applications • Project 6: An instrument of sound and visual creation driven by biological signals • Project 7: Emotion Detection in the Loop from Brain Signals and Facial Images • Project 8: Realtime and Accurate Control of Expression in Singing Synthesis • Project 9: Multimodal Driving Simulator

26. ITI-CERTH in eNTERFACE 2005 • ITI-CERTH was the leader of Project 7. • The work has been published in IEEE Multimedia. “Masterpiece: Physical Interaction and 3D content-based search in VR Applications”, vol.13, no.3, pp. 92-100, July, 2006.

27. ITI-CERTH in eNTERFACE 2006 • ITI-CERTH was the leader of Project 2. • Intercommunication between blind and deaf-mute users • Utilizing cross-modal transformation • Tools are integrated in treasure hunting game that serves for both entertainment and education to the new technologies for the disabled • Modalities: Haptics, Speech, Sign language, Gesture

28. THANK YOU! INFORMATICS & TELEMATICS INSTITUTE 1st km. Thermi-Panorama Road PO BOX 361, 57001 THERMI THESSALONIKI, GREECE TEL: +30 2310 464160 FAX: +30 2310 464164 http://www.iti.gr Konstantinos Moustakas Email: moustak@iti.gr