Multimodal Affective Technologies a concrete answer from the perspective of a large company

1. Multimodal Affective Technologies a concrete answer from the perspective of a large company Diego Arnone Engineering Ingegneria Informatica icann2006, Athens Sept 14th 2006

2. Multimodal Interfaces vs. Digital Media • Digital Media (iTV, podcasting, iPod, weblog, smartmobs,…) as a Dynamic two-ways Participative Process calling for improved and more affective human-human and human-machine interaction • “User in the loop”: from a merely passive “consumer” to “active member” of a networked and participated community fully immersed in an intelligent pervasive environment • Emotional Multimodal Interfaces as the key enablers for allowing a more effective many-to-many two-way communication and interaction • Affective Multimodal Interaction supporting Digital Art & Entertainment icann2006, Athens Sept 14th 2006

3. Conveying Affectiveness in Leading-edge Living Adaptive Systems

4. The CALLAS project experience • CALLAS aims to define and develop a multimodal architecture including emotional aspects, targeted to digital media applications, essentially in an “ambient intelligence” paradigm. • CALLAS aims to be the creative laboratory for experimenting new concepts for a more participative and affective-enriched user interaction in digital media applications • The key assumption for CALLAS will be to allow a rapid and incremental prototyping, where some basic components, sub-assemblies or initial prototypes will be made available for extensive re-use icann2006, Athens Sept 14th 2006

5. CALLAS Objectives • To develop a set of software components dedicated to the processing of individual modalities, their fusion and semantic interpretation (the CALLAS Shelf), with input modalities (speech, sound, non-verbal behaviour and basic physical interaction) and output modalities (speech, non-verbal behaviour, affective ECA, music, haptics) • To propose a Software Engineering Framework, based on the customised integration of the above components into an architecture with authoring of application-specific control, supporting rapid prototyping and development • To implement prototype demonstrators (CALLAS Showcases), targeted to Multimodal interaction in the context of new Media (Digital Arts and Entertainment), with emphasis on emotional aspects icann2006, Athens Sept 14th 2006

6. Working Areas icann2006, Athens Sept 14th 2006

7. Advancing the state of the art • The project will investigate an interface paradigm in which multimodal input will be analysed in terms of representation mediating the interaction process rather than in terms of interface commands or information access. • a new way for investigating emotions in multimodal interfaces tailored to New Media environments. The advent of new media will be the enabler for a new way of conceiving emotions in multimodal interfaces, since these could be considered as “social objects”, emerging from a diffuse mediated interaction between group of users • Emotions as the key driver for allowing New Media to unleash yet partially unexplored potentialities for a more active two-ways communication and natural user interaction • From a technological perspective the project will design a software engineering approach supporting the development of multimodal applications icann2006, Athens Sept 14th 2006

8. Physical paradigm for Multimodal Interaction • Multimodal interaction in Mixed Reality environments tends to focus on commands/information access targeting objects (in the CALLAS proposal this has been referred to as “physical paradigm”) • One of the objectives of CALLAS is to support Multimodal interaction with environments as a whole (e.g. artistic installations), or with new Media content to better capture user experience • This should be achieved by: • An integration of Multimodal input over larger fragments of user interaction • The interpretation of Multimodal input in terms of emotional representations, which will mediate the installation response • Multimodal interpretation should capture the user experience through her relation to the multimodal application (e.g. measure “empathy”), going beyond the measuring of the traditional emotions icann2006, Athens Sept 14th 2006

9. CALLAS Showcases • Augmented Reality for Art, Entertainment, and Digital Theatre • Interactive Installations for Public Spaces • Next-Generation Interactive Television icann2006, Athens Sept 14th 2006

10. Augmented Reality for Art, Entertainment, and Digital Theatre • AR installations actually constitute an ideal environment for Multimodal interaction for the following reasons: • Because AR environments are already designed to accommodate a range of sensors for tracking the user, the same environment can track the user’s non-verbal behaviour, to be interpreted in terms of emotional response. • The sensing devices that support most implementations of AR systems can similarly support seamless interaction within the artistic installation. • This explains that more recently, digital artists have used professional AR environments, such as the ARToolkit, to develop interactive installations (Maia Engeli). icann2006, Athens Sept 14th 2006

11. Augmented Reality for Performing Arts • Every year throughout Europe new AR Artistic Installations are being developed, although individual production costs remain high (100k-700k€) • Traditional solutions to this problem have been: • the development of these installations as part of mixed Art and Technology structures, for instance academic centres in Digital Arts • the use of software development kits that facilitate the integration of the installation devices, such as Max/MSP (used for instance to support the development of Magali Desbazeille’s “you think hence I follow you”, or some of Studio Azzurro’s performances). icann2006, Athens Sept 14th 2006

12. Augmented Reality for Performing Arts: Digital Scenographies (1) • The emerging affective multimodal interfaces allow today new yet partially unexplored ways of conceiving, experience, living and enjoying pervasive and immersive theatre-based performances (dance, drama, plays, opera, etc.), which are more and more interactive, immersive, emotional, and open to new forms of relationships between the humans (actors, audience, stage director) and the adaptive interactive spaces. • This showcase will demonstrate how the real time detection of the affective state of the people involved in a live performance (directors, actors, audience) can improve the “naturalness” of the experience, paving the way for unexplored ways of broader and more pervasive participation of the users (audience). icann2006, Athens Sept 14th 2006

13. Augmented Reality for Performing Arts: Digital Scenographies (2) • The showcase aims at allowing all the performing arts practitioners (director, stage manager, actors, playwrights, scenic designer, costume designer) to use and master the latest affective and multi modal for interactively conceiving, designing, authoring, sharing and experiencing naturally interactive performing arts works. • It will also provide innovative natural immersive interfaces and devices for creating or living interactive performing art works within multidimensional (virtual, augmented, mixed) realities. • The real challenge will be just to move forward from an off-line design of a virtual stage to a real-time virtual scenography. icann2006, Athens Sept 14th 2006

14. 1. Storyboard 2. 3-d model A Prototype example Augmented Reality for live performances 1. Components analysis and identification – Storyboard, interaction concept, modality analysis (month 6) 2. Proof of Concept: AR technology baseline + Basic components integration and fusion, interpretation of components: speech, gesture recognition, sound capture (month12) 3. First trials with prototype installation, evaluation of interaction concepts support (month 18) 4. Extended prototype (at month 24) including: • Extended emotions representation • additional modality combinations • upgraded versions of the selected shelf components • Additional components from the framework (i.e. ECA) 5. Complete installations supporting users trials (month 36) icann2006, Athens Sept 14th 2006

15. Overall Exploitation strategy • The main business aspect for CALLAS will be the commercial exploitation and selling of the CALLAS framework, as well as consultancy to vertical application developers, service providers, producers, event organisers and network operators to create new interactive entertainment formats, accessible content and e-training applications. • The market for CALLAS framework will be mainly constituted by: • CALLAS customers, players (artists, technology providers) which are aimed at developing multimodal applications • CALLAS users, organisations/institutions which are interested in committing such multimodal applications (local authorities for festivals, theatres, etc.) • The CALLAS showcases will provide tangible evidence of the commercial potential offered by the technology and will be used by the supplier partners to focus their industrial offering. • The showcases demonstrated within the project are also expected to directly generate revenues in the final year of the project, and these would be rolled out by the user partners and user group members (Studio Azzurro, Teatro Massimo and Ars Electronica) to wider audiences. • The CALLAS consortium intends to provide the framework as open source. icann2006, Athens Sept 14th 2006

16. Multimodality fostering innovation in the New Media Market • Emotional multimodal interfaces as enablers of innovation in the Digital Media domain • Multimodal technology is pushing Process Innovation at two levels • Emerging technologies are changing the way in which the same things are made • New services/applications are emerging (mobile tv, internet TV, Digital Art, Digital Media, Co-operative Geo-based Videogames,…) icann2006, Athens Sept 14th 2006

17. Business Cases (1) CALLAS business is based on a wide range of different business cases: Cost reduction for: CR1 SMEs developing digital interactive installations (e.g. Studio Azzurro, Humanware) CR2 Traditional theatres (Teatro Massimo) using multimodal applications complementing the physical scenographies CR3 Local authorities, Digital Arts Museums setting-up interactive live performances (“white night” events, etc.) … Revenue increase for: RI1 Large Service Providers enlarging their market to Digital Entertainment Business (Engineering, Metaware) RI2 Companies providing new services or tools in the growing interactive TV market (DigitalVideo, XIM, BBC) RI3 Spin-offs based on providing advanced mobile services in festival (VTT, SNS) RI4 “Theme Parks” using Mixed Reality Installations … icann2006, Athens Sept 14th 2006

18. Business Cases (2) • Business case: Cost reduction in live performance • Description: large re-using of CALLAS components and/or sub-assembliesin the preparation and setting-up of a digital interactive opera • Analysis of estimated costs of two real cases: Michael Nyman opera “Facing Goya” and Neither opera in Stuttgart Staatsopera • Key assumption: • CALLAS will reduce costs and time for designing and implementing High-quality multimodal applications for artistic mixed-reality digital installations, since powerful and suitable software engineering techniques will be used • Conclusion: A 20% average cost reduction is expected in the production of a digital opera icann2006, Athens Sept 14th 2006

19. Engineering Business Case • Business case 1. Revenue Increase in Digital Entertainment Market • BC description: increase the market share for industry, telecommunication and services business line (currently 22%) throughout the penetration in the digital entertainment market • A strong commitment from the CEO (see Engineering internal magazine, “home page”) • Key financial assumptions: • growth rate for Engineering in last years of 17%, as in the corporate accounts • Analysis of revenue and market share in last three years • Extrapolation of data considering a changing in the market share, thanks to a larger penetration in the digital media one (despite with a decreeing of the finance one) • Conclusion: an expected value creation for 21 millions of Euro after 2007 (average) in term of revenue increasing and of 1,06 millions (average) in term of net profit. icann2006, Athens Sept 14th 2006

20. ENG Business Case: Figures icann2006, Athens Sept 14th 2006

21. iTV Business case • Creation of new iTV services • Problem: • saturation of traditional content (DTV in 75% European homes and broadband in 60% by 2010) • Many delivery models competing for consumer revenue (IPTV, 3G…) • At the same time, large scale investment from network operators in modernising and extending infrastructure • Opportunity: • Users becoming increasingly sophisticated while current DTV UIs are first generation • Set top box hardware and network performance is evolving to enable advanced applications and interfaces • Need for new services to grow the market • CALLAS will provide a framework to enable iTV developers to create empathic, emotion-sensitive interactive services to increase revenue and consumer loyalty icann2006, Athens Sept 14th 2006

22. iTV Business case • Cost reduction for iTV adoption of affective-based multimodal applications • Problem: • Deployment of affective interfaces would currently require complex bespoke development, resulting a specific application • The high volume of end users requires that hardware must be as low cost as possible with maximum use of efficient software/firmware and intelligent reuse of standard hardware • In order to be deployed to millions of homes, new iTV applications must be resilient, robust and be based on well-tested reusable components • Opportunity: • Callas is creating a set of reusable components which spread development costs across multiple new media applications, platforms and entertainment domains • Callas will therefore reduce costs and time for designing and implementing affective-based multimodal applications • Conclusion: Callas will help to make affective-based multimodal applications viable for iTV developers by sharing costs with other domains and maximising reusability icann2006, Athens Sept 14th 2006

23. Open source • CALLAS intends to provide the framework as “open source” software • The benefit of open source licence and an open architecture are: • The free availability of the framework and the architecture enables every interested user to be part of the CALLAS system with low investment costs. • New or improved functionalities can be added by the user community also after the lifetime of the project. • The CALLAS system can be ported to new platforms which are not supported by the project or which cannot be foreseen at the moment. • The CALLAS results will be sustainable as the maintenance can be done by the user community as well as by companies. icann2006, Athens Sept 14th 2006

24. Technology Transfer for Multimodal Interfaces • Many R&D institutions involved in multimodal interfaces domain • Plenty of dynamic and innovation-led SMEs • Multimodal interfaces market very risky and uncertain • “Incubator” Model for transferring technology • Large IT & Service Provider as “hub” for leveraging the research and the innovation to the market icann2006, Athens Sept 14th 2006

25. The Consortium Large Industrial Players:Engineering, BBC, Metaware New Media SMEs XIM, Nexture, Digital Video, Studio Azzurro, Humanware Research and AcademicVTT, UOA, ICCS, FPMS, TEES, TKK, PAR8, SNS, UOR, HIT, TMAS icann2006, Athens Sept 14th 2006

26. Technology TransferModel R&D Innovation-led SMEs Large Industrial Players Market (Application Developers, Third-Party Services, Artists,…) icann2006, Athens Sept 14th 2006

27. THANK YOU Diego Arnone Engineering Ingegneria Informatica Contacts and references :http://www.eng.it R&D Direction Roma – Italy Tel.: +39 06 49201425 Mob: +39 335 1038042 Email: massimo.bertoncini@eng.it icann2006, Athens Sept 14th 2006