Summary

1. IST-1999-20410MEGA Multisensory Expressive Gesture ApplicationsConcertation MeetingBrussels, 3-4 October 2000Antonio Camurri (DIST-University of Genoa)MEGA Project Coordinator

2. Summary • MEGA: Concepts, Objectives and Mission • The Consortium • Project Overview: Workplan, WPs, and Deliverables

3. MEGA: Concepts, Objectives and Mission • Concepts: • Multisensory • Expressive (emotion, kansei, affect) • Gesture • Applications

4. MEGA: Concepts, Objectives and Mission • Objectives: • modelling and communication of expressive and emotional content in non-verbal interaction by multi-sensory interfaces • design of novel paradigms of interaction • analysis, synthesis and communication of expressive content in networked environments • implementation of the MEGA System Environment • development of artistic public events showing the usefulness, flexibility and power of the MEGA System Environment

5. MEGA: Concepts, Objectives and Mission • Mission: • Consolidating a third original European routein affective computing (USA) and KANSEI (Japan). • Raise the competitiveness of European Industry on • Digital Music Instruments • Entertainment and Edutainment • Multimedia: e.g. VEs, e-commerce, intelligent interfaces, ... by building the MEGA System Environment and showing its effectiveness in public events.

6. Participant no. Participant name Participant short name Country Status* 1 DIST - University of Genoa Lab. on Musical Informatics DIST I C 2 DEI - University of Padova DEI I P 3 IPEM – University of Gent IPEM B P 4 KTH - Royal Institute of Technology, Stockholm KTH S P 5 Uppsala University, Department of Psychology UUP S P 6 Telenor TLN NO P 7 Generalmusic GEM I A 8 Consorzio Pisa Ricerche CPR I P 9 Eidomedia EM I A Participants

7. Workplan: Workpackages • WP1 Project management • WP2 Requirements • WP3 Analysis in human movement and gesture • WP4 Analysis in musical gesture performance • WP5 Synthesis in visual media and movement • WP6 Synthesis in music performance • WP7 MEGA System Environment • WP8 MEGA Multimedia Performance • WP9 Dissemination and implementation

8. Workplan: gantt - Spiral approach - Three phases, each concluded at a project milestone (13, 25, 36 months).

9. WP2 Requirements • WP Leader: IPEM • State of the art, share of results among partners, Requirements and setup for the research, the MEGA System Environment and the MEGA Multimedia Performance products. • Definition of a standard communication protocol for expressive data. The protocol is defined in this WP and is implemented in the WP7 (MEGA System Environment). • Contributes from all the partners. • D3 (month 6) Requirements • D9 (month 14) First requirement revision • D12 (month 26) Final requirements

10. WP3-6: Research on analysis and synthesis of expressivity • WP3 Analysis in human movement and gesture (DIST) • WP4 Analysis in musical gesture performance (DEI) • WP5 Synthesis in visual media and movement (EM) • WP6 Synthesis in music performance (KTH) • Results: a selection of techniques and algorithms, integrated as s/w modules in the MEGA System Environment (WP7). These developed tools and modules will be continuously integrated and updated in the EyesWeb architecture, i.e., WP3-6 will deliver tested library modules and associated manual pages using the standard adopted in the project • Deliverables: Reports (D5, D13), S/w tools (D6, D14)

11. WP3 Analysis in human movement and gesture • WP Leader: DIST • To extract expressive content from movement patterns. • To evaluate and compare the results and performance of the obtained methods and algorithms (DIST and KTH) with the results obtained by psychological analysis of movement (UUP) with selected groups of users, both non-expert and expert (dancers, choreographers). • To develop a project reference archive of microdances and movement patterns(DIST, KTH, UUP), for both psychological analysis and automated analysis, using the standard adopted in the project (audiovisual, mocap data).

12. WP4 Analysis in musical gesture performance • WP Leader: DEI • Analysis of the different cues characterising an expressive musical performance (UUP, DEI); • Analysis of the correlation between cues and the instrument played by the performer (DEI); • Development of a model to extract automatically an expressive content of a musical performance (IPEM); • Development of tools implementing the model (DEI, IPEM); • Evaluation and test of the effectiveness of the model by means of performances analysis with the implemented tools (KTH, UUP);

13. WP5 Synthesis in visual media and movement • WP Leader: EM • Techniques inspired to the audiovisual language in cinema (EM) and their utilisation in virtual environments (TNR) and in visual and robot-human interaction (DIST). • Development of models and methods to synthesise expressive visual media outputs (EM). • Expressive synthesis of movements of synthetic characters and expressive modulation of VEs and Mixed Reality envs (TNR), depending on the expressive communication goals and content. • Expressive movement of small robots in concrete apps on robot-human interaction and mobile scenery (DIST). • Integration with Dovre VE architecture (TNR).

14. WP6 Synthesis in music performance • WP Leader: KTH • Development of tools both in MIDI and in digital audio format (KTH, DEI), to be used for the project products (WP7-8). • Synthesis on MIDI format based on Director Musices (KTH) • Adapting and integrating audio electronics for real-time expressive processing (GEM)

15. WP7 MEGA System Environment • WP Leader: DIST • Deliverable D4: EyesWeb software release, including tutorial and reference manuals (month 6) • Deliverable D7: 1st version of MEGA Sys Env. (12) • Deliverable D10: 2nd version of the MEGA Sys Env. (24) • Deliverable D15: Final product - MEGA Sys Env (35) • DIST: EyesWeb and architecture CPR: Wireless Sensors • TNR: Dovre IPEM: Mapping Strategies, Eval • EM: Wireless Sensors

16. MEGA System Environment (WP7) • Main components: • the EyesWeb software architecture (from DIST), • the (open) libraries of software modules concerning analysis and synthesis of expressive gesture (WP3-6), • the implementation of the protocol for expressive communication (WP2), • the h/w electronics and sensor systems for wireless non-invasive real-time motion capture (adapted from existing systems by CPR, DIST, EM), integration of commercial sensor systems (e.g., radio baton) and audio electronics systems (by GEM).

17. WP8 MEGA Multimedia Performance • WP Leader: IPEM • Main components: • the specification of the artistic use of the MEGA System Environment (WP7) in terms of the expressive content analysis and synthesis (IPEM, DEI); • the support to specific multimedia artistic events, based on the MEGA System Environment (DIST, KTH); • the evaluation of the MEGA System Environment with respect to audience acceptance, effectiveness, usability, and flexibility (IPEM).

18. WP9 Dissemination and implementation • WP Leader: TNR • To set up the Liason Board • To disseminate products (WP7-8) and research results (Workshops, book, Frankfurt and other exhibits, etc). • Internet-based dissemination and exploitation strategyby a free distribution to users of the software (under certain constraints and on-line contract forms), • To enable external users to develop new “plugins” of the MEGA System Environment. • Evaluate (UUP) the results from the project from WP7-8 and the internet dissemination strategy, with respect to the planned outputs. • D2: Dissemination and Use plan (6) • D17: Technological Implementation Plan (36)

19. Dissemination (WP8 and WP9) • via public performances (WP8) • An internet site providing the software and support to users (WP9): • s/w: the full MEGA System Environment, incl. a Wizard enabling users to develop new system plugins • library of plugins deriving from WP3-6 (research) • full documentation in the standard adopted • Workshops, special tracks and sessions at IEEE CS, ACM Conf, at CEC Workshops, at Frankfurt Messe 2001 and/or 2002 (WP9)