Interface technologies and cognitive systems

1. Interface technologies and cognitive systems Philippe Gelin European Commission - Information Society Granada 16th December 2002

2. Content • PART I : European Research Area • PART II : Interface Technologies • PART III : Cognitive Systems • PART IV : FP6 Instruments

3. PART I The European Research Area (ERA) and its Sixth Framework Programme (FP6)

4. FP6 • The Treaty establishing the European Community provides for the creation of multiannual research and technological development • The Sixth Framework Programme (FP6) is the European Unions main instrument for the funding of research in Europe within 2002-2006. • The overall budget for FP6 is 17.5 billion euro.

5. ERA • In the past, collaborative actions have been initiated at European and Community level, • but now is the time to bring our endeavours together and to build a research and innovation equivalent of the "common market" for goods and services. • That structure is called the European Research Area (ERA) and is regrouping all Community supports for the better coordination of research activities and the convergence of research and innovation policies, at nationaland EU levels.

6. ERA

7. From ERA to IST 1. Focusing and integrating Community research 1.1. Priority thematic areas of research 1.1.1. Life sciences, genomics and biotechnology for health (i) Advanced genomics and its applications for health. (ii) Combating major diseases. 1.1.2. Information Society Technologies(3.625 billion Euro)1.1.3. Nano-technologies and nano-sciences, knowledge-based multifunctional materials, and new production processes and devices 1.1.4. Aeronautics and space 1.1.5. Food quality and safety 1.1.6. Sustainable development, global change and ecosystems (i) Sustainable Energy Systems (ii) Sustainable surface transport (iii) Global change and ecosystems 1.1.7. Citizens and governance in a knowledge-based society 1.2. Specific activities covering a wider field of research 1.2.1. Policy support and anticipating scientific and technological needs 1.2.2. Horizontal research activities involving SMEs 1.2.3. Specific measures in support of international cooperation 2. Strengthening the foundations of the European Research Area 2.1. Support for the coordination of activities 2.2. Support for the coherent development of policies

8. IST Programme 2002-2006 • Instantiate the European challenges (ERA), Initiatives • Trans national partnerships • Critical mass of skills and adequate resources hence ... • Focus, Excellence, Impact and RoI

9. IST Vision IST today …………………………..The IST in FP6 vision • PC based ………………………………. “Our surrounding” is the interface • “Writing and reading”…………….…. …Use all senses, intuitive • “Word” based information search……….Context-based knowledge handling • Low bandwidth, separate networks…. ….Infinite bandwidth, convergence, .. • Mobile telephony (voice)………….. ……Mobile/Wireless full multimedia • Micro scale………………………………Nano-scale • Silicon based……………………………..+ new materials • e-Services just emerging……………….. Wide adoption (eHealth, Learning,..) • < 10% of world population on-line….. …World-wide adoption

10. Knowledge andInterface Technologies Communication ComputingSoftware Future andEmerging Technologies Technologies for major economicand societal challenges ComponentsMicrosystems IST www.cordis.lu/ist/fp6/fp6.htm

11. Call 2 Advanced displays Optical, opto-electronic, & photonic functional components Open development platforms for software and services Cognitive systems (25ME) Embedded systems Applications and services for the mobile user and worker Cross-media content for leisure and entertainment (55ME) GRID-based Systems for solving complex problems Improving Risk management Call 1 Pushing the limits of CMOS, preparing for post-CMOS Micro and nano-systems Broadband for all Mobile and wireless systems beyond 3G Towards a global dependability and security framework Multimodal interfaces (65ME) Semantic-based knowledge systems Networked audiovisual systems and home platforms Networked businesses and governments eSafety of road and air transports e Health Technology-enhanced learning and access to cultural heritage (65ME) ! ! ! ! ! PROVISIONAL PROVISIONAL PROVISIONAL PROVISIONAL Strategic Objectives in 2002-2003 WP 80% of budget is pre-allocated Remainder flexible, to be allocated after evaluation Allocation only for the first 2 years: to be revised for 2005-2006

12. PART IIInterface Technologies in FP 6scope and focus in 2003

13. Outline • Why interface technologies? • IST Vision • What research? • Focus of first calls in FP6 • Where are we today? • Ongoing activities

14. Vision • Building the knowledge society for all • Allow people to use the surrounding hi-tech easily and in a way that is consonant to humans rather than to computers • Facilitate interpersonal communication anywhere, anytime, beyond linguistic and cultural barriers • Assist people and augment their abilities when interacting and communicating

15. Needs • Bring people to the foreground – build technologies for the background • Our surrounding is the interface: everywhere, embedded in everyday objects (e.g: clothes, furniture, vehicles, smart materials, …) • Exploring interfaces beyond the PC, screens & keyboard. (brake the chains…) • Enabling multi-sensorial dialogues supported by ubiquitous computing & networking • Multi-linguistics, multi-cultural interactions

16. Known Challenges • Strongly multidisciplinary with many component technologies • Significant system level integration issues • Research resulting in autonomous (self-learning, self-organising), adaptive (time varying) systems that can work with partial and uncertain information

17. Multimodal Interfaces (in call 1) Objective: To develop natural and adaptive multimodal interfaces, that respond intelligently to speech and language, vision, gesture, haptics and other senses. Focus 1 Multimodal interaction Focus 2 Multilingual communication 2003-2004 FOCUS

18. Interaction between and among humans and the virtual and physical environment intuitive multimodal interfaces that are autonomous and capable of learning and adapting to the user environment in dynamically changing contexts. They should recognise emotive user reaction and feature robust dialogue capability with unconstrained speech and language input. Multimodal Interaction • Human-to-human: technology mediated communication • Human-to-things: virtual and physical • Human-to-self: health, well being • Human-to-content: information retrieval/browsing • Device-to-device: human mediated device communication • Human-to-embodied robots

19. facilitating translation for unrestricted inputs, especially for spontaneous (unrestricted) or ill-formed (speech) inputs, in task-oriented settings Unrestricted spontaneous speech-to-speech translation in task-oriented settings Statistical/mixed approaches to translation Adaptation to task/user, learning Robustness Multilingual Systems

20. Basic research, component technologies Examples: accurate vision gesture tracking speech and audio processing language technologies affective computing machine learning autonomous systems fusion of multiple channels Basic Research

21. Proof of concept in challenging application domains, including: wearable interfaces and smart clothes intelligent rooms and interfaces for collaborative working tools cross-cultural communications usability issues and evaluation System Level Integration

22. Data: large amounts of multimodal data, synchronisation and IPR issues Metrology, technology evaluation, usability Infrastructural knowledge: machine learning, applied mathematics Best practice and standards Tools and platforms Socio-economic issues (e.g. human factors) Shared Infrastructure

23. Current Status

24. Collection, processing and browsing of multimodal meeting data systems that enable recording, structuring, browsing and querying of an archive of multi-modal recordings of meetings. Multimodal, multicultural, multilingual communication integration of multiple communications modes - vision, speech and object manipulation - by combining the physical and virtual worlds to provide support for multi-cultural communication and problem solving Speech-to-speech translation development of speech-to-speech translation and its integration in automatic in e-commerce and e-service environments data collection for speech-to-speech translation Ongoing Activities

25. Preparing future multisensorial interaction research providing technological baselines, comparative evaluations, and assessment of prospects of core technologies for speech-to-speech translation, the detection and expressions of emotional states, and technologies for children’s speech. Automatic animation of human models design and development of a virtual person animation system in controlled environments, which enables the modeling, analysis and simulation of human motion. Recognition of the user’s emotional state human computer interaction that can interpret its users' attitude or emotional state in terms of their speech and/or their facial gestures and expressions. See www.hltcentral.org for more information Ongoing Activities

26. PART III Cognitive Systems in FP6scope and focus

27. Outline • why cognitive systems? IST vision • what research? focus of call • where are we starting from? current activities

28. Vision Ambient intelligence “… ability of computationally empowered devices to interconnect with each other and with us …” • sensors provide a window from world of interconnected computation into real physical world • using sensors, these devices will sense the world around us and respond by interacting with the world or by communicating with us • they need to ‘see’, ‘hear’, …. ‘understand’ and ‘act’ • Ambient Intelligence requires perceptual systems capable of cognition

29. Objective To constructphysically instantiatedorembodiedsystemsthat can : • perceive, • understand(the semantics of information conveyed through their perceptual input) • learning (structured memory) • interactwith their environment, (CCI & CHI communication) • evolvein order to achieve human-like performance in activities requiring context-(situation and task) specific knowledge.

30. Focus 2002-2003 • Research will aim at realizing: • completesystems with • real-timeperformance and/or bounded rationality, • have well developedmemory capacities(e.g. short term, long term, iconic, associative) with efficient representation, • and thatacquire representationsas needed to realise performance goals. • to performgoal-directedtasks • The emphasis is on closing the loop in realistic test cases.

31. Known Challenges • A main target of this research isinterdisciplinarity: • computer vision, • natural language understanding, • robotics, • artificial intelligence, • mathematics and • cognitive neuroscience • and its impact on overall system design.

32. Outcomes • Direct Applications: • in image recognition, in behavioural interpretation, in video annotation, speech recognition, in automatic categorisation and classification, in goal-specification and decision support,… • can serve to demonstrate and measure progress but applications NOT main target of research • Methods & approaches for constructing robust & adaptive systems capable of cognition

33. Ongoing Activities • cognitivevisionsystems 2000 – 2002 • robust image recognition • from application-specific togeneralised solutions • from focus on low-level processing & robustness of individual components to systems approach where every component (incl. high-level cognitive functionalities) has a role to play in assuring robust behaviour of system • longer-term, largely academic, interdisciplinary effort • 8 RTD projects & 1 Network: www.ecvision.info

34. Related Topics • Multimodal Interfaces, call 1 intuitive multimodal interfaces ... autonomous and capable of learning and adapting … in dynamically changing contexts • Beyond Robotics, call 1 focus on open-ended lifelong learning systems • Disappearing Computer, call 1 focus on open architectures allowing arbitrary combinations of ‘building blocks’ for ‘universal application’ • Embedded Systems, call 2 focus on networked systems, distributed control

35. PART IV Implementation

36. Outline • Schedule • New Instruments • Old Instruments • Applied to IFT & CS. • Expected results • Learning from EoI • “Do’s and don’ts” • Conclusions

37. FP6 ROADMAP • 21 February 2001 Commission decides on first proposal for Framework Programme • March 2001 Informal presentation to the Ministers of Research and Education during the Uppsala Informal Council (1 - 3 March) • May 2001 Adoption by the Commission of proposals regarding the specific programmes and the implementation of the Framework programme • September 2001 Adoption by the Commission of proposals for decisions on the rules for participation and dissemination of results • 26 June 2001 Council discusses Commision proposal • July/Nov. 2001 First reading by the European Parliament • 12/2001- 01/2002 Common position of the Research Council • 11 March 2002 Council adopt final text • 15 May 2002 The European Parliament approves the Sixth Framework Programme • 3 June 2002 Council of Ministers formally adopted the Sixth Framework Programme • 12 June 2002 Parliament approves specific programmes • 19 June 2002 FP6 rules of participation compromise approved by ITRE • 3 July 2002 FP6 rules of participation compromise approved by EP • September 2002 Specific Programmes has been adopted (October) • End 09/2002 Rules for Participation to be formally adopted by Council and Parliament • November 2002 Conference marking the launch of the Sixth Framework Programme • 12/2002 First calls for proposals • end 2003 Second calls for proposals

38. ! ! ! ! ! PROVISIONAL PROVISIONAL PROVISIONAL PROVISIONAL Call 1 Timetable Publication: 2nd half December 2002 Call deadline: 24 April 2003 Evaluation: weeks of 12th and 19th May 2003 IP/NoE hearings: week of 9th June 2003 Negotiations: from July 2003 Commission Decision: from October 2003 Project start: January 2004

39. Overview of FP6 instruments • Integrated Projects (IP) • Networks of Excellence (NoE) • Article 169 (joint implementation of national programmes) • Specific Targeted Research Projects (STReP’s) • Co-ordination actions (CA) • Specific Support Actions (SSA) „New“ „Old“

40. Use of New Instruments

41. Why new instruments ? • Adapt to the size of the challenge • The philosophy of both instruments is to movefrom multiple projectfunding to the funding of coherent programmes of research activities, • Flexibility and responsibility • leaving the highest degree possible of autonomy and flexibility to European research consortia.

42. Integrated Projects • Expected to address the objectives within a holistic approach enabling, • where justified, competition within and across projects • A major target is to integrateactors and disciplines= build a community • An IP should be THE project in the target area • “critical mass”

43. Integrated Projects • Core activities • genuine research work • “engineering” tasks (esp. methods & tools) • system integration & validation (“total system” approach) • along with • promotion & dissemination of results • training, awareness & best practice (researchers, integrators, launching users) • cooperation & exchanges with related nationaland international efforts (incl. standards bodies) • socio-economic impact & consequences

44. Integrated project Current projects Example of an IP Area: Multi-sensorial interfaces Knowledge & Data bases Virtual & Augmented Realities Intelligent Assistants & Virtual Humans Intelligent Virtual Worlds Physical Virtual World Interface Layer SpeechOutput Gesture & Facial Recognition Actuators Displays Body Sensors Haptic devices Smell Taste Speech

45. Reporting Accepted costs Detailed work plan Reporting Accepted costs Adjusted advance Accepted costs Detailed work plan Reporting Adjusted advance Reporting Adjusted advance Accepted costs Detailed work plan Initial advance Detailed work plan 0 6 12 18 24 30 36 42 48 Months IP - payment & report schedule Example of 4 year contract

46. Networks of Excellence • Spread excellence • Xfertilisation among disciplines • Bring together the best research in constituent disciplines, integrate resources, (e.g. image, video, multimodal,…) • Create a NEW research space (ERA) • Have a lasting nature • Define joint activities

47. Networks of Excellence • Joint programme of activities: • Integrating activities • coordinated programming of the partners’ activities • sharing of tools/facilities/ infrastructures • managing the knowledge portfolio • staff mobility and exchanges • Joint researchactivities • undertaking research on topics supporting the network’s goals • Spreading excellence • training of researchers and other key staff • dissemination and technology transfer to industry • “virtual” centres of excellence New or additionalto ongoing efforts

48. NoE - grant • Each call will fix the founding per researcher, based on the characteristics of the field of research concerned • Computer science less expensive than nuclear research • The number of researchers implied in the integration • PhD or more than 4 years of research • Existing head count at time of proposal. • Table of Found Per researcher issued at each call • @ max the cost incurred to implement the JPA • @ max 25% of the existing value proposed to be integrated

49. Use of Old Instruments

50. Specific targeted research projects • Similar to Old instrument, except for: • joint and several financial liability • costing / funding adapted to grant scheme • rates / activities: research (50%), demonstration (35%), combined research & demonstration • Activities • focussed RTD and / or demonstration; management • Partnership • more limited than for IPs; in terms of breadth, time, scope