Fore sight Cognitive Systems Project

1. Foresight Cognitive Systems Project InterAction Conference “The Connective Tissue” Richard Morris and Lionel Tarassenko 5th September 2003

2. Cognitive Systems A working definition “ Cognitive systems are natural or artificial information processing systems, including those responsible for perception, learning, reasoning, and decision-making, and both communication and action ” Foresight Cognitive Systems IAC (LT and RGMM)

3. What this Foresight Projecthas been about... • It is a project that has been run by scientists and supported by Foresight who haveprovided the scope for events and activities designed by the communities themselves • Interaction and collaboration between the physical and life sciences communities has been one of the key strengths of the project • The main aim has been to provide a vision for the future of research in cognitive systems, both for the design of artificial cognitive systems (applications) and to further our scientific understanding of biological systems

5. The biologically-inspired approach focuses on the scientific study and, where appropriate, exploitation of naturally-occurring cognitive systems • The pure engineering approach seeks to create artificial systems that exhibit some desired level of cognitive performance or behaviour • Can we achieve more by combining the biologically inspired approach with the mathematical models of the engineering approach?

6. Overview • Speech & language (Grand Challenge) • 3-D vision (Grand Challenge) • How brains wire themselves (Keynote lecture) • Brain rhythms (Grand Challenge) • Memory & forgetting (Debate) • Framework for the future?

7. Not covered here but some very important themes to emerge in discussion • Robotics • Agents • Learning • Levels of analysis

9. Not covered here but some very important themes to emerge in discussion • Robotics • Agents • Learning • Levels of analysis and emergent properties • Sensor fusion • Affective cognition

10. Grand Challenge Language and speech Goals: • To construct a neuro-biologically realistic, computationally specific account of human language processing. • To construct functionally accurate models of human interaction based on and consistent with real-world data. • To build and demonstrate human-computer interfaces which demonstrate human levels of robustness and flexibility. To understand and emulate human capability for robust communication and interaction. 10 Grand Challenge

11. Progress in Automatic Speech Recognition Easy Word Error Rate Hard 11 State of the Art: Speech Recognition

12. What Humans Do that Today’s Systems Don’t • Use context to interpret and respond to questions • Ask for clarification • Relate new information to what’s already been said • Avoid repetition • Use linguistic and prosodic cues to convey meaning • Distinguish what’s new or interesting • Signal misunderstanding, lack of agreement, rejection • Adapt to their conversational partners • Manage the conversational turn • Learn from experience 12 State of the Art: Computational Language Systems

13. 790 ms 800 ms 750 ms 760 ms 770 ms 780 ms 740 ms 720 ms Demonstration using MEG to track cortical activity related to spoken word recognition 700 ms 13 State of the Art: Cognitive Neuroscience of Speech and Language

14. Summary of Benefits To understand and emulate human capability for robust communication and interaction. • Greater scientific understanding of human cognition and communication • Significant advances in noise-robust speech recognition, understanding, and generation technology • Dialogue systems capable of adapting to their users and learning on-line • Improved treatment and rehabilitation of disorders in language function; novel language prostheses 14 Grand Challenge

15. Grand Challenge - 3D Vision

16. Measurements

17. 3D Structure using computational geometry

18. A 3D Visual Task

19. Biological systems can perform 3D tasks using visual information • Pointing to remembered objects • Ants homing Real-time 3D localisation in natural scenes using the engineering approach will not be possible simply as a result of increase in computational power • “computer vision hitting a computational wall” • The design of a biologically plausible model of 3D localisation is a Grand Challenge which would take us through the wall

20. Overview • Speech & language (Grand Challenge) • 3-D vision (Grand Challenge) • How brains wire themselves (Keynote lecture)

21. How brains wire themselves (Keynote lecture – Mriganka Sur) Specificity Plasticity

22. Optic tract Primary visual cortex Lateral geniculate n. Optic radiations The visual cortex has specific processing networks D.H. Hubel T.N. Wiesel Orientation selectivity in V1: How do orientation networks form?

23. Stimulus computer Amplifier Camera Light guide Video data acquisition Optical imaging of cortical activity Cortical vasculature Composite orientation map Single orientation images

24. Auditory cortex Visual cortex Superior colliculus LGN Inferior colliculi MGN Visual cortex LGN MGN Normal Visually responsive auditory cortex Rewired Rewiring alters the pattern of activity to the developing cortex M. Sur and C. Leamey, Nature Reviews Neurosci, 2001

25. Orientation maps arise in rewired A1 J. Sharma, A. Angelucci, M. Sur, Nature, 2000

26. Some aspects of arise by virtue of Implications : should engineering systems incorporate self-organisation and, if so, how? How brains wire themselves Specificity Plasticity

27. Brain rhythms (Grand Challenge) • How are representations of perceptual events given the correct temporal organisation for storage and recall? • The rhythmic activities of different groups of neurons in the brain may play a fundamental role in helping us to do this. • Computer scientists working in the area of asynchronous computing are interested in any insight into how complex asynchronous natural systems can deliver coherent behaviours

28. Gamma-theta interaction From VanRullen & Koch, 2003 Hypothesis is that each period of the fast gamma rhythm underlies a specific representation. Gamma is superimposed on a slower rhythm (alpha or theta) that effectively multiplexes the representations This mechanism could explain how the interactions between neuronal rhythms participate in shaping the holding in short-term (working) memory of perceptual events: the fast wave representations would constitute the contents of each discrete snapshot, the entire percept being mediated by the slow waves.

29. Memory and Forgetting (Debate) • New “intelligent” information processing software should take more account of advances in our growing understanding of human memory systems. • But should they compensate for, or mimic their known failings, including forgetting?

30. Synaptic potentiation shows varying persistence

31. Memory also fails for psychological reasons….

32. The sin of transience The sin of absent-mindedness The sin of blocking The sin of misattribution The sin of suggestibility The sin of bias The sin of persistence Weakening or loss Breakdown of attention Thwarted memory search Assigning to wrong source Implanted by a leading question Editing and rewriting Repeated recall of disturbing information The “seven sins” of memory After Schacter (2001)

33. The “seven sins” of memory (Dan Schacter) • Schacter argues that they are not really “failures” at all, but reflect the proper operation of a finely tuned system…. not vices but virtues. • Should we build into artificial devices for storing and retrieving information the same “trade-offs” between memory and forgetting that we see in human systems?

34. Overview • Speech & language (Grand Challenge) • 3-D vision (Grand Challenge) • How brains wire themselves (Keynote lecture) • Brain rhythms (Grand Challenge) • Memory & forgetting (Debate) • Framework for the future?

35. Framework for the Future • In the study of “cognitive systems”, can the interaction between life and physical sciences be better promoted by setting up an appropriate infrastructure? • Critical mass of scientists • Technology and instrumentation • Dedicated research programs • Relevant training programs

36. If yes…. • Momentum generated by this Foresight Project must be maintained • The following might be fruitful areas of interaction: • 3D-Vision for natural scenes • Speech and language – natural dialogue • Memory systems • Action and robotics • Social cognition • Other topics also offer important and tractable problems.

37. Possible research themes (not an exhaustive list) • How do humans recognise objects? • How could the new paradigm of generative models help us to understand sensory processing in mammalian brains? • The use of context in both artificial and natural systems • Can appreciation of the social context of speech get us beyond the apparent limits of machine-learning? • How does the brain encode and remember time? • How are we to understand intentionality in action? • What are the principles and functional consequences of self-organisation?