Action-Perception-Learning Cycles 2012 Fall Graduate Course

1. Action-Perception-Learning Cycles2012 Fall Graduate Course Byoung-Tak Zhang Department of Computer Science and Engineering & Cognitive Science and Brain Science Programs Seoul National University http://bi.snu.ac.kr/

2. What is a Learning System? • Learningis the improvement of performance in some environment through the acquisition of knowledge resulting from experience in that environment. the improvement of behavior on some performance task through acquisition of knowledge based on partial task experience 2012 (c) SNU Biointelligence Lab, http://bi.snu.ac.kr/

3. Machine Learning: An Example Error Backpropagation Output Comparison Information Propagation Weights Input x1 Input x2 Output Input x3 Input Layer Hidden Layer Output Layer Scaling Function Activation Function Activation Function 2012 (c) SNU Biointelligence Lab, http://bi.snu.ac.kr/

4. Application Example:Autonomous Land Vehicle (ALV) • NN learns to steer an autonomous vehicle. • 960 input units, 4 hidden units, 30 output units • Driving at speeds up to 70 miles per hour ALVINN System 2012 (c) SNU Biointelligence Lab, http://bi.snu.ac.kr/

5. Google “Self-Driving Car” • DARPA Grand Challenge (2005) • DARPA Urban Challenge (2007) • Google Self-Driving Car (2009) 2012 (c) SNU Biointelligence Lab, http://bi.snu.ac.kr/

6. Machine Learning (ML): Three Tasks • Supervised Learning • Estimate an unknown mapping from known input and target output pairs • Learn fw from training set D = {(x,y)} s.t. • Classification: y is discrete • Regression: y is continuous • Unsupervised Learning • Only input values are provided • Learn fw from D = {(x)} s.t. • Compression • Clustering • Reinforcement Learning • Not target, but rewards (critiques) are provided “sequentially” • Learn a heuristic function fw from Dt= {(st,at,rt) | t = 1, 2, …} s.t. • Action selection • Policy learning 2012 (c) SNU Biointelligence Lab, http://bi.snu.ac.kr/ Zhang, B.-T., Next-Generation Machine Learning Technologies, Communications of KIISE, 25(3), 2007

7. Machine Learning Models • Probabilistic Learning • Bayesian Networks • Helmholtz Machines • Markov Random Fields • Hypernetworks • Latent Variable Models • Generative Topographic Mapping • Other Methods • Decision Trees • Reinforcement Learning • Boosting Algorithms • Mixture of Experts • Independent Component Analysis • Symbolic Learning • Version Space Learning • Case-Based Learning • Neural Learning • Multilayer Perceptrons • Self-Organizing Maps • Support Vector Machines • Kernel Machines • Evolutionary Learning • Evolution Strategies • Evolutionary Programming • Genetic Algorithms • Genetic Programming • Molecular Programming 2012 (c) SNU Biointelligence Lab, http://bi.snu.ac.kr/ Zhang, B.-T., Next-Generation Machine Learning Technologies, Communications of KIISE, 25(3), 2007

8. FromMachine Learning to Brain-Like Cognitive Learning

9. Machine Learning vs. Human Learning Machine Learning • Clear separation of learning and inference • Examples are assumed to be statistically independent • Mainly numerical, quantitative change • One-shot learning is difficult • Requires uniquely labeled examples (supervised classification) • Good at discrimination and classification (discriminative) Human Learning • Learning and inference interleaved • Previous learning affects the next learning (dynamic) • Relational, qualitative change possible • One-shot learning is frequent • Learns from unlabeled or self-labeled examples (self-supervised) • Can generate prototypes and instances (generative) 2012 (c) SNU Biointelligence Lab, http://bi.snu.ac.kr/

10. Human Learning: Properties • Sensorimotor • Real-time • Predictive • Incremental • Dynamic • Structural • One-shot • Self-supervised • Prototypical • Generative • Recall 2012 (c) SNU Biointelligence Lab, http://bi.snu.ac.kr/

11. Humans and Computers The Entire Problem Space Human Computers What Kind of Computers? Current Computers 2012 (c) SNU Biointelligence Lab, http://bi.snu.ac.kr/

12. Cognitive Systems Cognitive Systems Require Cognitive Computing or Cognitive Information Processing Cognitive Computing Cognitive System Real-Time Dynamics Openness Multisensory Integration Perception Sequential Generation Action Zhang, B.-T., Communications of KIISE, 30(1):75-111, 2012 2012 (c) SNU Biointelligence Lab, http://bi.snu.ac.kr/

13. TU Munich “Rosie” the Cognitive Robot 2012 (c) SNU Biointelligence Lab, http://bi.snu.ac.kr/

14. Apple “Siri” Personal Assistant 2012 (c) SNU Biointelligence Lab, http://bi.snu.ac.kr/

15. Toward Human-Level Computational Intelligence:A Perspective of the SNU Biointelligence Lab • Q1: What capability is fundamentally missing for achieving human-level computational intelligence? • A1: Human-level machine learningthat enables rapid, flexible, and robust decisions and actions in dynamic and uncertain environments. • Q2: What aspect is the most essential to study human-level machine learning? • A2: Lifelong learning with perception-action cycles, i.e. the circular flow of information that takes place between the organism and its environment in the course of a sensory-guided sequence of behavior towards a goal (Fuster, 2004). • Q3: What capabilities are required for lifelong learning in perception-action cycle systems? • A3: Dynamic, incremental, online, and predictive learning. Flexible representation and fast reorganization. Multisensory integration, sensorimotor imagery, and sequential decision making. Active, selective attention. Balancing exploration and exploitation. Self-awareness, motivation, self-sustainability…. 2012 (c) SNU Biointelligence Lab, http://bi.snu.ac.kr/

16. Course Introduction • From machine learning to brain-like cognitive learning • Brain as a physical, thermodynamic computer • Perception-action cycles and Carnot cycles • Models of action-perception-learning cycles 2012 (c) SNU Biointelligence Lab, http://bi.snu.ac.kr/

17. Brain as a Physical, Thermodynamic Computer

18. Brain as a Physical, Thermodynamic Computer • Brain is an open, dissipative system, operating far from thermodynamic equilibrium. • Brain requires energy and matter to exchange with its environment to maintain stability. • Brain can be excited internally by chemical (enzymes) and electrical means (action potentials) as well as externally. • Continuous sensing of external world and internal world. • Continuous action on external world and internal world. 2012 (c) SNU Biointelligence Lab, http://bi.snu.ac.kr/

19. Mapping the World 2012 (c) SNU Biointelligence Lab, http://bi.snu.ac.kr/

20. 2012 (c) SNU Biointelligence Lab, http://bi.snu.ac.kr/

21. 2012 (c) SNU Biointelligence Lab, http://bi.snu.ac.kr/

22. 2012 (c) SNU Biointelligence Lab, http://bi.snu.ac.kr/

23. 2012 (c) SNU Biointelligence Lab, http://bi.snu.ac.kr/

24. Carnot Cycle for a Pyramidal Neuron [Fry, 2005; Fry, 2008] 2012 (c) SNU Biointelligence Lab, http://bi.snu.ac.kr/

25. Carnot Cycle for the Brain 2012 (c) SNU Biointelligence Lab, http://bi.snu.ac.kr/ [Freeman et al., 2012]

26. Information Physics of Biological Systems [Bialeket al., 2007] 2012 (c) SNU Biointelligence Lab, http://bi.snu.ac.kr/

27. [Slide by Robert Fry] 2012 (c) SNU Biointelligence Lab, http://bi.snu.ac.kr/

28. [Slide by Robert Fry] 2012 (c) SNU Biointelligence Lab, http://bi.snu.ac.kr/

29. Perception-Action Cycles

30. (참고: Andrew Ng, Stanford Univ.) Perception-Action Cycle in Autonomous Helicopter Control Stanford Autonomous Helicopter - Airshow #2: http://www.youtube.com/watch?v=VCdxqn0fcnE 2012 (c) SNU Biointelligence Lab, http://bi.snu.ac.kr/

31. Perception-Action Cycle in Humans [Trommershaeuseret al., Sensory Cue Integration, 2011] 2012 (c) SNU Biointelligence Lab, http://bi.snu.ac.kr/

32. Perception-Action Cycle in Communication between A and B 2012 (c) SNU Biointelligence Lab, http://bi.snu.ac.kr/

33. Perception-Action Cycle in Language Comprehension 2012 (c) SNU Biointelligence Lab, http://bi.snu.ac.kr/

34. Perception-Action Cycle in Robots 2012 (c) SNU Biointelligence Lab, http://bi.snu.ac.kr/ [Zahediet al., Adaptive Behavior, 2009]

35. Perception-Action Cycle 2012 (c) SNU Biointelligence Lab, http://bi.snu.ac.kr/ [Zahediet al., Adaptive Behavior, 2009]

36. Predictive Information 2012 (c) SNU Biointelligence Lab, http://bi.snu.ac.kr/ [Zahediet al., Adaptive Behavior, 2009]

37. Sensory Prediction 2012 (c) SNU Biointelligence Lab, http://bi.snu.ac.kr/ [Zahediet al., Adaptive Behavior, 2009]

38. Free Energy and the Perception-Action Cycle 2012 (c) SNU Biointelligence Lab, http://bi.snu.ac.kr/ [Friston, Trends in Cognitive Sciences, 2009]

39. Reinforcement Learning and the Perception-Action Cycles = (information-to-go) – (value-to-go) 2012 (c) SNU Biointelligence Lab, http://bi.snu.ac.kr/ [Tishby & Polani, 2010]

40. Brain Mechanisms for the Perception-Action-Learning Cycle

41. Brain Computation: Speed, Flexibility, Robustness • How can brain computation be so fast, flexible, and robust in a changing environment? • Fast • Object recognition: within 100 ms • Anomaly detection: N400, P600 • Instant decision-making • Flexible • Invariant to shift, scale, and rotation • Various utterances for the same meaning • Art, music, literature, and dancing • Robust • Cluttered image • Noisy speech • Intention reading under complex situations • What brain mechanisms for information processing and organization allow this? 2012 (c) SNU Biointelligence Lab, http://bi.snu.ac.kr/

42. Language Processing in the Brain • N400: a brain wave related to linguistic processes. • Increased when semantically mismatched Fig. 9.30: ERP waveforms differentiate between congruent words at the end of sentences (work) and anomalous last words that do not fit the semantic specifications of the preceding context (socks). 2012 (c) SNU Biointelligence Lab, http://bi.snu.ac.kr/

43. Syntactic Processing in the Brain • LAN (left anterior negativity): negative wave over the left frontal areas when words violate the required word category in a sentence (syntactic violation) • e.g. “the red eats”, “he mow” Semantic Syntactic ERPs related to semantic and syntactic processing. 2012 (c) SNU Biointelligence Lab, http://bi.snu.ac.kr/

44. Brain as a Widely Distributed, Parallel, Interactive, Overlapping, Dynamic Relational Memory Network [Fuster, 2004] 2012 (c) SNU Biointelligence Lab, http://bi.snu.ac.kr/ [Fuster, 2004]

45. Neural Representations and Processing • “Chemical” and “molecular” basis of synapses • Distributed representation • Multiple overlapping representations • Hierarchical representation • Associative recall • Population coding • Assembly coding • Sparse coding • Temporal coding • Synfire chain • Dynamic coordination • Correlation coding 2012 (c) SNU Biointelligence Lab, http://bi.snu.ac.kr/

46. Bayesian Brain: Multisensory Integration 2012 (c) SNU Biointelligence Lab, http://bi.snu.ac.kr/ [Knill & Pouget, 2004]

47. Population Coding (Representation) Rate Coding Gain Coding 2012 (c) SNU Biointelligence Lab, http://bi.snu.ac.kr/

48. Probabilistic Inference with Population Codes [Knill and Pouget, Trends in Neurosciences, 2004] 2012 (c) SNU Biointelligence Lab, http://bi.snu.ac.kr/ [Knill and Pouget, Trends in Neurosciences, 2004]

49. Dynamics in Sensory Cue Integration 2012 (c) SNU Biointelligence Lab, http://bi.snu.ac.kr/ [Deneve et al., Nature Neuroscience, 2001, from Knill and Pouget, Trends in Neurosciences, 2004]

50. Models of Perception-Action-Learning Cycles