A Mechanism for Learning, Attention Switching, and Cognition

1. A Mechanism for Learning, Attention Switching, and Cognition Janusz Starzyk School of Electrical Engineering and Computer Science, Ohio University, USA http://people.ohio.edu/starzykj Cathedral of Applied Information Systems University of Information Technology and Management Poland Dagstuhl Seminar, March 27- April 1, 2011.

2. Motivated Learning • Definition: Motivated learning (ML) is pain based motivation, goal creation and learning in embodied agent. • ML applies to EI working in a hostile environment. • Machine creates abstract goals based on the pain signals. • It receives internal rewards for satisfying its goals • Various pains, internal, and external signals compete for attention. • Attention switching results from competition. • Cognitive perception is aided by attention switching.

3. Reinforcement Learning Motivated Learning External rewards Predictable Objectives set by designer Maximizes the reward Potentially unstable Learning effort increases with complexity Always active Internal rewards Unpredictable Sets its own objectives Solves minimax problem Always stable Learns better in complex environment than RL Acts when needed http://www.bradfordvts.co.uk/images/goal.jpg

4. faucet refill garbage sit on water w. can tank open - + Dual pain Pain Primitive level Dry soil Primitive Goal Creation • Reinforcing a proper action

5. Motor pathway (action, reaction) Sensory pathway (perception, sense) tank refill Level III - + faucet open Level II - + w. can water Level I - + Activation Stimulation Primitive Level Dry soil Inhibition Reinforcement Difference Need Expectation Abstract Goal Hierarchy • Abstract goals are created to reduce abstract pains and to satisfy the primitive goals • A hierarchy of abstract goalsis created to satisfy the lower level goals

6. Primitive needs Water Reservoir Abstract Needs Wash in Water Drink Water Irrigate Dirty Thirsty Drought Primitive Needs

7. Abstract needs Well Public Money Spend Money to Buy Draw own Water Spend Money to Build Water Reservoir Abstract Needs Wash in Water Drink Water Irrigate Dirty Thirsty Drought Primitive Needs

8. Abstract needs Tourists' Attractions Ground Water Well Building Wealthy Taxpayers Build Ecotourism Dig a Well Rise Taxes Water Supply Well Public Money Build Water Recreation Spend Money to Buy Draw own Water Spend Money to Build Water Reservoir Abstract Needs Wash in Water Drink Water Irrigate Dirty Thirsty Drought Primitive Needs

9. Abstract needs Management Planning Policy Resource Management and Planning Regulate Use Receive Salary Employment Opportunities Develop Infrastructure Tourists' Attractions Ground Water Well Building Wealthy Taxpayers Build Ecotourism Dig a Well Rise Taxes Water Supply Well Public Money Build Water Recreation Spend Money to Buy Draw own Water Spend Money to Build Water Reservoir Abstract Needs Wash in Water Drink Water Irrigate Dirty Thirsty Drought Primitive Needs

10. ML vs. RL agentsinhierarchical environments • 6 levels of hierarchy • Initially ML agent experiences similar primitive pain signal Pp as RL agent. • ML agent converges quickly to a stable performance. • 10 levels of hierarchy • Initially RL agent experiences lower primitive pain signal Pp than ML agent. • RL agent’s pain increases when environment is more hostile. J.A. Starzyk, P. Raif, and A.-H. Tan, “Mental Development and Representation Building through Motivated Learning” , WCCI 2010 - Special Session on Mental Architecture and Representation, Barcelona, Spain, July 18-23, 2010.

11. Grid world problem Four kinds of resources distributed over 25 x 25 grid. P. Raif, J.A. Starzyk, Motivated Learning In Autonomous Systems, submitted to IJCNN2011 - Special Session on Autonomous learning of object representation and control, San Jose, CA, July 31-Aug. 5, 2011.

12. Consciousness • Intelligence • Central executive • Attention and attentionswitching • Mental saccades • Cognitive perception • Cognitive action control Photo: http://eduspaces.net/csessums/weblog/11712.html

13. Computational Model of Conscious Machine Inspiration: human brain Photo (brain): http://www.scholarpedia.org/article/Neuronal_correlates_of_consciousness Central Executive Episodic Memory & Learning Attention switching Action monitoring Emotions, rewards, and sub-cortical processing Planning and thinking Queuing and organization of episodes Motivation and goal processor Episodic memory Sensory-motor Semantic memory Motor skills Motor processors Sensory processors Data encoders/ decoders Data encoders/ decoders Motor units Sensory units

14. Central Executive Central Executive Attention switching Action monitoring • Tasks • cognitive perception • attention • attention switching • motivation • goal creation and selection • thoughts • planning • learning, etc. Planning and thinking Motivation and goal processor

15. Central Executive Central Executive Attention switching Action monitoring • Interacts with other units for • performing its tasks • gathering data • giving directions to other units • No clearly identified decision center • Decisions are influenced by • competing signals representing motivations, pains, desires, plans, and interrupt signals • need not be cognitive or consciously realized • competition can be interrupted by attention switching signal Planning and thinking Motivation and goal processor

16. Attention Switching ! • Dynamic process resulting from competition between • representations related to motivations • sensory inputs • internal thoughts including • spurious signals (like noise). blog.gigoo.org/.../

17. Visual Saccades What Where C C D D A A B B C D A Input image B

18. Mental Saccades • Selected part of the image resulting from an eye saccade. • Perceived input activates object recognition and associated areas of semantic and episodic memory. • This in turn activates memory traces in the global workspace area that will be used for mental searches (mental saccades).

19. Mental saccades in a conscious machine Advancement of a goal? Continue search? Plan action? Action? No No Advancement Advancement Attention spotlight Attention spotlight of a goal? of a goal? Yes Yes Loop 1 Loop 1 Mental saccades Mental saccades Learning Learning Yes Yes Continue Continue Changing motivation Changing motivation search? search? No No Loop 3 Loop 3 Associative memory Associative memory Plan action? Plan action? Yes Yes Loop 2 Loop 2 No No No No Action? Action? Changing perception Changing perception Perceptual saccades Perceptual saccades Loop 4 Loop 4 Yes Yes Action control Action control Changing environment Changing environment Loop 5 Loop 5

20. Action and subgoal planning Environment Perform action Decide action Learning Next mental saccade Pain reduction Intended action Dual pain Pain Attention spotlight Memory Induced pain Desired item Perception

21. Action control Action? Intended action Predicted changes Predicted changes known Pain increase Cognitive action control Cognitive abort Associative memory Lower level action control Lower level action control Lower level action control

22. A Mechanism for Learning, Attention Switching, and Cognition: Summary • A mechanism of switching attention is fundamental for building cognitive machines. • Attention switching is a dynamic process resulting from competition between goals, representations, sensory inputs, and internal thoughts. • Motivated learning provides a mechanism for creation of abstract goals and continuous goal oriented motivation • Mental saccades of the working memory are fundamental for cognitive thinking, attention switching, planning, and action monitoring http://www.inspirationfalls.com/the-key-to-success-quotes/key-to-all-success-concepts-1/

23. A Mechanism for Learning, Attention Switching, and Cognition: Summary • Motivations for actions are physically distributed • competing pain (need) signals are generated in various parts of machine’s mind • Before a winner is selected, machine does not interpret the meaning of the competing signals • Cognitive processing is predominantly sequential • winner of the internal competition is an instantaneous director of the cognitive thought process • Top down supervision of perception, planning, internal thought or motor functions • results in conscious experience • decision of what is observed and where is it • planning how to respond • a train of such experiences constitutes consciousness

24. Conclusions • Consciousness is computational • Motivated intelligent machines can be conscious

25. Questions ?? Photo: http://bajan.wordpress.com/2010/03/03/dont-blame-life-blame-the-way-how-you-live-it/

26. P.A.O. Haikonen, “The cognitive approach to conscious • machines”. UK: Imprint Academic, 2003. • J. Bach, “Principles of Synthetic Intelligence PSI: An Architecture of Motivated Cognition”, Oxford Univ. Press, 2009. • B. J. Baars “A cognitive theory of consciousness,” Cambridge Univ. Press, 1998. • A. Sloman, "Developing concept of consciousness," Behavioral and Brain Sciences, vol. 14 (4), pp. 694-695, Dec 1991. • J. Schmidhuber, “Curious model-building control systems,” Proceedings Int. Joint Conf. Neural Networks, Singapore, vol. 2, pp. 1458–1463, 1991. • B. Bakker and J. Schmidhuber, “Hierarchical Reinforcement Learning with Subpolicies Specializing for Learned Subgoals,” in Proc. of the 2nd Int. Conf. on Neural Networks & Computational Intelligence, Switzerland, pp.125-130, 2004. • A. Barto, S. Singh, and N. Chentanez, “Intrinsically motivated learning of hierarchical collections of skills, Proc. 3rd Int. Conf. Development Learn., San Diego, CA, pp. 112–119, 2004. • J. A. Starzyk, "Motivation in Embodied Intelligence"  in Frontiers in Robotics, Automation and Control, Oct. 2008, pp. 83-110. • J.A. Starzyk, “Motivated Learning for Computational Intelligence,” in Computational Modeling and Simulation of Intellect. ed. B. Igelnik, IGI Publ, 2011. References Photo: http://s121.photobucket.com/albums/o209/TiTekty/?action=view&current=hist_sci_image1.jpg

27. Embodied Intelligence Mechanism: biological, mechanical or virtual agent with embodied sensors and actuators EI acts on environment and perceives its actions Environment hostility is persistent and stimulates EI to act Hostility: direct aggression, pain, scarce resources, etc EI learns so it must have associative self-organizing memory Knowledge is acquired by EI Definition • Embodied Intelligence (EI) is a mechanism that learns how to minimize hostility of its environment

28. Embodiment of a Mind • Embodiment is a part of the environment that EI controls to interact with the rest of the environment • It contains intelligence core and sensory motor interfaces under its control • Necessary for development of intelligence • Not necessarily constant or in the form of a physical body • Boundary transforms modifying brain’s self-determination