Unveiling the Mind: Bridging Reinforcement Learning, Behaviourism, and Neurology

1. Reinforcement learning and behaviourism NeuroMod Institute 1st meeting Fréjus, 1-2 July 2019

2. The mind, the brain and the machine workings of the brain Neuro workings of the machine Mod workings of the mind Cog

3. The mind, the brain and the machine workings of the brain whatsomethought: connectionism workings of the machine whatsomethought: expert systems workings of the mind

4. The mind, the brain and the machine workings of the brain the integration challenge workings of the machine Bermúdez, José Luis 2014. Cognitive Science. An Introduction to the Science of the Mind. Cambridge: CUP. workings of the mind Schrödinger's cat in physics: how to relate the very small (probabilistic) and the very big (deterministic)?

5. The mind, the brain and the machine Reinforcementlearning machine (maybe brain?) thatwastried out some time ago Behaviourism mind

6. The mind and the machine Every time I fire a linguist, the performance of our speech recognition system goes up. 1988 He was not a pioneer of speech recognition, he was THE pioneer of speech recognition. says Steven Young from the Signal Processing Society in 2010 Fred Jelinek (1932 -2010) IBM Research, then Johns Hopkins

7. The mind and the machine • linguists are supposed to have an idea of how languageworks. • ...of how itworksin the mind. • so do we or don'twewant machines thatmimic the workings of naturalsystems? • is the implementation of naturalworkings in a machine enhancing or impedingits performance? • was Jelinek right back in 1988 because the technologywas not ready for the implementation of naturalsystems? Is itreadytoday? One day? Doesthatdepend on technology at all?

8. The mind and the brain • talking about the mind... • materialobjects have non-materialproperties: e.g. the center of gravity • the mindis a non-materialproperty of the brain • if the center of gravityis real, sois the mind. • the mindis the subjectmatter of Cognitive Science

9. The brain and the machine Computation in the mind should be brain-style. 1989 • the connectionist promise: • makeyourmind look likeyourbrain • makeyour machine look likeyourbrain • and felicitywill come uponyou. The operations in our models then can best be characterized as 'neurally inspired.' David Rumelhart (1942-2011) psychologist Rumelhart, David 1989. The Architecture of Mind: A Connectionist Approach. Foundations of Cognitive Science, edited by Michael Posner, 133-159. Cam., Mass.: MIT Press.

10. The brain and the machine • to whichextentexactly, if any, has the connectionist promise come true? • are current neural networks successfullymimicking the brain? • what does brain-style computation tell us about the (human) cognitive system? • is striving for functional efficiency and performance corrupting the ambition for bio-inspiration?

11. The brain and the machine • withoutknowinganything of the internalworkings of current machines, threeexternal observations show thatthisiscertainly not how humanswork: • energy • exposureneeded for successfullearning • errortolerance

12. reinforcementlearningwastried out in the Humanities: Behaviourism

13. Behaviourism • dominant theory in psychology in the first half of the 20th century. • bornfrom the wish to do somethingelsethan introspection and speculation. • if psychologyis to become a true science itneeds to bebased on data and measurements. John B. Watson 1878-1958 Watson, John B. 1913. Psychology as the Behaviorist Views It. Psychological Review 20: 158-177. • wecannotmeasureanythingthatisgoing on inside the skull, but wecanmeasurewhatgoes in and whatcomes out.

14. Workings of behaviourism • whathumans do (behaviour) is a function of twothings: • the data thatreachtheirperceptual system – stimulus • the feedback they get upon their response • feedback: negative (punishment) or positive (reinforcement) • otherfactors: mimicking and repetition (habit) • theseworkings are shared by humans and animals

15. Workings of behaviourism Ivan Pavlov 1849-1936 Pavlov, I. 1903.The Experimental Psychology and Psychopathology of Animals.14th International Medical Congress, Madrid. • applications • politics, teaching, management, advertising • behaviouraltherapy: autism, drug abuse, phobias, PTSD etc.

16. Radical Behaviourism • not only observable behaviour (Pavlov's dog) is due to the stimulus-response-feedback loop: • the samegoes for "internal" facultiessuch as knowledge, feelings, learning. • human and animal behaviouris due to the stimulus-response-feedback loopand to nothingelse. B. F. Skinner 1904-1990 Skinner, Frederic 1957. Verbal Behaviour. Acton, MA: Copley. • a version of empiricism: • anti-innatism: Locke'sblankslate • anythingthatis in yourbrainwas first in yoursenses • example: gift does not exist. The onlyreasonwhysomebodyis good at playing the piano isexercise and the s-r-f loop.

17. Radical Behaviourism General learningtheory • thereisonly one mechanismthatunderlieslearning in living beings: the stimulus-response-feedback loop. • thisgoes for • playing the piano • filling in a taxdeclaration • piloting an airplane B. F. Skinner 1904-1990 Skinner, Frederic 1957. Verbal Behaviour. Acton, MA: Copley. • as much as for • learning how to walk (infants) • learning how to speak (infants) • learning how to hear, feel, taste etc.

18. Radical Behaviourism Monism Empiricism Dualism Rationalism there are two distinct sources of knowledge thereisonly one source of knowledge • sensory input • stimulus-response-feedback loop • mimicking, habit • propertiesthatowenothing to and are entirelyindependent of sensory input • origin: heritage (innate)

19. Why behaviourism declined • behaviourismdeclinedfrom the 1950s on and todayis more or lesshistory (but there are crypto- and neo-behaviourists, sometimesunconscious of thatfact) • reason #1 • the cognitive revolution of the 50s-60s: emerging Cognitive Science • in science youwant to unpack the black box.

20. Why behaviourism declined Chomsky, Noam 1959. Review of Skinner's Verbal Behavior. Language 35: 26-58. Skinner, B. F. 1957. Verbal Behaviour. Acton, MA: Copley. • reason #2 • the review of Skinner's book Verbal Behaviour by the linguistNoam Chomsky

21. Why behaviourism declined • artefacts (filling in a taxdeclaration) and naturalproperties (walking) are different in kind. • learningmechanisms are different • learning how to speak = learning how to walk and eat.

22. Universal Grammar (UG) • Chomsky concludes that there is something invariable and independent of the environment that contributes to language acquisition: Universal Grammar (UG). • UG is invariable in humans and genetically coded. • example of a universal, environment-independent property of all languages of the world • word order is used to mark a number of things, e.g. interrogation: elle vient vs. vient-elle ? • but there is no language where word order is used for negation: elle vient "she comes" vs. vient-elle "she does not come". • there is no reason for this arbitrary selection of what can or cannot be expressed by word order.

23. Conclusion • Mind • reinforcement (the stimulus-response-feedback loop) is part of the workings of learning • but not enough for some items to belearnt: theyalsorequireenvironment-independent, invariable rules. • Brain • shouldbe the same, i.e. a mechanismsupporting the srf-loop, combined in certain cases withsomeothermechanism • Machine • do youwantyour machine to worklike the mind / brain? • thenmaybereinforcementwon't do the trick alone.

24. Thankyou for your attention