Andy Clark Cognitive Science Program Indiana University andy@indiana As of Fall 2004

1. Embodiment and Ecological Control Andy Clark Cognitive Science Program Indiana University andy@indiana.edu As of Fall 2004 Dept of Philosophy Edinburgh University Scotland

3. disembodied symbolic abstract rule-following clunky chunky high-level slow brittle…..etc

6. embodiment grounding dynamics perception-action systems real-world real-time embedded enactive……etc

7. Just how different ARE these two paradigms really? Just what is it about embodiment that (when taken seriously) really matters for the very shape of a science of the mind? What is the RELATION between those two apparently very different sets of target activities (between abstraction - rich reflective reason and more basic capacities for embodied action)? If embodiment matters, does it matter ‘all the way up’?

8. Does full-blown human cognition come about by gradually adding refinements (“bells and whistles”) to basic (embodied, embedded) strategies for adaptive success, or does ‘higher cognition’ depend on the use of fundamentally novel forms of internal representation and processing? = The Continuity Debate

9. Two Questions √ 2.Embodiment: What Really Matters? 3.Sensory Substitution 4.From There to Here (Continuity with a Twist) 5.Can the be a Science of Hybrids?

10. Two Questions √ 2.Embodiment: What Really Matters? 3.Adaptive Ecological Control 4.From There to Here (Continuity with a Twist) 5. Can there be a Science of Hybrids?

11. Mere Embodiment Modest Embodiment Profound Embodiment

13. For Shakey, the body and the environment were first and foremost problems to be solved. The environment was the problem arena. The sensors detected the lay-out in that arena. The reasoning system planned a solution. The body was just another problem, that then needed to be micro-managed so as to put the solution into practice.

14. Passive Dynamic Walkers (PDW’s) • (Andy Ruina Lab, Cornell: original work by Tad McGeer) • No actuation except gravity, and no control system, except for a mechanical knee. • Inner and outer legs are paired to constrain it from falling over sideways. • Surprisingly, PDW’s are capable (when set on a gentle incline) of very stable, human-looking walking.

16. Systematically pushing, damping and tweaking a system in which Passive Dynamic effects play a very major role • = a very simple example of • ecological control

17. Ecological Control • An ecological control system is one in which goals are not achieved by micro- managing every detail of the desired action or response, but by making the most of robust, reliable sources of relevant order in the bodily or worldly environment of the controller.

18. systems that are specifically designed so as to constantly search for opportunities to make the most of body and world, checking for what is available, and (at various time-scales and with varying degrees of difficulty) integrating it deeply, creating new functional wholes. i.e. a dynamically adaptive form of ecological control.

19. Any control system that is highly engineered so as to be able to learn to make maximal problem-simplifying use of an open-ended variety of internal, bodily, or external sources of order. = dynamically adaptive ecological control

20. “Ecological control” names an overall effect not a single mechanism. That effect (the achievement of delicate adaptive balances between environmental, neural and bodily dynamics) comes in many degrees and flavors, all the way from more-or-less hard-wired ecological balances to learnt-on-the-fly ecological balances.

21. Stelarc

22. The third hand is controlled by EMG signals detected by electrodes placed on four strategic muscle sites on Stelarc’s legs and abdomen. The third hand is controlled by Stelarc’s brain via muscle commands to these sites that are then relayed to the prosthesis. Since these sites are not normally used for hand control, the third hand can be moved independently of the other two

23. Stelarc simply feels as if he wills the third hand to move, just as he wills his biological hands to move. In each case, the control is fluent and intuitive. It seems to require no special effort or conscious focus.

24. Normally, you don’t feel as if you are (for example) USING your hand to do the washing up. Instead, you just feel as if YOU are washing up. Your hand functions as what some philosophers call TRANSPARENT EQUIPMENT : equipment through which you can act on the world without first willing an act on anything else. (see Heidegger (1927) on the ‘ready-to-hand’)

25. The same is true in the biological case. The human infant must learn, by trial and error and practice, which neural commands bring about which bodily effects,and must then practice until she is skilled enough to issue those commands without conscious effort (so the body becomes transparent equipment)

26. A monkey, with implanted electrodes monitoring brain activity, learns to control a joystick to move a cursor to get rewards. The monitoring computer learns what neural commands correspond to what joystick motions. Next, the joystick is disconnected. The monkey discovers, though, that it can still use its own neural commands ( as transmitted by the implanted electrodes and decoded by the monitoring computer ) to directly control the cursor. Finally, the commands are diverted to control a distant robot arm, whose motions are reflected in the on-screen cursor movements, thus closing the loop.

27. Picture from New Scientist webpage

28. “The integrative processes by which the brain adapts to control interaction are relatively agnostic concerning the source of the structure participating in the process...” • Christensen (In Press) • See also Rob Wilson on “wide computationalism’ and (especially) on ‘exploitative control’

29. Whereas modest embodiment treats the body as a fixed (though highly significant) resource, profound embodiment is characterized by constant learning and re-calibration. Biological forms of embodiment, unlike a lot of current work in robotics, all tend towards the ‘profound’ end of this spectrum, though we primates seem especially plastic and well-engineered for multiple embodiment and fluent tool -use.

30. Even the minds that, in the movie The Matrix, populate the Matrix ‘dream-world’ count as profoundly embodied, since those minds display the same adaptive ecological control abilities as our own. For example, a Matrixer could learn to fluently incorporate a Stelarc-style third-hand, or to use a thought-controlled robot arm. It is just that the physical dynamics of the new components would be held in place by the Machines’ computer simulation rather than worldly physics.

31. These kinds of minds are promiscuously body-and-world exploiting. They are forever testing and exploring the possibilities for incorporating new resources and structures deep into their problem-solving regimes. These systems are continuously re-negotiating their own limits, components, and (as we’ll see) data-stores and interfaces.

32. Mere Embodiment Body and world as problem arenas only. Modest Embodiment Body and world as parts of problem-solutions, exploited by ecological control systems, but playing fixed roles. Profound Embodiment Body and world as resources for dynamically adaptive ecological controllers, yielding (as we’ll see) a rather permeable and reconfigurable agent/world boundary.

33. Break for questions and comments Andy Winter

34. Two Questions √ 2.Embodiment: What Really Matters?√ 3.Sensory Substitution 4.From There to Here (Continuity with a Twist) 5.A Science of Hybrids?

35. Tactile Visual Sensory Substitution (TVSS) Work by Paul Bach y Rita and colleagues

36. STOP feeling the tickling on the back and START to report rough, quasi-visual experiences of looming objects etc. After a while, a ball thrown at the head causes instinctive and appropriate ducking. The causal chain is ‘deviant’: it runs via the systematic input to the back. But the nature of the information carried, and the way it supports the control of action, is distinctive of the visual modality.

37. “TVSS systems [have] been sufficient to perform complex perception and ‘eye’-hand co-ordination tasks. These have included face recognition, accurate judgment of speed and direction of a rolling ball with over 95% accuracy in batting the ball as it rolls over a table edge, and complex inspection-assembly tasks”. Bach-y-Rita 2001

38. The head-mounted camera was under the subject’s motor control. This meant that the brain could, in effect experiment via the motor system, giving commands that systematically varied the input, so as to begin to form hypotheses about what information the tactile signals might be carrying. For example, you hear someone approaching from the left, turn the camera that way, and see what tactile pattern corresponds to this event…

39. The motor system operating the camera could be changed, eg to a hand-held camera, with no loss of acuity. The touch pad, too, could be moved to new bodily sites. Also, there was no confusion: an itch scratched under the grid caused no ‘visual’ effects. (see Bach y Rita and Kercel “Sensory Substitution and the Human-Machine Interface” Trends in Cognitive Sciences 7:12:2003)

41. Tactile Flight Suit (US Navy) Jacket delivers small puffs of air controlled by complex sensors that determine if a plane or helicopter is tilting to the right or left or forward or backward. The pilot feels a puff-induced vibrating sensation on the side of the body corresponding to the direction of tilt, and can control the vehicle’s response by moving their body so as to cancel the puff/vibration.

43. The suit is so good at transmitting and delivering information in an intuitive way that it allows even inexperienced helicopter pilots to perform difficult tasks such as holding the helicopter in a stationary hover, while military pilots can use it to fly blindfold. The suit thus rapidly links the pilot to the aircraft in the same kind of closed loop interaction that linked Stelarc and the third hand, or the monkey and the robot arm, or the blind person and the TVSS system While wearing the suit, the helicopter behaves very much like an extended body/sensory sheath for the pilot.

44. What matters, in each case, is the provision of closed-loop signaling so that motor commands affect sensory input. What varies is the amount of training (and hence the extent of deeper neural changes) required to fully exploit the new agent-world circuits thus created.

45. “a conception of the senses in terms of Gibson’s (1966) perceptual systems (i.e. as a whole and complex system that is functionally constituted as one piece from beginning to end…) but [going] beyond in allowing for a conception of the senses as contingent modalities that are tributory of the overall perceptual system’s performance” González and Bach-y-Rita (ms)

46. The specific details of the circuitry by which the world is engaged become ‘transparent’ in normal use. The conscious agent is aware of the oncoming ball, not of seeing the ball, or (by the same token) of using a tactile substitution channel to detect the ball.

47. Profoundly Embodied Agents Integrated but negotiable platforms of sensing, moving and reasoning able to confront a larger and potentially hostile world. (aside: spatial contiguity of parts is not essential to such platforms, as long as the right sensorimotor feedback loops are in place).

48. Two Questions √ 2.Embodiment: What Really Matters?√ 3.Adaptive Ecological Control√ 4.From There to Here (Continuity with a Twist) 5.Can there be a Science of Hybrids?

49. External information-processing resources are also apt for recruitment and incorporation by processes of adaptive ecological control. As a result, we are not just bodily and sensorily but also cognitively permeable agents. Moreover, these cognitively permeable agents stack and iterate, as ‘ratchet effects’ build new cognizers out of old.

50. Dennett, Hutchins, Donald, Wilson, Vygotsky, Varela, Thompson Rosch, Bruner, Norman, Heidegger, Gregory, Gibson, Merleau-Ponty Bateson ……just fill in your favorites….