Neural Prosthetics IV: Pushing preferred directions around

1. Neural Prosthetics IV:Pushing preferred directions around Beata Jarosiewicz Laboratory of Andrew Schwartz

2. The explanatory gap • Absolute gap: Why is neural activity accompanied by any conscious experience at all? • Comparative gap: Why does certain neural activity give rise to, say, visual rather than auditory experience? Or to the experience of red instead of green? ‘Red’ Neural activity resulting from 670 nm light 670 nm wavelength light

3. The explanatory gap • Absolute gap: Why is neural activity accompanied by any conscious experience at all? • Comparative gap: Why does certain neural activity give rise to, say, visual rather than auditory experience? Or to the experience of red instead of green? ‘Red’ absolute gap Neural activity resulting from 670 nm light 670 nm wavelength light

4. The explanatory gap • Absolute gap: Why is neural activity accompanied by any conscious experience at all? • Comparative gap: Why does certain neural activity give rise to, say, visual rather than auditory experience? Or to the experience of red instead of green? comparative gap ‘Red’ ‘Green’ absolute gap Neural activity resulting from 670 nm light Neural activity resulting from 530 nm light 670 nm wavelength light 530 nm wavelength light

5. Cortical “deference” vs. “dominance” from “Neural Plasticity and Consciousness” (Hurley & Noë, Biology & Philosophy, 2003)

6. Cross-modal remapping: “vOICe” Movie by Alex Storer, BU Left and Right Image is scanned left to right, one snapshot per second. So left and right get encoded by time (and stereo). Dark and Light Brightness is encoded by loudness: the brighter the visual pattern, the louder the sound. Up and Down Elevation is encoded by pitch: the higher the position of the visual pattern, the higher the pitch.

7. Cross-modal remapping: “vOICe” 2000: “Well the other day I was again washing dishes. I had let the water out of the sink and turn to get a towel to dry my hands. Then when I turn back to rinse the sink I was stunned to see the sink in a “depth” like image. I stepped away from the sink and walked slowly up to it again to see if my mind was playing tricks on me. No, the feeling of seeing depth in the sink bowl was still there.” 2002: “Just sound?.... No, It is by far more, it is sight! There IS true light perception generated by the vOICe. When I am not wearing the vOICe the light I perceive from a small slit in my left eye is a grey fog. When wearing the vOICe the image is light with all the little greys and blacks. Yet a definite light image.” - Pat Fletcher, long-term vOICe user

8. Cross-modal remapping: “vOICe” 2000: “Well the other day I was again washing dishes. I had let the water out of the sink and turn to get a towel to dry my hands. Then when I turn back to rinse the sink I was stunned to see the sink in a “depth” like image. I stepped away from the sink and walked slowly up to it again to see if my mind was playing tricks on me. No, the feeling of seeing depth in the sink bowl was still there.” 2002: “Just sound?.... No, It is by far more, it is sight! There IS true light perception generated by the vOICe. When I am not wearing the vOICe the light I perceive from a small slit in my left eye is a grey fog. When wearing the vOICe the image is light with all the little greys and blacks. Yet a definite light image.” - Pat Fletcher, long-term vOICe user

9. What caused the switch from cortical dominance to cortical deference? Maybe what’s important is how the information is interpreted downstream. Is there a way we can experimentally change how the activity of particular neurons is interpreted downstream, and test whether they can change to acquire their newly-assigned function?

10. What caused the switch from cortical dominance to cortical deference? Maybe what’s important is how the information is interpreted downstream. Is there a way we can experimentally change how the activity of particular neurons is interpreted downstream, and test whether they can change to acquire their newly-assigned function?

11. What caused the switch from cortical dominance to cortical deference? Maybe what’s important is how the information is interpreted downstream. Is there a way to experimentally manipulate the way neural activity is interpreted downstream, and test whether the neurons “oblige”?

12. There is a way, with brain control! • With the brain control paradigm, we know the precise relationship between the activity of each of the neurons under study and the “behavioral” output. • This allows for a new kind of perturbation that can target selected subsets of neurons by altering the way those neurons contribute to the object’s movement, and to test whether neurons can change their activity specifically and selectively when the monkey learns to compensate for the perturbation.

13. Cosine tuning with (intended) movement direction 3D center-out task under “brain control” Population Vector Algorithm

14. Experimental setup: 4 sessions • Adaptive: • Each cell’s tuning function (PD and modulation depth) was estimated by iteratively regressing firing rate against target direction as the monkey performed the brain control task. • Once the monkey’s performance stabilized, the tuning functions obtained from this iterative regression were frozen and used for decoding (“dPDs”). • 2) Control 1 (C1): • Monkey performed brain control using the decoding • parameters obtained from the adaptive session. • 3) Perturbation (P): • A subset (~25%) of the recorded units (the • “perturbed units”) were given reassigned dPDs • by rotating their original dPDs 90 degrees about • the x, y, or z axis (chosen randomly each day). • 4) Control 2 (C2): • The perturbation was removed (i.e. the original • dPDs were reinstated).

15. C1 Direction of perturbation Deflection in early trajectory Early P Late P Monkey learns to control cursor by end of Perturbation session

16. Possible neural bases • The perturbed units could have suppressed their contribution to the PV by firing at baseline rate everywhere; i.e. by decreasing their modulation depths. • The perturbed units could have shifted their actual PDs toward their reassigned dPDs. • The monkey could have “re-aimed” the cursor to offset the perturbation caused by the reassignment, disregarding the relative contributions of the perturbed vs. unperturbed units to the error. This would appear as a shift in all (perturbed and unperturbed) measured PDs in the direction of the perturbation.

17. Possible neural bases • The perturbed units could have suppressed their contribution to the PV by firing at baseline rate everywhere; i.e. by decreasing their modulation depths. • The perturbed units could have shifted their actual PDs toward their reassigned dPDs. • The monkey could have “re-aimed” the cursor to offset the perturbation caused by the reassignment, disregarding the relative contributions of the perturbed vs. unperturbed units to the error. This would appear as a shift in all (perturbed and unperturbed) measured PDs in the direction of the perturbation.

18. Target Deflection caused by perturbation Re-aiming direction Possible neural bases • The perturbed units could have suppressed their contribution to the PV by firing at baseline rate everywhere; i.e. by decreasing their modulation depths. • The perturbed units could have shifted their actual PDs toward their reassigned dPDs. • The monkey could have “re-aimed” the cursor to offset the perturbation caused by the reassignment, disregarding the relative contributions of the perturbed vs. unperturbed units to the error. This would appear as a shift in all (perturbed and unperturbed) measured PDs in the direction of the perturbation.

19. Testing the possible neural bases • To obtain each unit’s “measured” tuning function in each session, the same linear regression was done on all firing rates vs. target directions in that session (but post-hoc): • measured PD = direction for which its firing rate was the highest (direction at the peak of the cosine fit) • measured modulation depth = the magnitude of the fitted cosine (the difference between the peak firing rate and the baseline firing rate)

20. Modulation depths did not change

21. Target Deflection caused by perturbation Re-aiming direction Possible neural bases • The perturbed units could have suppressed their contribution to the PV by firing at baseline rate everywhere; i.e. by decreasing their modulation depths. • The perturbed units could have shifted their actual PDs toward their reassigned dPDs. • The monkey could have “re-aimed” the cursor to offset the perturbation caused by the reassignment, disregarding the relative contributions of the perturbed vs. unperturbed units to the error. This would appear as a shift in all (perturbed and unperturbed) measured PDs in the direction of the perturbation.

22. M M elevation Testing for shift in measured PDs MR MC1 0 R O 0 azimuth

23. C1 -> P P -> C2

24. “Dose-response”

25. Target Deflection caused by perturbation Re-aiming direction Possible neural bases • The perturbed units could have suppressed their contribution to the PV by firing at baseline rate everywhere; i.e. by decreasing their modulation depths. • The perturbed units could have shifted their actual PDs toward their reassigned dPDs. • The monkey could have “re-aimed” the cursor to offset the perturbation caused by the reassignment, disregarding the relative contributions of the perturbed vs. unperturbed units to the error. This would appear as a shift in all (perturbed and unperturbed) measured PDs in the direction of the perturbation.

26. Conclusions • This study demonstrates selective functional plasticity, even when only a global feedback signalis available. • How can the brain do that? Perhaps noise allows it to explore synaptic weight spaces that optimize cursor control? • We still don’t know where or how the changes took place that underlie the PD shifts in motor cortex. • Further studies using the brain control paradigm promise a deeper understanding of plasticity as a neural substrate of learning. • In combination with the dominance/deference studies, this result might have interesting implications for “neural coding”: perhaps it isn’t as important what “activates” a neuron (i.e. its “tuning function”) – as how that neuron’s activity is used downstream.

27. Thank you! • Rob Kass • Meel Velliste • George Fraser • Chance Spalding • Ingrid Albrecht • Nathaniel Daw • Valerie Ventura • NIH-NINDS-NO1-NS-2-2346

29. Introduction • Many studies have shown that neurons can change their “coding” properties in association with learning. • In motor cortex, neurons can change their tuning functions when monkeys adapt to perturbations that interfere with the execution (e.g. Bizzi and collaborators) or visual feedback (e.g. Wise, Paz, Vaadia) of their movements. • With the brain control paradigm, we know the precise relationship between the activity of each of the neurons under study and the “behavioral” output. • The brain control paradigm also allows for a new kind of perturbation that can target selected subsets of neurons by altering the way those neurons contributes to the object’s movement; this makes it possible to test whether neurons can change their activity specifically and selectively when the monkey learns to compensate for the perturbation.

30. Expected Actual Cursor deflection resulting from perturbation Rotation axes: red = Xgreen = Y blue = Z

34. Expected cursor perturbations rotation axis:X = redY = greenZ = blue

35. Estimating preferred directions: = firing rate at time t = baseline firing rate = cell’s preferred direction = (intended) movement velocity at time t = angle between the movement direction and the cell’s preferred direction The “Population Vector” algorithm: = normalized preferred direction of cell i = maximum firing rate of cell i

36. Re-aiming occurs later in trajectory