Encoding & Decoding of Neuronal Ensembles in Motor Cortex Nicholas Hatsopoulos Dept. of Organismal Biology & An

1. Encoding & Decoding of Neuronal Ensembles in Motor Cortex Nicholas Hatsopoulos Dept. of Organismal Biology & Anatomy Committees on Computational Neuroscience & Neurobiology University of Chicago Co-founder & board member of Cyberkinetics, Inc.

2. Encoding Problem trial 1 trial 2 trial 3 trial 4 trial 5 Multi-trial averaging Behavior Decoding Problem Single-trial prediction

3. “The motor cortex appears to be par excellence a synthetic organ for motor acts… the motor cortex seems to possess, or to be in touch with, the small localized movements as separable units, and to supply great numbers of connecting processes between these, so as to associate them together in extremely varied combinations. The acquirement of skilled movements, though certainly a process involving far wider areas (cf. V. Monakow) of the cortex than the excitable zone itself, may be presumed to find in the motor cortex an organ whose synthetic properties are part of the physiological basis that renders that acquirement possible.” Leyton & Sherrington (1917)

4. The two components of language • Words or elementary primitives of meaning • Rules or grammar by which the primitives • are combined Pinker (1999)

5. The language of motor action • in motor cortex • Motor primitives: position, velocity, • direction, trajectory • Motor grammar: addition

6. Center-Out Task

7. 2 0 4 0 Directional Tuning 9 0 ° 1 3 5 ° 4 5 ° 0 ° 180° 3 1 5 ° 2 2 5 ° 2 7 0 ° 150 frequency (Hz) 0 0 1.0 -0.5 time (s)

8. Behavioral Apparatus

9. Random-walk task A B

10. Chronic Multi-electrode array Utah/Bionic Technologies Probe Richard Normann U Utah

11. Leg Arm Face Primary Motor Cortex (MI) MI 5 mm

13. Long-term Reliability & Stability 0.8 0.7 0.6 0.5 ARRAY YIELD 0.4 0.3 subject 1 (tom) 0.2 subject 2 (coco) 0.1 subject 3 (buddy) subject 4 (radley) 0 0 100 200 300 400 500 600 700 800 900 1000 Many Neurons Every Day (19 tests over 110 days) Blue - no recording Red - best recordings DAYS POSTIMPLANT

14. Why view motor cortical encoding as time-dependent? • Trajectory-selective activity in motor cortex • (Hocherman & Wise, 1990, 1991) • Preferred directions shift in time (Mason et al., 1998; • Sergio et al., 2005; Sergio & Kalaska, 1998)

15. Center-out task Shifts versus time

16. Center-out task Random-walk task Shifts versus time Shifts versus lead/lag time lag lead movement onset

18. Temporal tuning (information theory) Lead/lag (s)

19. = preferred velocity trajectory The Encoding Model A class of general linear models (e.g. logistic regression) that estimates the probability of a spike given a particular movement trajectory: integrated = preferred trajectory and path (“pathlet”)

21. model input = probability of a spike model input lead/lag (ms)

22. Temporal stability of pathlet representation

23. ROC analysis to quantify the performance of the encoding model

24. Encoding Performance as a function of trajectory length

25. Decoding Performance as a function of trajectory length

26. Map of Pathlets

28. Huntley & Jones (1991) 1 mm Horizontal connectivity in motor cortex

29. Rule for combining pathlets: Additive rule Assuming conditional independence,

33. Potential violations of conditional independence neuron 40 vs. neuron 41 neuron 39 vs. neuron 51 100 260 220 80 180 60 140 Counts/bin 100 40 60 20 20 -0.4 -0.2 0 0.2 0.4 -0.4 -0.2 0 0.2 0.4 Case #1 Case #2 1 ms bin

34. Spike Jitter Method -J +J 2 parameters: +/-w w, time resolution of synchrony J, the jitter window neuron 1 (reference) neuron 2 (target) 2 spike jittered trains

35. Case #1 1000 jitters Case #2 10% of all cell pairs (N=1431) show significant synchrony at a resolution of +/-5 ms, p<0.05

36. Potential violations of conditional independence Case #1 Case #2

38. When conditional independence appears to be violated Conclusion: Synchronization preserves additive rule but increases the sensitivity of the tuning function by increasing the gain

39. Neuro-motor prosthetic system 1) Multi-electrode array implant 2) Decoding of neural signals 3) Output interface Computer Personal computer: • mouse • keyboard Assistive Robotics: • robotic arm • mechanized prosthetic arm Biological interface: • muscles • peripheral nerve • spinal cord

40. BrainGateTM Pilot Device Cable Cart Sensor

41. Array on Cortex 2 months post implant BrainGateTM Sensor Implantation and Post-Op Recovery as Planned • Surgery as planned • Post-op recovery unremarkable • Wound healing around pedestal complete Insertion

42. Binary Modulation-Imagined Opening/Closing of Hand

43. BrainGate Signal Detection and Analysis;SCI Able to Modulate Neural Output

44. ( ) - 1 ˆ = T T = f R R R X X(t) Rf 2. Algorithmic level: Optimal Linear Filter Reconstruction Response of neural ensemble in time Estimated position of hand in time filter coefficients Warland et al. (1997)

45. Two Dimensional Cursor control

47. Hatsopoulos Lab Qingqing Xu Wei Wu, PhD Sunday Francis Zach Haga Jignesh Joshi John O’Leary Dawn Paulsen Jake Reimer Jonathan Ko Joana Pellerano Richard Penn, MD Matt Fellows Yali Amit Cyberkinetics, Inc.

48. Cross-Validated Performance as a function of pathlet length

49. Cross-validated Pathlet Population Vector Decoding