Decoding Speech with ECoG – Computational Challenges

1. Decoding Speech with ECoG – Computational Challenges Chris Holdgraf Helen Wills Neuroscience Institute, UC Berkeley

2. Challenge in neuroscience • Neuroscience is a very broad field. It covers everything from gene expression, to a single neuron firing, to activity across the whole brain in humans. • As such, one must have a wide range of knowledge and a diverse set of techniques. • Often makes it hard to have the best domain-specific knowledge.

3. Mapping the world onto the brain • The trick is to fit some function that links brain activity with the outside world. • However, we also want it to be a function that is scientifically meaningful.

4. Neuroscience/Psychology and computation • Historically, there has been a focus on tightly-controlled experiments and simple questions. • Advances in imaging and electrophysiological methods have increased the quality and quantity of data.

5. Electrocorticography – a blend of temporal and spatial resolution • ECoG involves the application of electrodes directly to the surface of the brain. • This avoid many problems with EEG, while retaining the rich temporal precision of the signal.

6. Complex and noisy data requires careful methods • ECoG is only possible in those with some sort of pathology. Moreover, recording time is short. • Data driven methods – bad data in = bad models out.

7. Merging ECoG and Computational Methods • Might be possible to leverage the spatial precision of ECoG to decode the nature of this processing.

8. Challenge 1: GLMs in Neuroscience

9. Computational Challenge #1 How to fit a model that is both interpretable and a good fit for the electrode’s response. • The parameter space is increasingly complex for more hypotheses. • Oftentimes, this is paired with a limited dataset. Especially in ECoG. • Regularization and Feature Selection become very important

10. Want it simple? Use a GLM! Linear models allow us to predict some output with a model that is both interpretable and (relatively) easy to fit.

11. One problem with this… • However, the brain assuredly does not vary linearly in response to inputs from the outside world.

12. Basis functions • Instead, we can decompose an input stimulus as a combination of “basis functions” • Basically, this entails a non-linear transformation of your stimulus, so that fitting linear models to brain activity makes more sense.

13. Exploring the brain through basis functions “dog” “hat” “car” “man”

14. Fitting weights with gradient descent • We can find the values for these weights by following the typical least-squares regression approach. • Early stopping must be tuned carefully in order to regularize. • Full gradient descent • Coordinate gradient descent • Threshold gradient descent

15. An application of the GLM for neural decoding

16. Neural Decoding • If you can map stimuli onto brain activity, then you could also map brain activity onto stimuli. • Same approach, but now our inputs are values from the electrodes, and the output is sound. • Implications in Neural Prostheses and Brain Computer Interfaces Speech Decoding

17. Decoding with a linear model ReconstructedSpectrogram Decoding Model High Gamma Neural Signal = X Original Spectrogram

18. Decoding Listened Speech High Gamma (60-200 Hz) Pasley et al. Plos Biology, 2012

19. Speech Reconstruction from ECoG

20. Challenge 2: From model output to language

21. Challenge #2 Turn a noisy, variable spectrogram reconstruction into linguistic output. • Simpler methods are often not powerful enough to account for these small variations • How to take advantage of temporal correlations between words / phonemes? • How to accomplish this without a ton of data?

22. How to classify this output? Town Doubt Property Pencil

23. From model output to language • Borrow ideas from the speech recognition literature. • Currently using Dynamic Time Warping to match output spectrograms to words.

24. Dynamic Time Warping • Computedissimilaritymatrix betweenevery pair of elements • Findthe optimal path in order to minimize the overallaccumulateddistance • Effectivelywarps and realigns the twosignals

25. Current output workflow Reconstructed Spectrogram DTW

26. Where to go from here?

27. Improving the decoder fit • Clever methods of dealing with finite and noisy datasets • Finding better features (basis functions) • Interactions between features • Fitting more complicated models • Interactions between features • Nonlinear models are useful for engineering, but require much more data

28. Turning output into reconstructed language • Leverage the spectro-temporal statistics of language • Focus on a classification rather than arbitrary decoding /w/ /ch/ /ks/ /g/

29. The “Big Data” Angle • Right now, the field of ECoG is in a bit of a transition period • Excitement around using computational methods, but many labs (including my own) don’t have the infrastructure and culture to tackle “big data” problems. • That said, we do have the potential to collect increasingly large datasets, once we know what to do with them.

30. The Long-Term Goal Create a modeling framework that allows us to use ECoG to decode linguistic information.

31. Fellow Decoders Special thanks Frederic Theunissenand co. Jack Gallant and co. STRFLab Team Brian Stéphanie Gerv Eddie Peter