Hidden Process Models: Decoding Overlapping Cognitive States with Unknown Timing

1. Hidden Process Models:Decoding Overlapping Cognitive States with Unknown Timing Rebecca A. Hutchinson Tom M. Mitchell Carnegie Mellon University NIPS Workshops: New Directions on Decoding Mental States from fMRI Data December 8, 2006

2. Overview • Open questions we address: • Treating fMRI as the time series that it is. • Allowing the testing of hypotheses. • Open questions we do NOT address: • Interpretability of time series or spatial representation of activity. • This talk • Motivation • HPMs (in 1 slide!) • Preliminary results

3. Motivation • Goal: connect fMRI to cognitive modeling. • Cognitive Model: • Set of cognitive processes hypothesized to occur during a given fMRI experiment. • Cognitive Process: • Spatial-temporal hemodynamic response function. • Timing distribution relative to experiment landmarks (like stimulus presentations and behavioral data).

4. Study: Pictures and Sentences Press Button View Picture Read Sentence • Task: Decide whether sentence describes picture correctly, indicate with button press. • 13 normal subjects, 40 trials per subject. • Sentences and pictures describe 3 symbols: *, +, and $, using ‘above’, ‘below’, ‘not above’, ‘not below’. • Images are acquired every 0.5 seconds. Read Sentence Fixation View Picture Rest t=0 4 sec. 8 sec.

5. One Cognitive Model Press Button View Picture Read Sentence • ViewPicture • begins when picture stimulus is presented • ReadSentence • begins when sentence stimulus is presented • Decide • begins within 4 seconds of 2nd stimulus Read Sentence Fixation View Picture Rest t=0 4 sec. 8 sec. ViewPicture or ReadSentence ViewPicture or ReadSentence Decide

7. ViewPicture in Visual Cortex

8. ReadSentence in Visual Cortex

9. ViewPicture

10. ReadSentence

11. Seconds following the second stimulus Multinomial probabilities on these time points Decide

12. Comparing Models 5-fold cross-validation, 1 subject P = ViewPicture S = ReadSentence S+ = ReadAffirmativeSentence S- = ReadNegatedSentence D = Decide D+ = DecideAfterAffirmative D- = DecideAfterNegated Dy = DecideYes Dn = DecideNo Dc = DecideConfusion B = Button ** - This HPM can also classify Dy vs. Dn with 92.0% accuracy. GNBC gets 53.9%. (using the window from the second stimulus to the end of the trial)

13. Conclusions • Simultaneous estimation of spatial-temporal signature (HRF) and temporal onset of cognitive processes. • Framework for principled comparison of different cognitive models in terms of real data.