1 / 38

Authors: Paul Sajda , Adam D. Gerson , Marios G. Philiastides

Single-Trail Analysis of EEG during Rapid Visual Discrimination: Enabling Cortically Coupled Computer Vision. Authors: Paul Sajda , Adam D. Gerson , Marios G. Philiastides Dept. of Biomedical Engineering, Columbia University, USA Lucas Parra Dept. of Biomedical Engineering

psyche
Download Presentation

Authors: Paul Sajda , Adam D. Gerson , Marios G. Philiastides

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Single-Trail Analysis of EEG during Rapid Visual Discrimination: Enabling Cortically Coupled Computer Vision Authors: Paul Sajda, Adam D. Gerson, Marios G. Philiastides Dept. of Biomedical Engineering, Columbia University, USA Lucas Parra Dept. of Biomedical Engineering City College of New Work USA

  2. Outlines • Introduction • Linear Methods for Single-Trail Analysis • EEG Correlates of Perceptual Decision Making • Identifying Cortical Process leading to Response Time Variability • EEG-Based Image Triage • Conclusion

  3. Introduction • Linear discrimination of multichannel electroencephalography (EEG) has used for single-trail detection of neural signatures of visual recognition events. • Demonstrate: relating neural variability to response variability • Studies for response accuracy • Response latency during visual target detection • Example: Running in the park with your head phones on, listening to your favorite tune, & concentrating on your stride, you look up & see a face that you immediately recognize as a high school friend.

  4. Introduction (Con..) • She is wearing a hat, glasses & has aged fifteen years since you last saw her. • You & she are running in opposite directions so you only see her for a fleeting moment, yet you are sure it was her. • Your visual system has just effortlessly accomplished a feat that has thus far baffled best computer vision systems. • Such ability for rapid processing of visual information is even more impressive in light of the fact that neurons are relatively slow processing elements to digital computers, where individual transistors can switch a million times faster than a neuron can spike.

  5. Introduction (Con..) • Noninvasive neuroimaging has provided a means to peer into brain during rapid visual object recognition. • Analysis of trial-averaged event-related potentials in EEG has enabled us to assess speed of visual recognition & discrimination in terms of timing of underlying neural process. • Recent work has used single-trial analysis of EEG to characterize neural activity directly correlated with behavioral variability during tasks involving rapid visual discrimination. • Components extracted from EEG can capture neural correlates of visual recognition & decision-making processes on a trail-by-trail basis.

  6. Introduction (con..) • Noninvasive BCI paradigms: • Having a subject consciously modulate brain rhythms • Having a subject consciously generate a motor plan/visual imagery • Directly modulating subjects cortical activity by stimulus frequency (SSVEP) • Exploiting specific ERPs (novelty/oddball P300) • Focus: Single trail detection of ERPs & relationship to visual discrimination/recognition • Cortically coupled computer vision: visual processor performs perception & recognition • EEG interface detects result (decision).

  7. Linear Methods for Single-Trail Analysis: LDA • Conventional paradigm of evoked response considers neuronal activity following presentation of a stimulus. • Analyzing EEG activity of multiple electrodes following presentation of an image. • Aim: identify one type of event, visual target recognition, & Differentiate from other • Binary classification based on temporal & spatial profile of potential evoked following stimulus presentation.

  8. Linear Methods for Single-Trail Analysis (Con..) • EEG activity following each stimulus is recorded as DxT values. • D: # of channels • T: # of samples • Record: 1000 Hz, 64 channels • ½ sec time window: one word acquire = 32000 samples • Train classifier: Larger feature vector than N = 100 trials • Low SNR • Brute-force classification fail (32000-dim feature vector)

  9. Linear Methods for Single-Trail Analysis (Con..) • Exploit prior info: • S1: Reduce trial-to-trail variability by filtering (60 Hz ) & slow drifts (<0.5 Hz), no info carry • S2: Reduce dimensionality by grouping signal into L sample blocks & does not change within time window. • S3: Increase # of trails by L redundant samples/window • L=50 signal of interest at 10 Hz>> signal: noise • Transform original data for each trail with TD dimensions into L trails: TD/L • L=50, N=100, # training exam=5000 • train classifier with 640-dimensional feature vector at 10 Hz

  10. Linear Methods for Single-Trail Analysis: LDA (Con..) • linear classifier: TD/L dimension, good results • Single training window: L=T, D-dimensional feature vector • Linear Classification: • Feature vector x is projected onto an orientation defined by vector w • Projection, y = wTx, optimally differentiates bet’n two classes.

  11. Linear Methods for Single-Trail Analysis (Con..) • Adv: • linear voltage combination-immediate interpretation (current) • Coefficients that coupled current with observed voltages given for linear model by a=<xTy>/<y2> • Angular bracket indicate average over trials & samples. • Coefficients a describe coupling (& correlation) of discrimination component y with sensor activity x. • a & x: D-dim vector(row/column) • Strong coupling: low attenuation & visualized intensity map-sensor projections • Easy to implement • Fast • Permit real-time operation • Disadv: • does not capture synchronized activity >10 Hz • Neither does it capture activity that is not a fixed distance in time from stimulus • Only phase-locked activity detection

  12. EEG Correlates of Perceptual Decision Making • Single-trail LDA to identify cortical correlates of decision-making during rapid discrimination of images. • Experiment: Psychophysical performance is measured for several subjects during RSVP task • A series of target (faces) and non-target (cars) trials presented in rapid succession (Fig. 25.2a) • Simultaneously recording neuronal activity from 64-channel EEG electrode array. • Stimulus evidence varied with phase coherence (Fig. 25.2b)

  13. EEG Correlates of Perceptual Decision Making (Con..) • Within trail block: face & car images over a range of phase coherence presented random order. • 12 face & 12 car, grayscale images (512x512, 8 bits/pixel) • Equal # of front & side views • Equal spatial frequency, luminance & contrast • Subjects are required to discriminate • type of image (face/car) • report decision by pressing a button.

  14. EEG Correlates of Perceptual Decision Making (Con..) • EEG data acquired in Electrosatically shielded room • Used: Sensorium EPA-6 Elecrophysiological Amp. from 60 Ag/Agcl scalp electrodes, 3 periocular electrodes placed below left eye, left & right outer canthi • Sample rate: 1000Hz, 0.01-300 Hz passband, 12 dB/octave HPF & 8th order elliptic LPF. • Software-based 0.5 Hz HPF (remove dc drifts) • 60/120 Hz notch filter (minimize line noise) • Record: EOG signals, remove motion & blink • Motor response & stimulus events recorded on separate channels

  15. EEG Correlates of Perceptual Decision Making (Con..) • Identify EEG components that maximally discriminate bet’n 2 conditions • Identify 2 time window for discrimination • Identify early (170 ms following stimulus) & late component (>300 ms) • LD trained by integrating data across both time windows (2D-feature vector) • Performance improved (higher Az) • Fig. 25.3: comparison of psychometric & neurometric functions for 1 sub in dataset. • All subjects a single function can fit behavioral & neuronal datasets & 2 sep functions.

  16. EEG Correlates of Perceptual Decision Making (Con..) • Data analyzed both stimulus & response-locked conditions • Both face selective components appear to be more correlated with onset of visual stimulation than response for one subject (Fig. 25.4) • Discriminate activity significant down to 30% phase coherence (early & late component) • Randomize Az: significant level of p<0.01. • Results: neural correlates of perceptual decision making identified using high-spatial density EEG & corresponding component activities

  17. EEG Correlates of Perceptual Decision Making (Con..)

  18. Identifying Cortical Process leading to Response Time Variability • Significant variability in response time observed across trials in many visual discrimination & recognition. • Factors: difficulty in discriminating an object on any given trial • Trial-by-trial variability of subject’s engagement • Intrinsic variability of neural processing • Identifying neural activity correlated with response time variability may shed light on the underlying cortical networks responsible • for perceptual decision-making processes & processing latencies that these networks may introduce for a given task

  19. Identifying Cortical Process leading to Response Time Variability (Con..) • Visual target detection: use RSVP & single trial spatial integration of high-density EEG • to identify time course & cortical origins leading to response time variability. • RSVP paradigm (Fig 25.5) • Varied scale, pose & position of target objects (people) requires subjects to recognize rather than low-level features • Continuous sequence of natural scenes • 4 blocks (50 sequences) rest period: no more than 5 mins.

  20. Identifying Cortical Process leading to Response Time Variability (Con..) • Each sequences: 50 images, 50% chance containing one target image with one or more people in a natural scene. • Target appear: middle 30 images/50 sequences • Detractor image: remaining natural scene without a person • Each image: 100 ms • Fixation cross: 2s bet’n sequences. • Instructed to press left button of a 3-button mouse with right index finger while fixation cross present, release as soon as recognize target image. • Used LD: to determine spatial weighting coefficients that optimally discriminate bet’n EEG resulting from different RSVP task conditions (target vs. distractor) over specific temporal window bet’n stimulus & response.

  21. Identifying Cortical Process leading to Response Time Variability (Con..) • Integration across sensors enhances signal quality without loss of temporal precision common to trail averaging in ERP. • Resulting discrimination components describe activity specific to target recognition & subsequent response for individual trails. • Intertrial variability estimated by extracting feature from discrimination components. • Peaks of spatially integrated discriminating components found by fitting a parametric function to extracted component y(t)

  22. Identifying Cortical Process leading to Response Time Variability (Con..) • Gaussion Profile: parameterized by its height , width, delay  and baseline offset : • Response-locking of discriminating components determined by computing linear regression coefficients that predict latency of component activity, measured by  from response time given by r. • `j = rj + b [`j-peak latencies, rj-response time for j-th trial, b-offset term for regression] (Fig. 25.6) • Proportionality factor from response time peak latency regression (): degree of response-locking (%) for each com. • Quantify extent to which component is correlated with response across trails.

  23. 0%: pure stimulus • 100%: pure response lock • =100% : slow responses late activity • fast responses • early activity • =0%: timing activity does not change with response time & therefore stimulus locked.

  24. Identifying Cortical Process leading to Response Time Variability (Con..) • Group results: discriminating component activity across 9 participants (Fig. 25.7) • Scalp projections normalized prior averaging • A shift of activity from frontal to parietal regions over the course of 200 ms. • Scaled response times and component peak time are concatenated across subjects • These registered group response times then projected onto scaled component peak times to estimate degree of response-locking across subjects. • Group response lock increased from 28% (200ms) to 78% (50 ms) after response.

  25. Identifying Cortical Process leading to Response Time Variability (Con..) • Significant processing delay introduced by early processing stages • Within 200 ms prior to response (250 ms following stimulus), activity is already, on average, bet’n 25 & 35% response-locked • It is possible that some of this early response-locking may be due to early visual processes (0-250 ms post stimulus • For 9 subjects: correlation Analysis reveal that Discrimination Component activity progressively becomes more response-locked with subsequent Processing stages

  26. EEG-Based Image Triage • EEG system capable of using neural signatures detected during RSVP to triage sequences of images, reordering them so target images are placed near the beginning of sequence. • Called: “Cortically coupled computer vision” • Robust recognition capabilities of human visual system (invariance to pose, lighting, scale) • use a noninvasive cortical interface (EEG) to intercept signatures of recognition events- • visual processor performs perception & recognition EEG interface detects the result (decision) of processing

  27. Image Triage (Con..) • Experimentation: Series of self-paced feedback slides were presented indicating position of target images within sequence before & after EEG triage. • Participants completed 2 blocks (50 sequences) with brief rest period lasting no more than 5 minutes bet’n blocks • During 2nd block: participants were instructed to quickly press left button of 3-button mouse with right index finger as soon as they recognized target images. • Button press twice if one target immediately followed other. • Not respond with a button press during 1st block. • Classify EEG online: used Fisher linear discriminator • to estimate a spatial weighting vector that maximally discriminate between sensor array signals evoked by target & non-target images.

  28. Image Triage (Con..) • 5000 images presented to subject in sequences of 100 images (with & without motor response) • Training: 2500 images (50 targets, 2450 non-target) • Training window: 400-500 ms following stimulus onset was used to extract training data • Weights updated adaptively with each trail during training • Classification threshold adjusted to give optimum performance for observed prevalence (Class-prior) • These weights & threshold were fixed at end of training period & applied to subsequent testing dataset (images: 2501-5000) • Offline triage: after experiment multiple classifier with different training window onsets were used.

  29. Image Triage (Con..) • Training window: 0~900 ms in steps of 50 ms • Duration: 50 ms • After trained, optimal weighting outputs found using logistic regression to discriminate target & non-target images. • EEG data evoked by 2500 images to train & find inter-classifier weights. • applied to testing dataset evoked by 2nd set 2500 images (2501~5000) • Following experiments: all image sequences were concatenate to create training & testing sequences that each contain 2500 images (target# 50, non-targets# 2450). • Image sequences are restored according to output of classifier with multiple training windows for EEG evoked by every image.

  30. Image Triage (Con..) • Comparison: Sequence triaged based on button response. • Image restored: • RT-onset of button response (occurs within1 sec) • P(target/RT)=0, no response • Priors P(target) = 0.02 • P(non-target)=0.98 • P(RT/target): Gaussian distributions with mean & variance determined from response times from training sequences • If target appeared 1st in sequence & 2 button response occurred within 1 sec-assigned to target images • 2nd response to 2nd target image

  31. Image Triage (Con..) • Testing sequences: if 2 or more within 1 sec, response with greatest p(target/RT) assigned to image. • P(RT/non-target) is a mixture of 13 Gaussians (same variance as p(RT/target)) • Means: shifting mean from p(RT/target) 600 ms in past to 700 ms in future, increments of 100 ms, exclude actual mean of p(RT/target) • Triage results (subject 2): Fig. 25.8 • # of targets as a function of # of distracter images both before & after triage based button press & EEG. • area under curve generated by plotting fraction of targets as a function of fraction of distracter images presented is used to quantify triage performance.

  32. Image Triage (Con..) • Area is 0.50 for all unsorted image sequences since target images are randomly distributed throughout sequences. • Ideal: 1.00 • No significant diff in performance (button-based & EEG-based) triage (0.930.06, 0.92  0.03, p=0.69, N=5) • EEG (motor & no-motor)(0.920.03, 0.91  0.02, p=0.81, N=5) • position of target images (black squares) & non-target images (white squares) in images sequences (Raster 25.8b-f) • Performance (5 subjects): Table 25.1

  33. Image Triage (Con..) • Performance (5 subjects): Table 25.1 • High-level performance • Button-based triage performance begins to fail, when subjects do not consistently respond to target stimuli (response time>1 sec), subject 2: only 74% • EEG (motor & button): increase triage performance

  34. Conclusion • Invasive & noninvasive EEG recording obtained during RSVP of natural image stimuli have shed on speed, variability & spatiotemporal dynamics of visual processing. • Future issues: learning/priming/habituation, effect of subject expertise, image type & category • Application level: intercepting neural correlates of visual discrimination & recognition events that effectively bypass “slow & noisy” motor response loop.

  35. Thanks

More Related