Parallel and Distributed Computing for Neuroinformatics

1. Parallel and Distributed Computing for Neuroinformatics Allen D. Malony University of Oregon Professor Department of Computerand Information Science Director NeuroInformatics Center Computational Science Institute

2. Outline • Neuroinformatics • Dynamic brain analysis problem • NeuroInformatics Center (NIC) at UO • Neuroinformatics research at the NIC • Dense-array EEG analysis (APECS, HiPerSat, Mc) • Brain image segmentation • Computational head modeling • Ontologies and tool integration (NEMO, GEMINI) • Parallel and distributed computing emphasis • ICONIC Grid HPC system at UO • Cerebral Data Systems • Oregon E-science

3. Neuroscience and Neuroinformatics • Application of computer and information science to the understanding of brain organization and function • Integration of information across many levels • Physical and functional • Gene to behavior • Microscopic to macroscopic scales • Challenges in brain observation and modeling • Structure and organization (imaging) • Operational and functional dynamics (temporal/spatial) • Physical, functional, and cognitive operation (models) • How to create and maintain of integrated views of the brain for both scientific and clinical purposes?

4. Human Brain Dynamics Analysis Problem • Understand functional operation of the human cortex • Dynamic cortex activation • Link to sensory/motor and cognitive activities • Multiple experimental paradigms and methods • Multiple research, clinical, and medical domains • Need for coupled/integrated modeling and analysis • Multi-modal observation (electromagnetic, MR, optical) • Physical brain models and theoretical cognitive models • Need for robust tools • Complex analysis of large multi-model data • Reasoning and interpretation of brain behavior • Problem solving environment for brain analysis

5. Neuroimaging Technology • Alternative techniques for analyzing brain dynamics • Blood flow neuroimaging (PET, fMRI) • Good spatial resolution • allows functional brain mapping • Temporal limitations to tracking of dynamic activities • Electromagnetic measures (EEG/ERP, MEG) • msec temporal resolution • can distinguish fast neurological components • Spatial resolution sub-optimal • requires a mapping to cortical sources • Need integrated neuroimaging technology (How?) • Achieve both good spatial and temporal resolution

6. good spatial poor temporal Cortical Activity Knowledge Base Head Analysis Source Analysis Structural / Functional MRI/PET spatial pattern recognition temporal dynamics Cortical Activity Model Experiment subject Constraint Analysis IndividualBrain Analysis Component Response Model neural constraints Dense Array EEG / MEG temporal pattern recognition Signal Analysis Response Analysis Component Response Knowledge Base poor spatial good temporal neuroimaging integration Integrated Dynamic Brain Analysis

7. Experimental Methodology and Tool Integration 16x256bits permillisec (30MB/m) CT / MRI segmentedtissues EEG NetStation BrainVoyager processed EEG mesh generation source localization constrained to cortical surface Interpolator 3D EMSE BESA

8. NeuroInformatics Center (NIC) at UO • Application of computational science methods to human neuroscience problems • Tools to help understand dynamic brain function • Tools to help diagnosis brain-related disorders • HPC simulation, large-scale data analysis, visualization • Integration of neuroimaging methods and technology • Need for coupled modeling (EEG/ERP, MR analysis) • Apply advanced statistical signal analysis (PCA, ICA) • Develop computational brain models (FDM, FEM) • Build source localization models (dipole, linear inverse) • Optimize temporal and spatial resolution • Internet-based capabilities for brain analysis services, data archiving, and data mining

9. NIC Organization • Allen D. Malony, Director • Don M. Tucker, Associate Director • Sergei Turovets, Computational Physicist • Bob Frank, Mathematician • Dan Keith, Software Engineer (distributed systems grid) • Chris Hoge, Software Engineer (computational) • Ryan Martin / Brad Davidson, Systems administrators • Gwen Frishkoff, Research Associate, Univ. Pittsburgh • Kai Li, Ph.D. student (brain segmentation) • Adnan Salman, Ph.D. student (computational modeling) • Performance Research Lab (my other hat)

10. Brainwave Research 101 • Electroencephalogram (EEG) • Electrodes (sensors) measure uV • EEG time series analysis • Event-related potentials (ERP) • link brain activity to sensory–motor, cognitive functions • statistical average to increase signal-to-noise ratio (SNR) • Signal cleaning and component decomposition • Localize (map) to neural sources

11. Electrical Geodesics Inc. (EGI) • EGI Geodesics Sensor Net • Dense-array sensor technology • 64/128/256 channels • 256-channel GSN • AgCl plastic electrodes • Net Station • Advanced EEG/ERP data analysisand visualization • Stereotactic EEG sensor registration • State of the art technology • Research / clinical products • EGI/CDS medical services

12. Epilepsy • Epilepsy affects more than 5 million people yearly • U.S., Europe, and Japan • EEG in epilepsy diagnosis • Childhood and juvenile absence • Idiopathic (genetic) • Distinguish different types • EEG in presurgical planning • Localize seizure onset • Fast, safe, inexpensive • Dense array improves accuracy • Requires good source modeling

13. EEG Time Series - Progression of Absence Seizure First full spike–wave Pre-spike“buzz’

14. Topographic Waveforms – First Full Spike-Wave 350ms interval

15. Topographic Mapping of Spike-Wave Progression • Palette scaled for wave-and-spike interval (~350ms) -130 uV (dark blue)  75 uV (dark red) • 1 millisecond temporal resolution is required • Spatial density (256) to capture shifts in topography

16. Neuroinformatic Challenges • Dense-array EEG signal analysis and decomposition • Artifact cleaning and component analysis • Automatic brain image segmentation • Brain tissue identification • Cortex extraction • Computational head modeling • Tissue conductivity estimation • Source localization • Statistical analysis to detect brain states • Discriminant analysis • Pattern recognition • Electromagnetic databases and ontologies

17. Applying ICA for EEG Blink Removal • Component analysis is used to separate EEG signals • Independent component analysis (ICA) • Blinks are a major source of noise in EEG data • Blink signals are separable from cognitive responses Raw EEG Formatting EEG preprocessing ICA Analysis Identify blinks and remove • Event info • Time markers • Blink events • Bad channel removal • Baseline correction • … • ICA algorithms - Infomax - FastICA - … • ICA components • Blink templates • Reconstitute EEG w/out blink data ERP Analysis

18. Independent Component Analysis EEG waveform Mixed sinusoids- raw EEG ICA Original sinusoids

19. Tool for EEG Data Decomposition (APECS) • Automated Protocol for Electromagnetic Component Separation (APECS) • Motivation • EEG data cleaning (increases signal-to-noise (SNR)) • EEG component separation (addresses superposition) • Data preprocessing prior to source localization • Distinctive Features • Implements different decomposition methods • Multiple metrics for component classification • Quantitative and qualitative criteria for evaluation

20. APECS Evaluation: Qualitative Criteria Spatio-temporal blink profiles Blink-free baseline Original Bad Good

21. APECS Evaluation: Quantitative Criteria Covariance between “baseline” (blink-free) and ICA-filtered data: Infomax and FastICA. Infomax gives consistently better results. FastICA results are more variable. ICA decompositions most successful when only one spatial projector is strongly correlated with blink “template” (spatial filter).

22. Parallelization of Component Analysis Algorithms • Dense-array EEG increases analysis complexity • Long time measurements require more processing • ICA algorithms are computationally challenging • Processing time and memory requirements • Increase performance through ICA parallelization • High-Performance Signal Analysis Toolkit (HiPerSAT) • Matlab ICA algorithms implemented in C++ • Infomax: Matlab (runica.m)  parallel (OpenMP) • FastICA: Matlab (fastica.m)  parallel (MPI) • Validate results with Matlab standard algorithms • Evaluate accuracy and compare speedup

23. HiPerSAT Parallel Infomax • Requires multi-threading • Over 3 timesfaster thanMatlab • 3-fold increaseon 4 processors • Speedup falls after4 processors • Limits on parallelizationof loop matrix operations • May be able to improvewith larger blocking

24. HiPerSAT Parallel FastICA • Linear speedup • Over 130 times fasterthan Matlab • 8-fold increaseon 32 processors • Performancemay allow finerprocessing • Tradeoff improvedaccuracy versus morecomplex processing andexecution time

25. Matlab Tool Integration • Many neuroimaging tools are based on Matlab • EEGLAB (UC San Diego) • BrainStorm (USC / Los Alamos National Lab) • SPM (University College of London) • Matlab is mostly a closed computational environment • EEG/MEG analysis can overwhelm Matlab • Limited to workstation processing resources • Memory requirements high due to Matlab workspace • Desire to use Matlab as a client in distributed systems • Matlab is not multi-threaded • Complicates building concurrent for external interfaces

26. Mc: Matlab Concurrent • Matlab is built on top of a JVM • Use for GUI and graphics • We can leverage JVM for concurrency and interaction • How do we create concurrent tasks in Matlab? • Matlab programming semantics issue • Create task abstraction • Provide Matlab package for constructing tasks • Concurrent tasks interface with runtime layer • Client task manager runs tasks on servers and monitors • Server task executor schedules tasks on resources • Voila!  Mc

27. Mc System Architecture

28. Mc Task Manager • Single background thread running in the JVM • Coordinate several components: • File transfer • Ganymed SSH2 implementation for Java • Matlab interaction • MatlabControl class • Scripting with Jython • dynamic access to objects in the JVM • Matlab m-scripting through MatlabControl class • waits for free cycles when execution Matlab code • Responds to user or timer events • Manages execution of tasks and status reporting

29. Task Manager Workflow

30. HiPerSAT, APECS, and Mc • Demonstrated APECS running HiPerSAT tasks • 20 simultaneous HiPerSAT servers on a cluster • Compare with sequential EEGLAB processing • ICA processing on 20 1-GB EEG files • Performance speedup • 5x for C++ optimization and 2-way parallelism • 10x speedup for 20 concurrent HiPerSAT tasks • Also demonstrate for multiple Matlab clients • Recently applied to EEG processing for imagery analysis • 100+ simultaneous tasks across 8 different platforms

31. Papers • K. Glass, G. Frishkoff, R. Frank, C. Davey, J. Dien, A. Malony, A Framework for Evaluating ICA Methods of Artifact Removal from Multichannel EEG, ICA Conference, Grenada, Spain, 2004. • R. Frank and G. Frishkoff, APECS: A Framework for Implementation and Evaluation of Blink Extraction from Multichannel EEG, Journal of Clinical Neurophysiology, to appear, 2006. • D. Keith, C. Hoge, A. Malony, and R. Frank, “Parallel ICA Methods for EEG Neuroimaging,” International Parallel and Distributed Processing Symposium (IPDPS 2006), May 2006. • C. Hoge, D. Keith, and A. Malony, “Client-side Task Support in Matlab for Concurrent Distributed Execution,” Austrian-Hungarian Workshop on Distributed and Parallel Systems (DAPSYS), September 2006.

32. Animated Topography of Spike–Wave Dynamics • Spatial and temporal dynamicsare important to observetogether • Linked cortical networks • Fronto-thalamic circuit (executive control) • Limbic circuit (episodic memory) • Problem of Superposition • How many sources? • Where are they located? • Can only infer locations using“scalp space”information

33. Addressing Superposition: Brain Electrical Fields • Brain electrical fields are dipolar • Volume conduction • Depth and location indeterminacy • Highly resistive skull (CSF: skull est. from 1:40 to 1:80) • Left-hemisphere scalp field may be generated by a right-hemisphere source • Multiple sources  superposition • Radial source Tangential sources • one and two sources  varying depths

34. Source Localization • Mapping of scalp potentials to cortical generators • Signal decomposition (addressing superposition) • Anatomical source modeling (localization) • Source modeling • Anatomical Constraints • Accurate head model and physics • Computational head model formulation • Mathematical Constraints • Criteria (e.g., “smoothness”) to constrain solution • Current solutions limited by • Simplistic geometry • Assumptions of conductivities

35. Brain Sources of Epileptic Seizure • Single time point source solution • Need to identify sources for each msec time sample • Visualize dynamics in “source space”

36. Dipole Sources in the Cortex • Scalp EEG is generated in the cortex • Interested in dipole location, orientation, and magnitude • Cortical sheet gives possible dipole locations • Orientation is normal to cortical surface • Need to capture convoluted geometry in 3D mesh • From segmented MRI/CT • Linear superposition

37. Advanced Image Segmentation • Native MR gives high gray-to-white matter contrast • Image analysis techniques • Edge detection, edge merger, region growing • Level set methods and hybrid methods • Knowledge-based • After segmentation, color contrasts tissue type • Registered segmented MRI

38. Network Flow Based Skull Stripping • Graph construction with LoG • Nodes, edges, edge weights, source terminals, sink terminals • Identify non-brain terminals • Scalp, eyeballs, orbits • Fourth ventricle, pons • Identify brain terminals • Knowledge-based terminal detection • Min-cut (Max-flow) • Separate source from sink • Sum of weights is minimum

39. Cortical Surface Extraction Pipeline • Initial segmentation of white matter • Scale-space based analysis of LoG • Incorporation of spatial knowledge • Cortical surface topology correction • Minimize cortical surface are increase • Based on network flow theory • Based on Cauchy-Crofton formula • Cortical surface geometry refinement • White matter shape knowledge • radius function constraints • sheet constraints • Medial surface (skeleton) manipulation

40. Topology Correction in Cortex Extraction Extracted Cortex Before topology correction After topology correction

41. Building Computational Brain Models • MRI segmentation of brain tissues • Conductivity model • Measure head tissue conductivity • Electrical impedance tomography • small currents are injectedbetween electrode pair • resulting potential measuredat remaining electrodes • Finite element forward solution • Source inverse modeling • Explicit and implicit methods • Bayesian methodology

42. Conductivity Modeling Governing Equations ICS/BCS Continuous Solutions Finite-DifferenceFinite-ElementBoundary-ElementFinite-VolumeSpectral Discretization System of Algebraic Equations Discrete Nodal Values TridiagonalADISORGauss-SeidelGaussian elimination Equation (Matrix) Solver  (x,y,z,t)J (x,y,z,t)B (x,y,z,t) Approximate Solution

43. Conductivity Optimization and Parallelization • Design as a conductivity search problem • Master launches new inverse problems with guesses • Inverse solvers run iterative forward calculations • Compare solutions with measured results • Parallelization approach • Forward solver • Alterative Direction Implicit (ADI) method • Optimization search • Simplex or simulated annealing (Monte Carlo) • Hybrid

44. Alternating Direction Implicit (ADI) Method • Finite difference method • C++ with OpenMP for parallelization • LAPACK for matrix operations

45. Conductivity Search Architecture

46. Conductivity Optimization Dynamics • Attempting todetermine tissueconductivity values • Scalp • Skull • Brain • CSF • Simplex algorithm • Values convergeto minimal error • Increase number of features in future work

47. Multi-Cluster Search Dynamics

48. Papers • K. Li, A. Malony, and D. Tucker, “Automatic Brain MR Image Segmentation with Relative Thresholding and Morphological Image Analysis,” International Conference on Computer Vision Theory and Applications (VISAPP), Setúbal, Portugal, February 2006. • K. Ki, A. Malony, and D. Tucker, “A Multiscale Morphological Approach to Topology Correction of Cortical Surfaces,” International Workshop on Medical Imaging and Augmented Reality (MIAR 2006), Sanghai, China, August 2006. • A. Salman, S. Turovets, A. Malony, J. Eriksen, and D. Tucker, “Computational Modeling of Human Head Conductivity,” International Conference on Computational Science (ICCS), best paper, May 2005. • A. Salman, S. Turovets, A. Malony, V. Vasilov, “Multi-Cluster, Mixed-Mode Computational Modeling of Human Head Conductivity,” International Workshop on OpenMP (IWOMP), June 2005. • S. Turovets, A. Salman, A. Malony, P. Poolman, C. Davey, D. Tucker, “Anatomically Constrained Conductivity Estimation of the Human Head in Vivo: Computational Procedure and Preliminary Experiments,” Electrical Impedance Tomography (EIT), July 2006.

49. Neural ElectroMagnetic Ontology (NEMO) • How can brain electromagnetic (EEG and MEG) data be compared and integrated across experiments and laboratories? • Need a system for representation, storage, mining, dissemination • Need standardization of methods for measure generation • Identification and labeling of components • Patterns of interest • General agreement on criteria for component identification • Patterns can be hard to identify • Variability in techniques for measure generation • NEMO will address issue by providing • Spatial and temporal ontology database • Use for large-scale data representation, mining, and meta-analysis • Components in average EEG and MEG (ERPs)

50. Electromagnetic Data Spaces and Representation