High-Performance Computing, Computational Science, and NeuroInformatics Research

1. High-Performance Computing, Computational Science, and NeuroInformatics Research Allen D. Malony Department of Computer and Information Science NeuroInformatics Center (NIC) Computational Science Institute University of Oregon

2. Outline • High-performance computing research • Interactions and funding • Project areas • TAU parallel performance system • Computational science at UO • Projects • Computational Science Institute • Neuroinformatics research • NeuroInformatics Center (NIC) • ICONIC Grid PNNL UO Visit

3. High-Performance Computing Research • Strong associations with DOE national laboratories • Los Alamos National Lab • Lawrence Livermore National Lab • Sandia National Lab (Livermore) • Argonne National Lab • National Energy Research Supercomputing Center • DOE funding • Office of Science, Advance Scientific Computing • ASCI/NNSA • NSF funding • Academic Research Infrastructure • Major Research Instrumentation PNNL UO Visit

4. Project Areas • Parallel performance evaluation and tools • Parallel language systems • Tools for parallel system and software interaction • Source code analysis • Parallel component software • Computational services • Grid computing • Parallel modeling and simulation • Scientific problem solving environments PNNL UO Visit

5. TAU Parallel Performance System Allen D. Malony Sameer S. Shende Department of Computer and Information Science Computational Science Institute University of Oregon

6. PerformanceTechnology • Instrumentation • Measurement • Analysis • Visualization Parallel Performance Research • Tools for performance problem solving • Empirical-based performance optimization process PerformanceTuning hypotheses Performance Diagnosis properties Performance Experimentation characterization Performance Observation PNNL UO Visit

7. Complexity Challenges for Performance Tools • Computing system environment complexity • Observation integration and optimization • Access, accuracy, and granularity constraints • Diverse/specialized observation capabilities/technology • Restricted modes limit performance problem solving • Sophisticated software development environments • Programming paradigms and performance models • Performance data mapping to software abstractions • Uniformity of performance abstraction across platforms • Rich observation capabilities and flexible configuration • Common performance problem solving methods PNNL UO Visit

8. General Problems How do we create robust and ubiquitous performance technology for the analysis and tuning of parallel and distributed software and systems in the presence of (evolving) complexity challenges? How do we apply performance technology effectively for the variety and diversity of performance problems that arise in the context of complex parallel and distributed computer systems?  PNNL UO Visit

9. TAU Performance System • Tuning and Analysis Utilities • Performance system framework for scalable parallel and distributed high-performance computing • Targets a general complex system computation model • nodes / contexts / threads • Multi-level: system / software / parallelism • Measurement and analysis abstraction • Integrated toolkit for performance instrumentation, measurement, analysis, and visualization • Portable performance profiling and tracing facility • Open software approach with technology integration • University of Oregon , Forschungszentrum Jülich, LANL PNNL UO Visit

10. Paraver EPILOG TAU Performance System Architecture PNNL UO Visit

11. TAU Performance System Status • Computing platforms • IBM SP / Power4, SGI Origin 2K/3K, ASCI Red, Cray T3E / SV-1 / X-1, HP (Compaq) SC (Tru64), HP Superdome (HP-UX), Sun, Hitachi SR8000, NEX SX-5/6, Linux clusters (IA-32/64, Alpha, PPC, PA-RISC, Power, Opteron), Apple (G4/5, OS X), Windows • Programming languages • C, C++, Fortran 77/90/95, HPF, Java, OpenMP, Python • Communication libraries • MPI, PVM, Nexus, shmem, LAMPI, MPIJava • Thread libraries • pthreads, SGI sproc, Java,Windows, OpenMP PNNL UO Visit

12. TAU Performance System Status (continued) • Compilers • Intel KAI (KCC, KAP/Pro), PGI, GNU, Fujitsu, Sun, Microsoft, SGI, Cray, IBM (xlc, xlf), Compaq, Hitachi, NEC, Intel • Application libraries (selected) • Blitz++, A++/P++, PETSc, SAMRAI, Overture, PAWS • Application frameworks (selected) • POOMA, MC++, ECMF, Uintah, VTF, UPS, GrACE • Performance technology integrated with TAU • PAPI, PCL, DyninstAPI, mpiP, MUSE/Magnet • TAU full distribution(Version 2.x, web download) • TAU performance system toolkit and user’s guide • Automatic software installation and examples PNNL UO Visit

13. Math Biology Computer Science Geoscience Neuroscience Paleontology Psychology Computational Science • Integration of computer sciencein traditional sciencedisciplines • Third model ofscientificresearch • Application ofhigh-performancecomputation, algorithmsand networking • Parallel computing • Grid computing PNNL UO Visit

14. Computational Science Projects at UO • Geological science • Model coupling for hydrology • Bioinformatics • Zebrafish Information Network (ZFIN) • Evolution of gene families • Oregon Bioinformatics Tool • Neuroinformatics • Electronic notebooks • Domain-specific problem solving environments • Dinosaur skeleton and motion modeling • Computational Science Institute PNNL UO Visit

15. Computational Science  Cognitive Neuroscience • Computational methods applied to scientific research • High-performance simulation of complex phenomena • Large-scale data analysis and visualization • Understand functional activity of the human cortex • Multiple cognitive, clinical, and medical domains • Multiple experimental paradigms and methods • Need for coupled/integrated modeling and analysis • Multi-modal (electromagnetic, MR, optical) • Physical brain models and theoretical cognitive models • Need for robust tools: computational & informatic PNNL UO Visit

16. Brain Dynamics Analysis Problem • Identify functional components • Different cognitive neuroscience research contexts • Clinical and medical applications • Interpret with respect to physical and cognitive models • Requirements: spatial (structure), temporal (activity) • Imaging techniques for analyzing brain dynamics • Blood flow neuroimaging (PET, fMRI) • good spatial resolution  functional brain mapping • temporal limitations to tracking of dynamic activities • Electromagnetic measures (EEG/ERP, MEG) • msec temporal resolution to distinguish components • spatial resolution sub-optimal (source localization) PNNL UO Visit

17. Integrated Electromagnetic Brain Analysis good spatial poor temporal Cortical Activity Knowledge Base Head Analysis Source Analysis Structural / Functional MRI/PET spatial pattern recognition temporal dynamics Cortical Activity Model Experiment subject Constraint Analysis IndividualBrain Analysis Component Response Model neural constraints Dense Array EEG / MEG temporal pattern recognition Signal Analysis Response Analysis Component Response Knowledge Base poor spatial good temporal neuroimaging integration PNNL UO Visit

18. Experimental Methodology and Tool Integration 16x256bits permillisec (30MB/m) CT / MRI segmentedtissues EEG NetStation BrainVoyager processed EEG mesh generation source localization constrained to cortical surface Interpolator 3D EMSE BESA PNNL UO Visit

19. NeuroInformatics Center (NIC) • Application of computational science methods to cognitive and clinical neuroscience problems • Understand functional activity of the brain • Help to diagnosis brain-related disorders • Utilize high-performance computing and simulation • Support large-scale data analysis and visualization • Advance techniques for integrated neuroimaging • Coupled modeling (EEG/ERP and MR analysis) • Advanced statistical factor analysis • FDM/FEM brain models (EEG, CT, MRI) • Source localization • Problem-solving environment for brain analysis PNNL UO Visit

20. NIC Organization • Director, Allen D. Malony • Associate Professor, Computer and Information Science • Associate Director, Don M. Tucker • Professor, Psychology; CEO, EGI • Computational Scientist, Kevin Glass • Ph.D., Computer Science; B.S., Physics • Computational Physicist, Sergei Turovets • Ph.D., Computer Science; B.S., Physics • Computer Scientist, Sameer S. Shende • Ph.D., Computer Science; parallel computing specialist • Mathematician, Bob Frank • M.S., Mathematics PNNL UO Visit

21. Funding Support • BBMI federal appropriation • DoD Telemedicine Advanced Technology Research Command (TATRC) • Initial budget of approximately $750K • Oct. 1, 2002 through March 31, 2004 • NSF Major Research Instrumentation • ICONIC Grid, awarded • New proposal opportunities • NIH Human Brain Project Neuroinformatics • NSF ITR PNNL UO Visit

22. NIC Approaches • Optimize spatial resolution • MRI structural information • Measurement of skull conductivity • Convergence / co-recording with MEG and fMRI • Optimize temporal resolution • Use EEG/MEG time course for fMRI signal extraction • Decomposition of component analysis (ICA, PCA) • Single-trial analysis • Computational brain models • Boundary and finite element brain models • Brain information databases and atlases PNNL UO Visit

23. EEG/ERP Methodology • Electroencephalogram (EEG) • Event-Related Potential (ERP) • Stimulus-locked measures of brain dynamics • Generated from subject- and trial-based analysis • Raw EEG datasets processed and analyzed • Segmentation to time series waveforms • Blink removal and other cleaning • ERP analysis • Averaging for increasing signal to noise • Characterization with respect to trial conditions • Results visualization • Source localization PNNL UO Visit

24. EGI Geodesics Sensor Net • Electrical Geodesics Inc. • Dense-array sensor technology • 64/128/256 channels • 256-channel geodesics sensor net • AgCl plastic electrodes • Carbon fiber leads • Future optical sensors • EGI + LANL PNNL UO Visit

25. EEG/ERP Experiment Management System • Support EEG-based cognitive neuroscience research • Based on experiment model • Experiment type • Subjects measured for trial types • Management of experiment data • Raw and processed datasets and derived statistics • Per experiment/subject/trial database • Secure protection and storage with selective access • Analysis tools and workflows • Generation of results (across experimental variables) • Analysis processes with multi-tool workflows PNNL UO Visit

26. EEG/ERP Experiment Analysis Environment processed datasets / derived results raw analysis workflow … … virtual services storage resources compute resources PNNL UO Visit

27. Source Localization • Mapping of scalp potentials to cortical generators • Single time sample and time series • Requirements • Accurate head model and physics • High-resolution 3D structural geometry • Precise tissue identification and segmentation • Correct tissue conductivity assessment • Computational head model formulation • Finite element model (FEM) • Finite difference model (FDM) • Forward problem calculation • Dipole search strategy PNNL UO Visit

28. Advanced Image Segmentation • Native MR gives high gray-to-white matter contrast • Edge detection finds region boundaries • Segments formed by edge merger • Color depicts tissue type • Investigate more advance level set methods and hybrid methods PNNL UO Visit

29. Building Finite Element Brain Models • MRI segmentation of brain tissues • Conductivity model • Measure head tissue conductivity • Electrical impedance tomography • small currents are injectedbetween electrode pair • resulting potential measuredat remaining electrodes • Finite element forward solution • Source inverse modeling • Explicit and implicit methods • Bayesian methodology scalp CSF skull cortex PNNL UO Visit

30. Conductivity Modeling Governing Equations ICS/BCS Continuous Solutions Finite-DifferenceFinite-ElementBoundary-ElementFinite-VolumeSpectral Discretization System of Algebraic Equations Discrete Nodal Values TridiagonalADISORGauss-SeidelGaussian elimination Equation (Matrix) Solver  (x,y,z,t)J (x,y,z,t)B (x,y,z,t) Approximate Solution PNNL UO Visit

31. Source Localization Analysis Environment raw … … virtual services storage resources compute resources PNNL UO Visit

32. NIC Computational Cluster (“Neuronic Cluster”) • Dell computational cluster • 16 dual-processor nodes • 2.8 MHz Pentium Xeon • 4 Gbyte memory • 36 Gbyte disk • Dual Gigabit ethernet adaptors • 2U form factor • Master node (same specs) • 2 Gigabit ethernet switches • Brain modeling • Component analysis PNNL UO Visit

33. NIC Relationships Utah LANL Argonne OHSU/ OGI UCSD Internet2 Sandia NCSA USC Academic Labs / Centers Intel IBM UO Departments EGI NIC Psychology BDL BEL Industry UO Centers/Institutes CIS Physics CSI CDSI BBMI CNI NSI PNNL UO Visit

34. NSF MRI Proposal • Major Research Instrumentation (MRI) • “Acquisition of the Oregon ICONIC Grid for Integrated COgnitive Neuroscience Informatics and Computation” • PIs • Computer Science: Malony, Conery • Psychology: Tucker, Posner, Nunnally • Senior personnel • Computer Science: Douglas, Cuny • Psychology: Neville, Awh, White • Approximately $1.2M over three years PNNL UO Visit

35. ICONIC Grid graphics workstations interactive, immersive viz other campus clusters Internet 2 Gbit Campus Backbone CNI NIC NIC CIS CIS 4x8 16 16 2x8 2x16 SMP Server IBM p655 Shared Memory IBM p690 Graphics SMP SGI MARS Distributed Memory IBM JS20 Distributed Memory Dell Pentium Xeon 5 Terabytes SAN Storage System PNNL UO Visit

36. Cognitive Neuroscience and ICONIC Grid • Common questions to be explored • Identifying brain networks • Critical periods during normal development • Network involvement in psychopathologies • Training interventions in network development • Research areas • Development of attentional networks • Brain plasticity in normal development and deprived • Attention and emotion regulation • Spatial working memory and selective attention • Attention and psychopathology PNNL UO Visit

37. Computer Science and ICONIC Grid • Scheduling and resource management • Assign hardware resources to computation tasks • Scheduling of workloads for • PSEs for computational science • Provide scientists an entrée to the computational and data management power of the infrastructure without requiring specialized knowledge of parallel execution • Marine seismic tomograph, molecular evolution • Interactive / immersive three-dimensional visualization • Explore multi-sensory visualization • Merge 3D graphics with force-feedback haptics • Parallel performance evaluation PNNL UO Visit