Center for Subsurface Sensing & Imaging Systems

1. HP Visit @ UPRM February 27, 2003 Center forSubsurface Sensing & Imaging Systems Overview of Research Thrust R3 • R3 Fundamental Research Topics • R3A Parallel Processing • Middleware Tools • FPGA Acceleration • R3B Solutionware Development • Image and Sensor Data Databases • Subsurface Toolboxes David Kaeli, Northeastern University

2. CenSSIS Barriers Addressed by R3 Projects Lack of Computationally Efficient, Realistic Models Barrier 4 Lack of Rapid Processing and Management of Large Image Databases Barrier 6 Lack of Validated, Integrated Processing and Computational Tools Barrier 7

3. Air Mine Soil Middleware Tools • Parallelization using MPI – a software pathway to exploiting GRID-level resources • Presently exploiting NCSA resources (Mariner Center at BU) and Internet-2 • Profile-guided program instrumentation/optimization • Developing parallelization strategies for MATLAB • Utilizing MPI-2 to address barriers in I/O performance Impact on SSI applications: • Reduced the runtime of a single-body Steepest Descent Fast Multipole Method (SDFMM) application by 74% on a 32-node Beowulf cluster • Hot-path parallelization • Data restructuring • Reduced the runtime of a Monte Carlo scattered light simulation by 98% on a 16-node SGI Origin 2000 • Matlab-to-C compliation • Hot-path parallelization

4. Techniques for Parallelizing MATLAB • Manage completely independent MATLAB processes distributed over different processors • Message passing within MATLAB (e.g., MultiMATLAB) • MATLAB calls to parallel libraries (multi-threaded LAPACK, PLAPACK) • Backend compilers can convert MATLAB to C, and automatically inserting MPI calls (e.g., RTExpress) Multiple MATLAB sessions A Single MATLAB session

5. Our Approach for Parallelizing MATLAB Main 0/2502 Convert MATLAB to C using the MATLAB mcc compiler 1 Function Self /Children Runtime 18/0 1 3/35 1 1 Number of Calls 16/0 Tfqmr 1/1604 Convert array structs (generated by mcc) to pointer-based structs where needed 273 0/0 273 273 Mult1 1/1602 0/0 Profile the C program to capture both data flow and control flow 273 Mult 4/1599 273 Multdifl 706/0 273 273 Multfaflone4m 523/6 Parallelize the “hot” regions of the the application using MPI Multfaflone4 359/5 54K 0.6M 2.4M 0/0 0/0 0/0

6. Eliminating I/O Barriers in ParallelSSI Applications • Many SSI applications tend to be file bound or memory bound (or both) • While we can use MPI to parallelize processing, and use MPI-I/O to perform collective I/O, we still are limited to access to a file on a single physical disk • Our present work looks at parallelizing I/O by partitioning files associated with MPI processes • We attempt to utilize, slower and commodity (IDE) local secondary storage

7. Eliminating I/O Barriers in ParallelSSI Applications • Parallelize application using MPI • Parallelize I/O using MPI collective I/O • Profile I/O access patterns on a per process basis • Use profile to guide partitioning to reduce overall execution time by 27%-82% • Presently targeting both file-bound and memory-bound applications

8. Some Recent Publications “Profile-based Characterization and Tuning Subsurface Sensing Applications”, M. Ashouei, D. Jiang, W. Meleis, D. Kaeli, M. El-Shenawee, E. Mizan, M. and C. Rappaport, Special issue of the SCS Journal, November 2002. “Parallel Implementation of the Steepest Descent Fast Multipole Method (SDFMM) on a Beowulf Cluster for Subsurface Sensing Application”, D. Jiang, W. Meleis, M. El-Shenawee, E. Mizan, M. Ashouei, and C. Rappaport, IEEE Microwave and Wireless Components Letters, January 2002. “Electromagnetics Computations Using MPI Parallel Implementation of the Steepest Descent Fast Multipole Method (SDFMM)”, M. El-Shenawee, C. Rappaport, D. Jiang, W. Meleis, and D. Kaeli, Applied Computational Electromagnetics Society Journal, August 2002. “Profile-Guided I/O Partitioning”, Y. Wang and D. Kaeli, Submitted to ICS’03. Other techniques being assessed in this work: • Globus Toolkit – NSF Middleware Initiative • MatlabMPI – MIT LL • RTExpress - Parallelizing compiler for Matlab • Sparse matrix representations

9. Embedded Processing for Accelerated Vessel Enhancement in Retinal Images The Application • Processing live video of the patient retina during laser retinal surgery • Highlighting the vascular structure helps the surgeon avoid damage • Retinal vascular tracing; detection of blood vessels in images of the retina • Utilizes a k-means matched filter to find blood vessels and traces out structure • Embedded into a Xilinx Virtex-based Firebird Computing Engine • Data are 1024x1024 images can be processed at 30 frames/sec • Embedded processing, combined with parallel processing, provides a powerful heterogeneous computing platform for discovery • Presently being applied to identify vascular structures present in sub-skin tumors @ MGH • NU/RPI Research Team • Miriam Leeser – NU-ECE • Badri Roysam – RPI-ECSE

10. CenSSIS Solutionware Toolbox Development • Support the development of CenSSIS Solutionware that demonstrates our “Diverse Problems – Similar Solutions” model • Delivered a software-engineered Multi-View Tomography Toolbox, developed in OOMATLAB • Developing three new CenSSIS Toolboxes • Registration – RPI/WHOI • Hyperspectral Imaging – UPRM • 3-D Modeling - NEU • Establish software development and testing standards for CenSSIS Image and Sensor Data Database • Develop an web-accessible image database for CenSSIS that enables efficient searching and querying of images, metadata and image content • Develop image feature tagging capabilities Matlab 6

11. CenSSIS Toolbox Development CenSSIS Applications written in MATLAB, C, C++, Java, MPI, VHDL and Verilog • Identify classes of CenSSIS algorithms that can be more generally specified to extend applications to multiple research domains • Utilize Software Engineering principles to produce reusable and extensible software toolboxes • Object-oriented design principles used • Exploratory Software Development Model • Library and revision control to support infrastructure MSD LPM Modeling MVT

12. The CenSSIS Multi-View Tomography (MVT) Toolbox MVT Matlab 6 Inverse Modeling Framework OO-MATLAB Electrical Impedance Tomography Diffuse Optical Tomography Frequency-Domain Diffusion Library OO-C++ and IML++ Electrical Resistance Tomography www.censsis.neu.edu/MVT/mvt.html

13. Current Toolbox Development • Development of multi-language toolboxes • Performance tuning • Presently working on three additional toolbox effort • Developing an MPI-2 version of the MVT Toolbox MSD Modeling LPM Matlab 6

14. CenSSIS Image Database System • Deliver an web-accessible database for CenSSIS that enables efficient searching and querying of images, sensor data, metadata and image content • Database Characteristics: • Relational complex queries (Oracle8i) • Data security, reliability and layered user privileges • Efficient search and query of image content and metadata • Content-based image tagging using XML adopting MPEG-7 standards • Easy upload and registration of user images and metadata • Indexing algorithms (2D, 3D and 4D) and partitioning of • the database for better performance • Explore object relational technology to handle collections nucleus 3 4 2 1

15. </MediaCoding> <MediaInstance> <Identifier IdOrganization='Clinomics' IdName= 'BreastCancerCell'>BreastCancerCell// image0001 </Identifier> <Locator> <MediaURL>file://D:/Breast/cells/imag0001.jpg</MediaURL> </Locator> </MediaInstance> </MediaProfile> <StructuredAnnotation> <Who>Patient239</Who> <whatObject>Human primary breast tumor cells</whatObject> <WhatAction> growing in a NASA Bioreactor </WhatAction> <where> St. Mary’s Hospital </where> <When> 09/25/2000 </When> <why> Investigate tumor cells behaviour on microcarrier beads </why> <TextAnnotation xml:lang='en-us'> Higher magnification of view illustrating breast cancer cells with intercellular boundaries on bead surface </TextAnnotation> </StructuredAnnotation> </StillRegion> </Image> <Image> <!-- General Cell Infomation --> <CellInformation> <ID> 9 </ID> <ClinomicsID> 931175495 </ClinomicsID> <DOB> 2/7/30 </DOB> <SEX> F </SEX> <COLL_DATE> 11/2/1993 </COLL_DATE> <Primary_site> Breast </Primary_site> <INITIAL> II </INITIAL> <GRADE> POORLY DIFFERENTIATED </GRADE> <HISTOLOGY> UNKNOWN </HISTOLOGY> <PRIM_SITE2> NONE </PRIM_SITE2> <PRIM_DATE> 4/1/1992 </PRIM_DATE> <MET1_SITE> NONE </MET1_SITE> <MET1_DATE> NONE </MET1_DATE> <TUBE_TYPE> p </TUBE_TYPE> </CellInformation> <StillRegion id="IMG0001"> <MediaProfile> <MediaFormat> <FileFormat>jpeg</FileFormat> <System>PAL</System> <Medium>CD</Medium> <Color>color</Color> <FileSize>332.228</FileSize> </MediaFormat> . . . . . . . . General Image Info Image Source Info Image File Location Info Image Donor Info Image Feature Info Image Format Info

16. CenSSIS R3 Research Thrust Summary • Providing both SSI-related computing research expertise and supporting CenSSIS infrastructure needs • Addressing key research barriers in computational efficiency, embedded computing and image/sensor data management • Developing GRID-level tools and an services • I/O Partitioning Algorithms • Subsurface Toolboxes • Image/Sensor Data Databases