Stream Processing of X-ray Microdiffraction Data on Multicores

1. Stream Processing ofX-ray Microdiffraction Data on Multicores Yuzhen Xie, University of Western Ontario (UWO) joint work with Alain Biem, IBM ResearchMichael A. Bauer, UWO Stewart McIntyre, UWO Nobumichi Tamura, Lawrence Berkeley National Lab AMMCS, July 2011

2. Motivation • Efficiently use of multi-core processors to process large blocks of synchrotron XRD data generated at high rates (1 to 10 images per second of each 4MB) • Develop high-performance kernels to achieve near real-time data analysis for synchrotron experiments, the goal of the Active Network Interchange for Scientific Experimentation (ANISE) project

3. CCD Camera Incident X-ray (5 – 30 KeV) Sample Synchrotron X-ray White-beam Microdiffraction Dectris Pilatus 1M CCD at ALS (2010): sub-second readout Diffracted beams An image showing the Laue microdiffration pattern of a unit-cell in a crystal sample

4. Process of Laue Patterns for Micro-texture Analysis Background fit and removal (optional)

5. Example of Crystallographic Orientation and Strain Maps (courtesy: Jing Chao and Marina Fuller, UWO) Orientation map Strain map, average strain: 9.92 x 10-3 Result by XMAS (X-ray Microdiffraction Analysis Software), Advanced Light Source

6. Reference Software Packages • XMAS (X-ray Microdiffraction Analysis Software), Advanced Light Source • 3D X-ray Microdiffraction Analysis Software Package in IDL, Advanced Photon Source • A prototype of C code for a selection of features in Laue pattern analysis, Science Studio and ANISE projects, UWO • Best sequential processing time: 25 to 50 seconds per image

7. Stream Processing Illustration Continuous Ingestion Continuous Analysis 7

8. IBM Streams Programming Model Process Output Input Streams Processing Language (SPADE) Platform optimized compilation

9. Laue XRD Processing System on Streams XRD Image Stream Split operator Bundle Sorting BackgroundRemoval Blob Searching Indexing Strain Peak Fitting Preprocessing • Filters available • -Parabolic • 2D Bruckner-2D Mean Filter • -Formatting • -Parsing • XRD image data -Blobs search -Scheduling for parallel peak fitting Functions Available - Lorentz - Gaussian- Pearson VII Processing Elements (mainly User-defined Operators (UDOPs))

10. Key Implementation Techniques • Efficient Source operator for parsing image files: block reading and type casting • Fine-grained pipelining and cache-efficient background filters • Memory-efficient parallel peak fitting • Organize common parameter values as a stream for shared-use in indexing and strain analysis

11. A Fine-pipelined Background Filter based on Parabolic Method

13. A Pilatus TIFF Image before and after Background Removal

14. Memory-efficient Parallel Peak Fitting

15. Data Management: the Key Issue Blob center b: data set Rb(db x db) is needed for fitting a peak with center at p. Peak center p: data set Rpis needed for integrated intensity computation. Assume p is not far from b. Define R to be the square region (2db x 2db) with center at b. Attach a data set R to a blob tuple rather than passing the whole image to each peaking fitting element. Determine Rb and Rp by coordinate mapping in R. Small data size, good locality, no memory contention, …, and hence efficiency.

16. A SPADE Code Snippet for Blob Searching and Parallel Peak Fitting ## Parse an image streamengStream(height: Integer, width: Integer, emax: …, evalues: DoubleList) := Source()[“file://c4-3_001.spe”,udfbinformat=“speParser”, blocksize=65536*15]{} ## Search blobs and generate blob stream streamblobStream( groupid: Integer, blobid: Integer, …, lroi: DoubleList) := Udop(engStream)[“blobSearch”]{np=“NUM_PF”} ## Split blobs to subgroups for_begin @c 0 to NUM_PF-1 streamsubBlobStream@c(groupid: Integer, blobid: Integer, …, lroi: DoubleList) for_end := Split(blobStream)[groupid]{} ## Parallel peak fitting for subgroups of bobs and bundle all peaks together bundlepeakBundle := () for_begin @c 0 to NUM_PF-1 streamsubPeakFitStream@c(numblobs: Integer, x: Integer, …, inten: Double) := Udop(subBlobStream@c)[“peakFitting”]{} peakBundle+= subPeakFitStream@c for_end

17. Organize Common Parameter Values as one Stream for Shared-use in Indexing and Strain Refinement of all XRD Images Known crystal structure and energy range (5-30 keV) Beam direction kin, Detector position and dimensions List of peak positions on the CCD q Calculated qhkl list of reflections Experimental qi list of reflections kout kin 2q q1 q2 Find triplets a1, a2, a3 (thus q1,q2,q3) matching calculated and measured values within a given angular tolerance a3 a1 q3 Choose triplets indexing the largest number of reflections within a given angular tolerance. Look for “missing” reflections. a2 Strain refinement

18. Streams Live Graph: One Pipeline with 4 Processing Elements for Parallel Peak Fitting Blob Search &Scheduling ImageSourcing Parallel Peak Fitting Parameter Sourcing Indexing Strain 2.5 seconds per image (2084*2084) on an Intel Core2 Quad CPU Q9550 (2.83 GHz, 8 GB RAM and 6 MB L2 cache)

19. Streams Live Graph: 4 Pipelines to Process 4 Images Concurrently in Streaming Mode Super-linear speedup obtained on an Intel Core2 Quad CPU Q9550

20. Conclusion • We present the first stream processing application in the field of synchrotron XRD data analysis. • We show that stream processing is an effective model for efficiently using multicore processors for XRD image data analysis. • Our system provides a high-performance processing kernel to achieve near real-time data analysis of image data from synchrotron experiments. • Our work-in-progress include: evaluation, optimization, configuration and deployment of this kernel to large systems with many cores to process large set of XRD images in parallel and streaming mode. Thank You!