220 likes | 327 Views
NEP-02 : Service Oriented Scientific Computing for SAFORAH – Non-linear Denoising Parallelization Demo.
E N D
NEP-02:Service Oriented Scientific Computing for SAFORAH –Non-linear Denoising Parallelization Demo Ashok Agarwal, Patrick Armstrong, Andre Charbonneau, Hao Chen, Ronald J. Desmarais, Ian Gable, David G. Goodenough, Piper Gordon, Aimin Guan, Roger Impey, Kelsey Lang, Belaid Moa, Susan Perkins, Wayne Podaima, Randall Sobie February 15, 2011
Road Map • Introduction • Introduction to non-linear denoising • Problem – algorithm is expensive • Solution – parallelize • Non-linear denoising application • How we parallelize using the grid? • Application work done during extension • Non-linear Denoising application • SAFORAH: CUDOS web interface • Future work
Introduction • Hyperspectral noise has been assumed to be linear • Linear Stochastic Processes were used to model the noise • Recently, Han and Goodenough (2009) have proved that this is not the case • Non-linear analysis is used to reduce the noise • Results show significant signal to noise ratio SNR boost:
Introduction Non-linear Denoising Results: Original Image Denoised Image Difference Image • Aviris Image (600x1000x179) • RGB: 1503, 750, 645 nm • Linear 2% trimming
Introduction • Typical hyperspectral image cube • 600 samples, 1000 lines, 179 bands • Over 200 MB • Problem: • The algorithm is computationally very expensive. • This 200MB image will take approximately four days to process on a single cpu. • Solution: • Give the algorithm more cpu’s!
Non-linear Denoising • Solution: Give the algorithm more cpu’s: • Parallelize using the computational grid. • Embarrassingly parallel problem: • No spatial data interdependence • Data can be divided spatially for processing
Non-linear Denoising • The image is split into N pieces using the gdal library: • Number of pieces based on image size. • For example, this sample image is broken into 250 pieces.
Non-linear Denoising • Each piece is sent to the grid with the application source code and a script to compile and run it: Job # n Computational grid Denoise.cpp Makefile Denoise.sh
Non-linear Denoising • The pieces are sent to the metascheduler • The metascheduler allocates resources for the jobs: Non-linear Denoise Computational grid Metascheduler Resource 1 Resource 2 Job # n Job # 1 Job # … Job # 2 Job # 3 Resource m
Non-linear Denoising • The nodes copy the completed images back to the local machine • The Non-linear Denoising application stitches the pieces together: Non-linear Denoising Computational grid
Non-linear Denoising • Advantages: • Two orders of magnitude faster! • Completed in as little as 1 hour on a grid with 74 nodes, • instead of 4 days, or about 100 hours. • The time to process dependent on: • Size of image file • Number of nodes available on the grid • How many other jobs are running on the grid
Extension • The activities in this quarter dealt with making the parallel Non-linear Denoising (NLD) application available in the CUDOS GUI, and preparing the application and software infrastructure for longevity. • Developed the parallel NLD application • Improved stability • Improved response time • Made NLD available from CUDOS GUI • Ran end-user testing for NLD • The infrastructure activities included: • Improvements to source code version management
Changes to the Architecture Extension 13
Extension Product Request Splitter Scatterer Pool of Computational Resources JobScheduler Gatherer Merger Results System Flow Chart • Gridway Metascheduler • Portable Batch System (PBS)
Future Work • Update the other existing EO applications according to the user requirements on parallelization (Radarsat 2 decomposition, Aboveground carbon mapping) • Update Grid Web Integration Service to use supported APIs • Globus 4.0 no longer supported • Other possibilities: • Condor • Globus 5.x • Cloud computing • Add more nodes to grid to improve performance. • Make more applications available on OGC and on CUDOS
Conclusion • Accomplished what we set out to do for extension: • Parallelized Non-linear Denoising. • Using grid-computing for speed and efficiency. • Obtained significant performance improvements from parallelization: ~100 x faster!! • Made the application available on SAFORAH • Improved the system infrastructure • Thank you CANARIE!
Results on noise reduction (1) The effectiveness of noise reduction by the LGP was evaluated by comparing the noise intensity and signal-to-noise ratio (SNR) between the original and the de-noised images. A covariance method proposed by Roger and Arnold (1996) was employed here for noise estimation. Without relying on any additional information, this method is capable of producing reasonable noise estimates based on the covariance matrix of the hyperspectral image alone. The average noise intensity of the original AVIRIS image was estimated at 1.2, which varies between 0.3 (minimum) and 11.3 (maximum) across all bands. The LGP removed an average of 34.4% of the noise in the AVIRIS image, which left the noise intensity in the post-denoised image varying between 0.2 and 9.0 with an average of 0.9.