1 / 21

Road Map

NEP-02 : Service Oriented Scientific Computing for SAFORAH – Non-linear Denoising Parallelization Demo.

zeus-rivers
Download Presentation

Road Map

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. NEP-02:Service Oriented Scientific Computing for SAFORAH –Non-linear Denoising Parallelization Demo Ashok Agarwal, Patrick Armstrong, Andre Charbonneau, Hao Chen, Ronald J. Desmarais, Ian Gable, David G. Goodenough, Piper Gordon, Aimin Guan, Roger Impey, Kelsey Lang, Belaid Moa, Susan Perkins, Wayne Podaima, Randall Sobie  February 15, 2011

  2. Road Map • Introduction • Introduction to non-linear denoising • Problem – algorithm is expensive • Solution – parallelize • Non-linear denoising application • How we parallelize using the grid? • Application work done during extension • Non-linear Denoising application • SAFORAH: CUDOS web interface • Future work

  3. Introduction • Hyperspectral noise has been assumed to be linear • Linear Stochastic Processes were used to model the noise • Recently, Han and Goodenough (2009) have proved that this is not the case • Non-linear analysis is used to reduce the noise • Results show significant signal to noise ratio SNR boost:

  4. Introduction Non-linear Denoising Results: Original Image Denoised Image Difference Image • Aviris Image (600x1000x179) • RGB: 1503, 750, 645 nm • Linear 2% trimming

  5. Introduction • Typical hyperspectral image cube • 600 samples, 1000 lines, 179 bands • Over 200 MB • Problem: • The algorithm is computationally very expensive. • This 200MB image will take approximately four days to process on a single cpu. • Solution: • Give the algorithm more cpu’s!

  6. Non-linear Denoising • Solution: Give the algorithm more cpu’s: • Parallelize using the computational grid. • Embarrassingly parallel problem: • No spatial data interdependence • Data can be divided spatially for processing

  7. Non-linear Denoising • The image is split into N pieces using the gdal library: • Number of pieces based on image size. • For example, this sample image is broken into 250 pieces.

  8. Non-linear Denoising • Each piece is sent to the grid with the application source code and a script to compile and run it: Job # n Computational grid Denoise.cpp Makefile Denoise.sh

  9. Non-linear Denoising • The pieces are sent to the metascheduler • The metascheduler allocates resources for the jobs: Non-linear Denoise Computational grid Metascheduler Resource 1 Resource 2 Job # n Job # 1 Job # … Job # 2 Job # 3 Resource m

  10. Non-linear Denoising • The nodes copy the completed images back to the local machine • The Non-linear Denoising application stitches the pieces together: Non-linear Denoising Computational grid

  11. Non-linear Denoising • Advantages: • Two orders of magnitude faster! • Completed in as little as 1 hour on a grid with 74 nodes, • instead of 4 days, or about 100 hours. • The time to process dependent on: • Size of image file • Number of nodes available on the grid • How many other jobs are running on the grid

  12. Extension • The activities in this quarter dealt with making the parallel Non-linear Denoising (NLD) application available in the CUDOS GUI, and preparing the application and software infrastructure for longevity. • Developed the parallel NLD application • Improved stability • Improved response time • Made NLD available from CUDOS GUI • Ran end-user testing for NLD • The infrastructure activities included: • Improvements to source code version management

  13. Changes to the Architecture Extension 13

  14. Extension Product Request Splitter Scatterer Pool of Computational Resources JobScheduler Gatherer Merger Results System Flow Chart • Gridway Metascheduler • Portable Batch System (PBS)

  15. Demo of Automated Parallelization of DenoisingSAFORAH CUDOS

  16. Future Work • Update the other existing EO applications according to the user requirements on parallelization (Radarsat 2 decomposition, Aboveground carbon mapping) • Update Grid Web Integration Service to use supported APIs • Globus 4.0 no longer supported • Other possibilities: • Condor • Globus 5.x • Cloud computing • Add more nodes to grid to improve performance. • Make more applications available on OGC and on CUDOS

  17. Conclusion • Accomplished what we set out to do for extension: • Parallelized Non-linear Denoising. • Using grid-computing for speed and efficiency. • Obtained significant performance improvements from parallelization: ~100 x faster!! • Made the application available on SAFORAH • Improved the system infrastructure • Thank you CANARIE!

  18. CUDOS Portal-Create New Product 18

  19. CUDOS Portal -Product Parameters 19

  20. CUDOS Portal -Product Order 20

  21. Results on noise reduction (1) The effectiveness of noise reduction by the LGP was evaluated by comparing the noise intensity and signal-to-noise ratio (SNR) between the original and the de-noised images. A covariance method proposed by Roger and Arnold (1996) was employed here for noise estimation. Without relying on any additional information, this method is capable of producing reasonable noise estimates based on the covariance matrix of the hyperspectral image alone. The average noise intensity of the original AVIRIS image was estimated at 1.2, which varies between 0.3 (minimum) and 11.3 (maximum) across all bands. The LGP removed an average of 34.4% of the noise in the AVIRIS image, which left the noise intensity in the post-denoised image varying between 0.2 and 9.0 with an average of 0.9.

More Related