1 / 27

Gen-Tao Chiang, Martin Dove, Stuart Ballard, Ian Frame

UK eScience AHM 2006 Visualization of a Hyperspectral Remote Sensing Model for Coastal Water Quality Applications. Gen-Tao Chiang, Martin Dove, Stuart Ballard, Ian Frame National Institute for Environmental eScience NERC and University of Cambridge September 18, 2006. Contents.

kylan-vance
Download Presentation

Gen-Tao Chiang, Martin Dove, Stuart Ballard, Ian Frame

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. UK eScience AHM 2006Visualization of a Hyperspectral Remote Sensing Model for Coastal Water Quality Applications Gen-Tao Chiang, Martin Dove, Stuart Ballard, Ian Frame National Institute for Environmental eScience NERC and University of Cambridge September 18, 2006 http://www.niees.ac.uk

  2. Contents • Monte Carlo Hyperspectral Remote Sensing Model (MCHRSM) • NIEeS Grid Environment • GRID-MCHRSM • Visualization tools • Conclusion http://www.niees.ac.uk

  3. Monte Carlo Hyperspectral Remote Sensing Model (MCHRSM) • Developed by Florida Tech and NASA Kennedy Space Center used for Submerged Aquatic Vegetation (SAV) monitoring project. • In water quality applications, remote sensing techniques have been used to assess water quality constituents. For instance, chlorophyll, temperature, suspended sediment…. etc. http://www.niees.ac.uk

  4. What is the purpose of MCHRSM? • Using MCHRSM to simulate a images for water quality algorithm development and testing. http://www.niees.ac.uk

  5. What is the problem now? • A Monte Carlo model needs to process on the order of one hundred thousand to one million photons or more for each pixel. • The MCHRSM and associated image processing techniques need a large amount of computation time. http://www.niees.ac.uk

  6. For example: • 1.6 GHz CPU • 1,000,000 photons • We need 30 minutes to run per pixel Today, we want to simulate a 1000x1000 pixel image: 30 min X (1000 X 1000)=30,000,000 min =500,000 hours =20833 days=57 years 57 years X 3(Bands, RGB)=171 years http://www.niees.ac.uk

  7. Hyperspectral Cube generated by Analytical Model with 60 bands (400nm to 700nm) Still not able to generate a MC based hyperspectral cube with limited computing resource http://www.niees.ac.uk

  8. How to solve this problem? from ParallelProcessing to Grid Computing http://www.niees.ac.uk

  9. Grid Computing (eScience) • MCHRSM was written by Fortran-90 and MPI. However, due to the limitation of computing ability within single cluster, one can only generate three bands and a 1000*1000 pixels image in a reasonable time (about a week on 20 processors). • We are moving to eScience environment for accessing more computing resource and better data management. http://www.niees.ac.uk

  10. Methods • Building an e-Infrastructure is the first step in order to achieve the scope of eScience. • Then we run the MCHRSM on this testbed to test the whole procedure. http://www.niees.ac.uk

  11. Modification MCHRSM for eScience Environment • The MCHRSM is using FORTRAN-90 and MPI. and has been developed in order to produce an image matrix. • Each matrix containing the water surface reflectance value. Value of each pixel in this matrix is calculated by the Monte Carlo model. • Each band using different coefficient values from each wavelengths. • One can then choose any three bands to display an RGB image using visualization tools such as GIS software or Google Earth. http://www.niees.ac.uk

  12. Parameters • specific absorption coefficient for pure water, • the specific absorption coefficient for chlorophyll, • the specific absorption coefficient for suspend sediment, • the specific absorption coefficient for dissolved organic matter (DOM), • the specific backscattering coefficient for pure water, • the specific backscattering coefficient for chlorophyll, • the specific backscattering coefficient for suspend sediment, • the concentration of chlorophyll, • the concentration of suspend sediment, • the concentration of dissolved organic matter (DOM), • the depth of the water column at that pixel location. http://www.niees.ac.uk

  13. GRID-MCHRSM • A Grid enable MCHRSM can be produced by two different modifications of the code. • The first modification approach is to take out the MPI code and rewrite the code or use a script to submit each pixel and input data to a Grid environment. However, in this case, one has to know which condor process is running which pixel, location, or band. This process would therefore be quite challenging. • The second modification approach is simply keep the original MPI version but using Grid tools for accessing more resources. However, the challenge will be on system side. http://www.niees.ac.uk

  14. First Approach http://www.niees.ac.uk

  15. Second Approach • This approach presented the challenge of configuring the grid environment to support MPI. • Unfortunately, most departments within the CamGrid VO are currently not supporting MPI, nor the file sharing system. Coordinating different departments to support MPI and overcome firewall issues between each site is the hardest part. • To overcome these issues, a MPI testbed was built within NIEeS grid environment, to test the whole workflow. http://www.niees.ac.uk

  16. http://www.niees.ac.uk

  17. GRID-MCHRSM • We have setup a small testbed to support MPI and testing the workflow using My_Condor_Submit (MCS), Condor-G, Globus and SRB. http://www.niees.ac.uk

  18. GRID-MCHRSM • MCS Job Description File • ****************************************************************************** • Executable = syntheticp2 • Notification = NEVER • GlobusRSL = (stdout=sig-test.out)(job_type=mpi)(count=2) • Globusscheduler = iguana.niees.group.cam.ac.uk/jobmanager-pbs • #Transfer_input_files = one, two_three, four • # Force overwriting when uploading / downloading files • SForce = true • SRBHome = /usr/local/srb/SRB3_4_0/utilities/bin • Sdir = /NIEeS/home/gtniees.NIEeS-1/SIGtest/ • Sget = * • Sput = * • queue • ******************************************************************************** http://www.niees.ac.uk

  19. GRID-MCHSIM Outputs will be put in Data Grid (SRB) [gtniees@iguana SIG_onelayer]$ Sls C-/NIEeS/home/gtniees.NIEeS-1/SIGtest/480 C-/NIEeS/home/gtniees.NIEeS-1/SIGtest/520 C-/NIEeS/home/gtniees.NIEeS-1/SIGtest/650 [gtniees@iguana SIG_onelayer]$ Scd 520 [gtniees@iguana SIG_onelayer]$ Sls /NIEeS/home/gtniees.NIEeS-1/SIGtest/520: matrixo-pure-water-520.txt matrixo-sav-520.txt http://www.niees.ac.uk

  20. Visualization for Model Results • Then, the outputs can be imported by ENVI and create a RGB image. Finally visualized using GoogleEarth or other tools. http://www.niees.ac.uk

  21. KML <?xml version="1.0" encoding="UTF-8"?> <kml xmlns="http://earth.google.com/kml/2.0"> <Folder> <name>NIEeS Google Earth Test</name> <GroundOverlay> <Icon><href>http://cete.niees.group.cam.ac.uk/googlemaptest/img/tmp1.jpg</href></Icon> <visibility>1</visibility> <name>tmp1.jpg</name> <color>ffffffff</color> <refreshInterval>60</refreshInterval> <LatLonBox> <north>27.874026246046025</north> <south>27.783348822698574</south> <east>-80.39687630293932</east> <west>-80.49796036022442</west> </LatLonBox> </GroundOverlay> <Placemark><name>NIEeS</name> <visibility>1</visibility> <description> Created by GT for SIG testing</description> </Placemark> </Folder> </kml> http://www.niees.ac.uk

  22. Pure water http://www.niees.ac.uk

  23. Submerged Aquatic Vegetation (SAV) http://www.niees.ac.uk

  24. Suspended Sediments http://www.niees.ac.uk

  25. Conclusions • We are submitting the GRID-MCHRSM to UK National Grid Services (NGS) to make sure it can be executed on a production GRID. Facing communication protocol issues. • we are evaluating OpenGIS software such as GRASS for image enhancement and GoogleEarth for visualization. • Next step, using FOX library. FOX is a set of Fortran libraries developed by eMinerals allowing one to deal with XML within Fortran program. http://www.niees.ac.uk

  26. Acknowledgement • Thanks for Richard Bruin helping on MCS, Andrew Walker, Mark Calleja on system issues, and all eMinerals, Materials Grid, and NIEeS people. http://www.niees.ac.uk

  27. Thank You Very MuchPlease contact us if you think any potential applications can be benefit from eScience and willing to run a pilot demo project.http://www.niees.ac.uk NIEeS

More Related